Recei ed Ap il 27, 2021, accep ed May 5, 2021, da e o publica ion May 11, 2021, da e o cu en e sion May 21, 2021.
Digi al Objec Iden i ie 10.1109/ACCESS.2021.3079375
Online S uden Au hen ica ion and P oc o ing
Sys em Based on Mul imodal Biome ics
Technology
MIKEL LABAYEN 1,3, RICARDO VEA 1, JULIÁN FLÓREZ2, (Membe , IEEE),
NAIARA AGINAKO 3, AND BASILIO SIERRA 3
1Smowl ech, 20009 Donos ia, Spain
2Vicom ech Resea ch Cen e , 20009 Donos ia, Spain
3Compu e Sciences and A i icial In elligence Depa men , Uni e si y o he Basque Coun y, 20018 Donos ia, Spain
Co esponding au ho : Mikel Labayen ([email p o ec ed])
This wo k was suppo ed by he Spanish Minis y o Sciences, Resea ch and Uni e si ies (Minis e io de Ciencia, Inno ación y
Uni e sidades (MCIU)/Agencia Es a al de In es igación (AEI)/Fondo Eu opeo de Desa ollo Regional (FEDER), Unión Eu opea (UE))
unde G an RTC-2016-5711-7.
ABSTRACT Iden i y e i ica ion and p oc o ing o online s uden s a e one o he key challenges o online
lea ning oday. Especially o online ce i ica ion and acc edi a ion, he aining o ganiza ions need o e i y
ha he online s uden s who comple ed he lea ning p ocess and ecei ed he academic c edi s a e hose who
egis e ed o he cou ses. Fu he mo e, hey need o ensu e ha hese s uden s comple e all he ac i i ies
o online aining wi hou chea ing o inapp op ia e beha iou s. The COVID-19 pandemic has accele a ed
(ab up ly in ce ain cases) he mig a ion and implemen a ion o online educa ion s a egies and consequen ly
he need o sa e mechanisms o au hen ica e and p oc o online s uden s. Nowadays, he e a e se e al
echnologies wi h di e en g ades o au oma ion. In his pape , we deeply desc ibe a speci ic solu ion
based on he au hen ica ion o di e en biome ic echnologies and an au oma ic p oc o ing sys em (sys em
wo k low as well as AI algo i hms), which inco po a es ea u es o sol e he main conce ns in he ma ke :
highly scalable, au oma ic, a o dable, wi h ew ha dwa e and so wa e equi emen s o he use , eliable and
passi e o he s uden . Finally, he echnological pe o mance es o he la ge scale sys em, he usabili y-
p i acy pe cep ion su ey o he use and hei esul s a e discussed in his wo k.
INDEX TERMS Biome ic au hen ica ion, cloud compu ing, compu e ision, da a science applica ions in
educa ion, dis ance educa ion and online lea ning, machine lea ning, secu i y, compu e ision.
I. INTRODUCTION
The e is no doub ha online lea ning has been gaining
popula i y h oughou he pas yea s. This phenomenon is
no su p ising gi en ha online lea ning allows educa ion
ins i u es o ope a e a a lowe cos and wi h g ea e each-
ou o mo e s uden s. Educa ional ins i u ions a e o e ing
cou ses online o le e age he bene i s o online lea ning.
This is especially so since he ad en o Massi e Open
Online Cou ses (MOOC). On he o he hand, COVID-19 has
been a challenge o adi ional ins i u es o e ing ace- o-
ace eaching, and hese ins i u ions ha e had o mig a e (in a
e y sho pe iod o ime) o a ully online educa ion model
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Tony Thomas.
o ced by he pandemic si ua ion. Howe e , online lea ning
implemen a ion p esen s challenges.
E-lea ning has a se ious de iciency, which is he lack o
e icien mechanisms ha assu e use au hen ica ion, in he
sys em login as well as h oughou he session. Especially o
online ce i ica ion and acc edi a ion, he aining o ganiza-
ions need o e i y ha he online lea ne s who comple ed
he lea ning p ocess and ecei ed he academic c edi s a e
p ecisely hose who egis e ed o he cou ses. Inadequa e
me hods o iden i y e i ica ion a ec he eliabili y o c e-
den ials and ce i ica ion ea ned online.
Wi hou ce ain y o he au hen ici y o he online lea ne ’s
iden i y, he aspi a ion owa ds ully online educa ion is
s ymied and he e alua ion o he knowledge and skills
ob ained by he online lea ne is un eliable. In o de o p e-
en comp omising he c edibili y o online acc edi a ion,
72398 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ VOLUME 9, 2021
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
alida ion mus be ca ied ou in a cons an o con inuous
manne . A he same ime, alida ion should be non-in asi e
and non-dis up i e, and does no dis ac he lea ning p ocess.
Online p oc o ing, gene ally e e s o p oc o s (humans)
moni o ing an exam o e he in e ne h ough a webcam.
I includes as well he p ocesses, occu ing a a dis ance,
o au hen ica ing he examinee as he pe son who should be
aking he exam. Online p oc o ing was i s in oduced by
K y e ion [1], [2] in 2006, ma ke ing i as a echnological
solu ion in 2008. Since hen, se e al o he o ganiza ions ha e
ollowed K y e ion’s lead c ea ing mo e capable echnology-
based al e na i es, which a e gaining a en ion, such as online
p oc o ing.
Nowadays, he e a e comme cial solu ions in he ma ke as
well as esea ch publica ions ha y o sol e his p oblem.
Some o hem only au hen ica e he iden i y, o he s mon-
i o , some in eal ime, o he s eco d he sessions. Some
co e only exams o speci ic ac i i ies. Some a e o ally
human based solu ions (non-scalable) o ully au oma ic ones
(non- eliable). The e a e also a ew scien i ic app oaches
which de elop he idea o combining some o he ci ed
unc ionali ies. Howe e , he e is no comp ehensi e and eli-
able solu ion which combines mul i-biome ic con inuous
au hen ica ion wi h con inuous isual and audio moni o ing,
wi h de ice ac i i y moni o ing and lock-down op ions and
human supe ision (only when equi ed) o gua an ee 100%
eliable esul s.
In his wo k we p esen a new sys em which gi es com-
me cial solu ions o all ha was needed. I is based on web
applica ions which o e a con inuous au hen ica ion iden i y
se ice o online s uden s h ough a cons an biome ic ( ace,
oice, yping) ecogni ion sys em (biome ic ai s canno be
los , s olen, o ec ea ed), as well as au oma ic con inuous
p oc o ing h ough au oma ic image and audio p ocessing
(de ice moni o ing & lock-down and inapp op ia e beha iou
de ec ion) allowing online cou ses o gain alue o wha
bene i s bo h ins i u ions and s uden s. This solu ion is based
on a high accu acy biome ics ecogni ion and digi al signal
p ocessing algo i hms and i is complemen ed wi h human
supe ision o hose si ua ions in which he au oma ic algo-
i hms a e no able o de e mine eliable esul s. I can be used
o con inuously au hen ica e he lea ne s, ei he h oughou
he en i e lea ning p ocess, o only a ce ain sensi i e s ages
o e-lea ning. I is con ac less and needs only a low le el o
use collabo a ion. In addi ion, he whole sys em is based on
cloud compu ing echnologies, which emo es geog aphical
and echnological ba ie s o online lea ning p o ide s.
The a icle is o ganized as ollows. Sec ion II gi es an
o e iew o some ele an ela ed wo ks and highligh s he
main di e ences wi h ou app oach. Sec ion III desc ibes he
whole sys em o e iew and wo k low. Sec ion IV con ains
a scien i ic- echnical desc ip ion o co e modules. Sec ion V
p esen s sys em es s o measu e he algo i hms’ pe o mance
as well as a su ey made o use expe ience e alua ion.
Sec ion VI p esen s he esul s o he es s. Finally, sec ion VII
d aws he conclusions and p esen s u u e wo ks.
II. RELATED WORK
The abili y o au hen ica e and moni o online use s is
becoming mo e impo an due o he inc ease o he in e -
ne wo ld (e-lea ning, e-banking, e-gambling, e-go e nmen ).
Since i s human based online p oc o ing sys ems, a ious
ully o semi-au oma ic au hen ica ion and p oc o ing ech-
nologies based on biome ic ea u es ha e appea ed in he
las ew yea s. Biome ics has p o ed i sel o be one o he
bes me hods o ecognizing people based upon physiolog-
ical o beha iou al cha ac e is ics [3]. These echnologies
can be di ided in o wo ca ego ies: hose ha a e based on
physical cha ac e is ics and hose ha a e based on beha iou
cha ac e is ics. The o me includes ace ecogni ion, in-
ge p in scanne s, i is scanne s, ein ma ching, e c. The
la e includes oice ecogni ion, handw i ing ecogni ion,
keys oke dynamics, e c. I is p o ed ha no echnology will
p o ide he igh answe on i s own, bu ha he combina-
ion o di e en solu ions will come up wi h he app op ia e
unc ionali y depending on cus ome needs. In addi ion, mos
emo e au hen ica ion p oc o ing echnologies in ol e some
le el o human in e en ion o ully eliable se ice, he eby
pu ing limi a ions on scale.
These biome ic echnologies ha e been widely used o
a ious pu poses, and hey ha e become mo e and mo e
common in ou daily li es. Howe e , e y ew o hem ha e
been success ully adop ed o online lea ning alida ion.
A. COMMERCIAL SOLUTIONS
Some ini ial app oaches ha e been b ough o ma ke as
comme cial solu ions. The ollowing is an o e iew o hese
se ices:
1) Fully Li e Online P oc o ing: S uden s a e on ideo
and wa ched emo ely by a li e p oc o . Li e p oc o -
ing is a li e online se ice o s uden s aking exams
online. A e making an appoin men , he s uden s a e
aken o he online p oc o ing oom whe e hey will
connec wi h a li e p oc o om one o he wo online
p oc o ing cen es ia hei web came as. The s uden s
connec hei sc een o he p oc o . This allows he
p oc o o see hei compu e sc een. The p oc o asks
hem o show a pho o ID and o answe a ew ques ions
abou hemsel es in o de o e i y hey a e in ac he
igh s uden . Du ing he exam, he p oc o looks a he
s uden di ec ly h ough a webcam. I is a secu e and
comple e solu ion o exam p oc o ing, bu since i is a
non-au oma ic solu ion, i canno deal wi h con inuous
iden i ica ion du ing all lea ning p ocess. Fu he mo e,
i needs a high speed in e ne channel o ansmi ideo
da a, p obably una o dable o di e en pa s o he
wo ld and i is no passi e o s uden s. Some comme -
cial solu ions in he ma ke a e P oc o U [4], Exami y
[5] and So wa e Secu e - PSI [6].
2) Reco ded and Re iewed P oc o ing: Sessions a e
eco ded as he compu e moni o s s uden s. A human
can hen e iew he ideo a any ime a e wa d.
VOLUME 9, 2021 72399
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
In hese sys ems, s uden s use hei own compu e and
a webcam o eco d assessmen sessions, he s uden
and he su ounding en i onmen a e eco ded du ing
he en i e exam. Ins uc o s can quickly e iew de ails
o he assessmen , and e en wa ch he eco ded ideo.
Reco ded p oc o ing has he same limi a ions as li e
p oc o ing. In addi ion, i is a passi e sys em. Howe e ,
nobody analyzes he ideos, so eache s mus wa ch
all o hem in o de o de ec undesi able beha iou s
and main ain he li e p oc o ing ad an ages. Some
comme cial solu ions in he ma ke a e K y e ion [1],
P oc o Exam [7], Respondus [8], Remo e P oc o
[9], P oc o Cam [10], B i ual [11] and Lea ne
e i ied [12].
3) Fully Au oma ed Solu ions: The compu e moni o s
s uden s, i au hen ica es hem and de e mines whe he
hey a e chea ing. These a e au oma ic and passi e
solu ions. They jus co e he beginnings o exams and
wo k submission p ocesses. Howe e , use s mus be
o ally ac i e in his kind o sys em ( hey mus ype
a p ede ined pa ag aph and ake an ID pho o hem-
sel es). In addi ion, his kind o sys em does no co e
all he lea ning p ocess con inuously. Some comme -
cial solu ions in he ma ke a e P oc o io [13], P oc o -
T ack [14], Comp obo [15], Sumadi [16], P oc o F ee
[17], Hono Lock [18] and ExamSo [19].
a) Au hen ica ion echnologies: Recogni ion ech-
nologies a e used o au hen ica e a s uden based
on a p io examina ion o some physical ea u e.
They a e ypically buil upon a be o e/du ing/a e
analysis o e i y ha he same s uden who ini-
ially egis e ed o he cou se was ac ually he
same s uden who ook he exam. Commonly-
known ecogni ion echnologies include acial,
inge p in , o oice ecogni ion. In he las
yea , new biome ic p ocedu es such as keys oke
dynamics (i ecognizes yping pa e ns based on
hy hm, p essu e, and s yle) a e gaining popula -
i y. I is likely ha ecogni ion echnologies will
be mos e ec i e when used wi h some combina-
ion o o he echnologies a ailable.
b) Moni o ing echnologies:
i) Webcams and mic ophones a e one o he
o iginal echnologies used o eplace a li e
p oc o and a e p esen in mos emo e
exam p oc o ing solu ions on he ma ke .
They can eco d indi idual s uden s when he
came a is pa o he compu e , o g oups
when he came a is placed in a class oom.
They can moni o he beha iou o he s u-
den s, whe he hey a e chea ing, ecei -
ing help om o he s uden s, using mobile
de ices, books...Webcam/Mic ophone ech-
nologies o en equi e signi ican s o age
capabili ies so ha ideo eco ds can be
e iewed i necessa y.
ii) Compu e lockdowns a e able o moni o he
ac i i y ca ied ou by he s uden wi hin hei
compu e p e en ing hem om ‘‘su ing he
in e ne ’’ while aking a es . This moni o -
ing will be done only and exclusi ely when
he s uden is doing an ac i i y ha can be
e alua ed.
None o he ci ed comme cial solu ions p o ides a mul i-
biome ic au hen ica ion solu ion o con inuous au hen ica-
ion/p oc o ing se ice (based on au oma ic analysis) h ough
he whole lea ning cou se (no only exams). In addi ion,
his wo k p esen s a comple ely new comme cial app oach
o o e come ba ie s such as low-speed in e ne connec-
ion (using da a samples, no con inuous hea y ideo
signals) o cos ly ex a HW/SW equi emen s (using non-
ins allable and ully in eg a ed in LMS web applica ions).
B. SCIENTIFIC AND ACADEMIC APPROACHES
1) TECHNICAL WORKS
Nowadays, al hough he e a e s ill some non-biome ic based
au hen ica ion app oaches [20], he la es a emp s o online
s uden au hen ica ion au oma ion ends o use biome ic
echnologies; acial [21]–[26], inge p in s [27] o yping
[28], [29]. On he o he hand, some app oaches y some
combina ion o hem, such as ace and oice [30] o ace,
oice and yping [31], [32]. All he app oaches a e ocused
mainly on s uden au hen ica ion wi hou p o iding p oc o -
ing se ice.
I is h ough acial au hen ica ion complemen ed wi h o he
biome ics such as oice o yping ecogni ion, ha an oppo -
uni y appea s in e-lea ning o e i y he absence o auds
while he s uden s do hei ac i i ies on he pla o m.
The main no el con ibu ion o he wo k we p esen in
his a icle includes a comple ely new combina ion wo k-
low o h ee main biome ics p o iding a con inuous and
non-in usi e au hen ica ion se ice. I also adds new au o-
ma ic and con inuous p oc o ing ea u es based on image
and audio signal p ocessing o he sys em. Fu he mo e,
i in eg a es compu e ac i i y moni o ing and lock-down
possibili y and, inally, i e en complemen s he se ice wi h
au oma ic ala ms which igge minimal human supe ision,
gua an eeing he eliabili y o esul s.
Finally, he ecen conce n o sa e y and p i acy has
also p o ided ecen esea ch on his opic ela ed o online
p oc o ing [33].
2) USER EXPERIENCE RELATED WORKS
On he o he hand, e y ew wo ks comple ed he esea ch
abou eache s and s uden use expe ience wi h his kind o
au hen ica ion and p oc o ing app oaches. One o hem com-
ple ed he esea ch abou he implemen a ion o acial e i-
ica ion in o educa ion wi h a success ul posi i e esul [34].
The objec i e was o gua an ee s uden s au hen ica ion and
o know exac ly he amoun o ime ha hey spend in on
o he compu e eading o ealizing hei i ual ac i i ies.
72400 VOLUME 9, 2021
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
TABLE 1. Comme cial solu ions s SMOWL (solu ion desc ibed in his a icle). Se ice cha ac e is ics: 1-Au hen ica ion du ing whole exam o session;
2-Mul i biome ic au hen ica ion (a leas 2 di e en ); 3-Exam moni o ing; 4-Con inuous ( ull cou se) moni o ing; 5-Dishones beha iou de ec ion;
6-To ally Passi e and non-in usi e sys em; 7-Au oma ically analyzed esul s; 8-100% gua an eed and eliable esul s; 9-Pe sonalised ala ms; 10-Human
eal- ime p oc o ; 11-De ice moni o ing. Technical ea u es: 12-Scalable sys em; 13-Flexible access o s uden s - no scheduled; 14-No ex a SW/HW
ins alla ion equi ed o au hen ica ion and p oc o ing; 15-Wo ks wi h low-speed connec ion; 16-Fully in eg a ed in ins i u ion LMS; 17-Mul i-B owse &
de ice. Legal aspec s: 18-EU-hos ed solu ion; 19-GDPR compliance. X- Yes | X- No.
In he same way, a acial au hen ica ion mechanism was also
p esen ed. This insu ed ha he s uden s a e no impe sona ed
o imp o e hei ma ks in i ual es s [35].
III. SYSTEM OVERVIEW
The sys em we p esen in his wo k aims o p o ide a p ac ical
cybe -secu i y solu ion o bo h a) con inuous online use
iden i ica ion (using biome ic echnology) and b) moni o ing
using au oma ic signal p ocessing and a compu e moni o ing
sys em. The au hen ica ion p ocess is based on au oma ic
au hen ica ion o acial images (cap u ed by webcams), audio
clips (cap u ed by he mic ophone) and keys oke dynamics
(cap u ed by he keyboa d), checking ha i is he pe son
ha i eally should be du ing he en i e online in e ac ion.
The moni o ing p ocess is suppo ed by webcams and mic o-
phones oo, checking con inuously ha he s uden is no
making any inapp op ia e beha iou (using o bidden de ices
and applica ions, ecei ing help. . .). I also locks down he
compu e s (wi h a p e ious ins alla ion in he lea ne com-
pu e and consen ) du ing exams o aining sessions p e en -
ing he use om isi ing web pages o o he documen s while
pe o ming he cou se.
The sys em can be used o any online use au hen ica ion
bu i is specialized in he ins i u ions ha o e online cou ses
TABLE 2. S a e-o - he-a solu ions s SMOWL (solu ion desc ibed in his
a icle). Au hen ica ion me hod: 1-Face ecogni ion; 2-Voice ecogni ion;
3-Typing ecogni ion; 4-Con inuous au hen ica ion du ing whole session
(no only a he beginning). P oc o ing-Moni o ing me hod: 5-Image
p ocessing; 6-Audio p ocessing; 7-Sc eensho s cap u e; 8-De ice
in o ma ion cap u e (ac i e window, open p ocesses, pe iphe als de ices,
copy/pas e commands...). P oc o ing-De ice Lock-Down: 9-De ice
lock-down. Gua an ee: 10-Human supe ision o cla i y doub s p o iding
100% gua an eed and eliable esul s. X-Yes | X- No.
p o iding aining and deg ee ce i ica ion, including e i ied
MOOCs and co po a e aining o employees. This sys em
can help e-lea ning p o ide s in hei objec i e o be awa ded
c edi by Quali y Educa ional Agencies o hei cou ses
by seeking aceabili y o e idence o s uden au hen ici y
and hei beha iou . I can be used o ack he con inuous
au hen ica ion o he s uden in all o in sensi i e s ages o
VOLUME 9, 2021 72401
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
FIGURE 1. Au hen ica ion and p oc o ing sys em se -up.
FIGURE 2. P ocessing co e desc ip ion.
e-lea ning. Figu e 1shows gene al se -up o he sys em and
Figu e 2de ails he p ocessing co e desc ip ion.
The comple e sys em wo k low is embedded in cloud
compu ing applica ions, and can be used anywhe e, emo -
ing geog aphical and echnological ba ie s. The gene al
scheme o ope a ion is as ollows and is gi en in mo e de ail
in Figu e 3:
1) The sys em is in eg a ed in o he i ual campus o he
aining cen e (a ailable o di e en LMS pla o ms).
2) The aining cen e sends a code (unique s uden
iden i ie ) wi h an image o he s uden o egis e in
he sys em. Acco ding o sys em da a p i acy policy,
he sys em wo ks wi h images, audio clips...no iden-
i ies, so i lacks connec ion wi h he s uden pe sonal
da a such as name, age o add ess [36].
3) The i s ime he s uden en e s he i ual campus he
sys em akes biome ic samples (pic u e, sho speech,
p ede ined pa ag aph yping) which will help us c ea e
he acking biome ical model.
4) The ea e , whene e he s uden is connec ed o wo k,
biome ic samples will be aken andomly and con in-
uously. This da a is sen o se e s in he cloud. The
online managemen module s o es and analyzes he
da a which is compa ed wi h he biome ical model ha
has been c ea ed p e iously o au hen ica ion pu poses
and analyzed o de ec inapp op ia e beha iou s. All
s o age, analysis and esul s epo and ala m c ea ion
asks a e execu ed in online se e s, making he in e-
g a ion, suppo and main enance asks o ins i u-
ions easie and mo e anspa en . Du ing his pe iod,
he compu e lockdown module can be ac i a ed o
moni o ing pu poses.
5) The esul leads o an indi idual use epo ha is
upda ed cons an ly and o which he aining cen e has
access.
72402 VOLUME 9, 2021
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
FIGURE 3. Sys em wo k low.
The key cha ac e is ics o he sys em a e:
1) Con inuous and no scheduled sys em. P oc o ing
and au hen ica ion p ocesses a e ca ied ou h oughou
he en i e session, no only when use s log in. Fu -
he mo e, in he e-lea ning case, i can ollow e e y
session o he cou se, no only he assessmen s. I is
e y lexible. Se ice is gi en 24/7, anywhe e. P e ious
schedule is no equi ed.
2) Passi e & non-in usi e sys em. The sys em o e s a
passi e sys em o s uden s when aking pho os, audio
clips o keys oke pa e n. I does no need he col-
labo a ion o he s uden and i is con ac less. Fo his
eason, in he case o images, i p ope ly wo ks when
he pose/appea ance/complemen s/exp essions o he
s uden s o he ligh condi ions o he oom a e no
con olled (in he wild), ge ing low-con as images
wi h pa ial occlusions due o w ong posi ion o he
appea ance/complimen s/exp essions a ia ions o he
s uden . Rega ding audio clips, he mic ophone only
eco ds when i de ec s some noise, no hing i he s u-
den is in silence. The clips a e la e analyzed and i
oice is de ec ed in he eco ding i is compa ed wi h
he da a ga he ed du ing egis a ion o he s uden ,
o alida e hei iden i y, o o de ec chea ing when
he e a e di e en oices in he eco ding.
3) Au oma ic and scalable: All cap u e, e i ica ion, da a
managemen and moni o ing epo modules a e ca ied
ou wi h cloud compu ing echnology as se ices in
he cloud. Pho os and pa e ns a e aken au oma ically
and andomly and compa ed wi h he biome ic model
made du ing egis a ion. This scalable au oma ic se -
up makes i possible o b ing his solu ion o o e -
c owded scena ios such as MOOCs.
4) Few equi emen s o he end use . Cloud-based
(SaaS) au oma ic solu ion. Needed Ha dwa e - So -
wa e (HW/SW): basic webcam, mic ophone, keyboa d
and any upda ed b owse . Final use s do no ha e o
ins all any hing. This sys em wo ks o e any de ice,
pla o m, OS and b owse s wi h no ins alla ion needed.
5) Au oma ic analyzed esul s. 100% gua an eed esul s
wi h cus om ala ms. I au oma ic alida ion canno be
con i med (i he pic u es o audio clips do no compile
wi h he quali y needed o allow he sys em o au oma -
ically alida e he s uden ), a manual checking by s a
will be se o ce i y he esul s 100%.
6) Fully in eg a ed in cus ome LMS. I can be in e-
g a ed in any Lea ning managemen sys em (LMS)
using a gene al API bu i has a speci ic plugin o Moo-
dle, Moodle ooms, Blackboa d, OpenedX, Can as, e c.
(mos used LMS).
7) Secu e. Da a is ansmi ed unde secu e in e ne p o-
ocol and s o ed in sa e cloud se e s.
8) P i a e. The use ’s iden i y emains p o ec ed because
we only handle da a ha a e no linked o iden i ies bu
o use codes p o ided by he online en i y.
A. DATA CAPTURE AND STORAGE MODULE
This module cap u es da a om he s uden webcam, mic o-
phone and keyboa d. The co e o his applica ion has been
de eloped using he la es HTML5 s anda d implemen a-
ion in web b owse s. The applica ion is downloaded in o
he s uden ’s e minal and execu ed wi hou any ins alla-
ion needed. Whene e he use is connec ed o he cou se,
quiz o speci ic exe cise in o LMS, pic u es, audio clips and
keys oke dynamics samples will be aken andomly and con-
inuously wi h p ede ined mean pe iodici y. This da a is sen
VOLUME 9, 2021 72403
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
o se e s in he cloud, h ough a SSL enc yp ed channel, wi h
he use iden i ica ion code. The sys em online managemen
module s o es and analyzes he images.
B. AUTHENTICATION MODULE
Once all da a is s o ed in cloud se e s, i is compa ed wi h
he biome ical model, linked o s uden ’s iden i ica ion code,
which has been c ea ed a egis a ion ime and has been
upda ed wi h ecen posi i e da a. The esul is s o ed in he
sys em da abase. The sys em ecogni ion and aining algo-
i hms a e de eloped using he la es algo i hms in a i icial
in elligence (explained in Sec ion IV) which a e imp o ing
cons an ly hei ecogni ion p ecision and obus ness acing
ligh , posi ion and s uden appea ance (physical changes and
complemen s such as ha , glasses...) change p oblems, noise
in audio clips and a iabili y in yping samples. The au hen-
ica ion esul is a combina ion o each biome ic au hen ica-
ion module esul ( ace, oice and yping).
C. PROCTORING AND COMPUTER LOCK-DOWN
MODULES
Du ing moni o ing sessions, he cap u ed image and audio
clips (which ha e been used o au hen ica ion pu poses)
a e p ocessed wi h di e en echniques in o de o de ec
inapp op ia e beha iou o s uden s du ing e-lea ning ac i -
i ies. Fo his eason, he sys em is able o de ec i he
s uden is ecei ing help (by phone, help om p esen ial
iend...) o is checking o bidden documen a ion (books,
o he de ices connec ed o he in e ne ...). All hese ac ions
can be s ic ly o bidden in some ace- o- ace lea ning ac i -
i ies acco ding o he ins i u ion code o honou .
In addi ion, a emp s o chea a e de ec ed and epo ed
i any s uden ies o ick he sys em, such as moun ing a
pho og aph in on o he came a o eplacing he image o
he ID ca d wi h someone else’s. A emp s o inse ano he
image o ideo signal in o he came a a e also de ec ed.
On he o he hand, he sys em con ains a compu e lock-
down module. Du ing all he online session, a compu e lock-
down module (Sec ion IV) will moni o he compu e o
he s uden de ec ing connec ed pe iphe als, ac i e windows,
compu e in o ma ion (HW/SW), execu ing p og ams o p o-
cesses, b owsing his o y/webs and copy-pas e commands.
All he in o ma ion cap u ed in each session is s o ed in he
da abase.
D. HUMAN VERIFICATION MODULE
As pa o he quali y wa an y, a andom da a and esul s
audi o y mus be se . This ask will es y he quali y assu -
ance mechanism de ini ion and implemen a ion wi h a huge
numbe o s uden s connec ed a he same ime. I will be
based on a andom da a c oss- e i ica ion (same images,
oice and keys oke pa e ns alida ed by di e en pe sons)
o images, oice and keys oke samples cap u ed du ing he
session wi h egis e ed da a. Besides, when he quali y o
he pho os o audio does no each he h eshold needed,
a human e i ica ion is made by ained s a deli e ing a
100% eliable e i ica ion o he s uden .
E. REPRESENTATION MODULE OF THE RESULTS
Final esul s a e p esen ed by he da a ep esen a ion module.
I c ea es g aphic cha s and ables on demand, 24h/365d,
as a dynamic web page. The inal epo s can be down-
loaded o p in ed in di e en o ma s. In addi ion, he da a
ep esen a ion module also gene a es au oma ed ala ms when
some p ede ined p ohibi ed beha iou happens.
IV. AUTHENTICATION AND PROCTORING MODULES
IMPLEMENTATION
As explained in he p e ious sec ions, he sys em p esen ed
in his wo k con ains a i icial in elligence-based modules
o use au hen ica ion as well as compu e lockdown ech-
nologies o de ice moni o ing. In his sec ion he scien i ic
algo i hm behind au hen ica ion modules and echnology and
unc ionali ies o he compu e moni o ing a e explained and
e e enced in dep h.
A. FACE DETECTION AND RECOGNITION
This sys em includes a acial de ec ion and ecogni ion mod-
ule h ough a biome ic model c ea ed using egis a ion ime
ace pic u es. The module ou pu esul s a e clus e ed in i e
g oups de e mining: a) I he e is someone in on o he
webcam o no , b) How many people (i any) a e in on o he
webcam, c) I one o hese people is he pe son who should
be in on o he sc een, d) when only one pe son is in he
image, whe he his pe son is he pe son i should be, e) I he
pe son who i should be is no in ol ed in any inapp op ia e
beha iou (book o elec onic de ice use). Some examples
a e shown in Figu e 4.
The e a e di e en app oaches o ace de ec ion in he
li e a u e [37]. Howe e , ew o hem a e obus enough when
dealing wi h a ia ion in pose and ligh ing o cap u ed images
( emembe ha pic u es a e aken wi hou s uden a en ion
and andomly). The acial de ec ion p ocedu e p esen ed in
his wo k is based on he FaceBoxes me hodology [38].
This me hodology is known o being he mos common
‘‘Deep Lea ning’’ based echnique whose op imal deploy-
men is based on use o GPUs. This me hodology ob ains
be e esul s in he Face De ec ion Da a Se and Benchma k
(FDDB) benchma k (Jain and Lea ned-Mille , 2010) han
o he me hodologies es ed in he de elopmen p ocess o his
module.
The image p ocessing and au hen ica ion p ocesses akes
[39] as he base e e ence me hod o he ex ac ion and
no maliza ion o acial ex u e. This algo i hm con ains he
ollowing sub asks: (1) ace de ec ion, (2) ace cha ac e is-
ic poin s de ec ion in he acial egion and (3) de o mable
pa ame ic 3D acial model adjus men based on he de ec ed
poin s. Howe e , he equi emen o sys em passi eness
makes i necessa y o ha e con inuous imp o emen s in he
de ec ion and au hen ica ion algo i hm o deal wi h high
72404 VOLUME 9, 2021
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
FIGURE 4. Au hen ica ion and p oc o ing sys em cap u ed and analyzed image examples.
a iabili y o inpu images. S a ing in his e e ence wo k,
a se ies o imp o emen s ha e been added:
1) Pose and exp essions co ec ion: A new me hod,
called as M3L (Mul i- le el, Mul i-modal, Mul i- ask
Lea ning) [40], is used o imp o e e iciency in ace
poin s and o he acial a ibu es de ec ion (ges u es
o he ace and eyes). M3L add esses he p oblem o
ex ac ing all hese acial and ocula da a h ough a
hie a chy o neu al ne wo ks using exis ing co ela-
ions be ween he da a. Fu he mo e, a new mul i-le el
de o mable 3D acial model adjus men dis ibu es he
de o ma ion e o in an equi able way, dis inguishing
h ee s ages wi h di e en le els o p io i y in es i-
ma ion o ( om g ea e o mino ): (1) pose, (2) in e -
pe sonal de o ma ions (use -speci ic acial shape) and
(3) in a-pe sonal de o ma ions (de o ma ions due o
acial exp essions).
2) Aspec no maliza ion, ea u e selec ion and classi i-
ca ion: The ex ac ion o biome ic ea u es h ough
a deep neu al ne wo k [41] has been imp o ed ain-
ing a da abase wi h 10M o images o 100K indi id-
uals wi h g ea a iabili y o appea ances and acial
shapes, ligh ing, acial exp essions, accesso ies and
poses (Guo e al., 2016).
3) No maliza ion o he ligh ing: The p ocedu e o no -
maliza ion o he ligh ing has been ca ied ou wi h
a hie a chical me hod in which he acial egion as a
whole as well as speci ic and no malized egions o
he ace a e analyzed. This no maliza ion is pe o med
using he Con as Limi ed Adap i e His og am Equal-
iza ion (CLAHE) algo i hm [42], which equalizes he
image locally, highligh ing he con as s, applied o
each RGB colo channel.
4) Robus ness agains pa ial occlusions: Occlusions
de ec ion is based on he MobileNe -SSD neu al ne -
wo k [2], [43]. Combining his pe son de ec o and he
ace de ec o , he sys em inc eases i s obus ness in
de ec ion when (a leas ) pa ial ace occlusion is occu -
ing. This people de ec o (body) is mo e obus han
he ace de ec o in hese cases. The e o e, i a pe son
is de ec ed, bu no a ace, i is mo e likely ha his ace
is a leas pa ially occluded. In his case, he ace de ec-
ion ala m is conside ed. Addi ionally, he me hod-
ology p oposed in [44] has been implemen ed and
adap ed o he amewo k o he needs o he p ojec
o handle he occluded no malized acial images. The
acial de ec ion e u ns mo e pa ially occluded acial
cu s han desi able ones. No mally hese occlusions
a e gi en ei he by he use ’s hands in on o he
ace o because he came a is only poin ing o he op-
hal o he ace. This occlusion nega i ely in luences
he la e s ages o acial poin de ec ion and biome ic
ec o ex ac ion. This sys em includes a acial image
syn hesis om Gene a i e Ad e sa ial Ne wo k [45],
which ills he occluded pa wi h close acial ea u es
ob ained om he ained model. In his way, he nega-
i e impac o occlusion can be educed.
B. VOICE DETECTION AND RECOGNITION
This module implemen s a con inuous oice de ec ion and
au hen ica ion algo i hm. The de elopmen s a e based on he
Kaldi ool [46] and he implemen a ion o he me hod o
[47]. Bo h include ools o he de elopmen o he biome ic
model, he ec o ep esen a ion o each speake ’s cha ac e -
is ics. The algo i hm wo ks on ou asks:
1) Analysis, in e p e a ion and no maliza ion o audio
by VoIP: Since he da a used in VoIP ( echnology in
which his sys em is based on) use he G.711 codec
wi h a 64 kbps bi a e, which implies a loss o
impo an in o ma ion in o de o comp ess he audio
signal, all aining da a om he a ailable acous ic
da abases a e ans o med in o his encoding and o -
ma . In his way, he aining and e alua ion audio
ma ches we e ob ained in he di e en equency
VOLUME 9, 2021 72405
M. Labayen e al.: Online S uden Au hen ica ion and P oc o ing Sys em
anges. Signal p e-p ocessing is in eg a ed o disca d
ha acous ic segmen s ha do no con ain speech
(silence, music o noise). The inal e sion o he VAD
ocal ac i i y de ec ion module has been de eloped
using GMM Gaussian mix u e models and p ocessing
unc ions p oposed in he Kaldi code ool. A o al o 3
model aining le el we e pe o med. The di e ence
be ween each o hem is based on he ans o ma ion
o aining da a o g ea e obus ness e sus he high
acous ic a iabili y o he applica ion scena io.
2) Backg ound and speake modelling: The speake
modelling is based on d- ec o s o speake embeddings
using deep neu al ne wo ks. This solu ion o e s be -
e pe o mance in e ms o obus ness and accu acy.
The implemen a ion ollows he solu ion p esen ed by
Google in 2018 [47]. In his app oach, a ecu en
neu al ne wo k based on LSTM cells is gene a ed.
I ecei es an acous ic cha ac e is ic o a speci ic audio
(Mel il e bank) as inpu and e u ns i s d- ec o . Once
he aining is inished, he neu al ne wo k can be used
o gene a e d- ec o s om he acous ic cha ac e is ics
o he speake . Then, a cen oid is gene a ed, which is
conside ed as he speake ’s biome ic oo p in .
3) Pa e ns compa ison: Fo a e i ica ion o iden i ica-
ion p ocess, gi en a ec o o acous ic cha ac e is ics
and i s associa ed d- ec o , hey a e compa ed wi h he
cen oids o each o he speake s in a new simila i y
ma ix.
4) Speake segmen a ion on s eaming audio: This
dia iza ion sys em employs d- ec o s o speake
embeddings and an agglu ina ion model based on
ecu en neu al ne wo ks [38]. This app oach aims o
o e come he adi ional agglu ina ion app oach p ob-
lems, which wo k wi h he sen ences indi idually and
independen ly, i being di icul o bene i om he
in o ma ion p o ided by la ge amoun s o labelled da a.
This sys em is based on he wo k p esen ed by [48].
An independen ex announce ecogni ion ne wo k is
used o ex ac d- ec o s o speake embeddings om
240 millisecond windows and a 50% o e lap. A ocal
ac i i y de ec o based on Gaussian models is used
o elimina e speechless pa s and spli he signal in o
segmen s less han 400 milliseconds. These segmen s
a e con e ed o d- ec o s and included in he RNN
ne wo k based dia iza ion sys em.
C. TYPING RECOGNITION
Keys oke dynamics a e an e ec i e beha iou al biome ic,
which cap u es he habi ual pa e ns o hy hms an indi idual
exhibi s while yping on a keyboa d. Acco ding o neu ophys-
iological analysis [49], hese yping s yles a e idiosync a ic,
in he same way as handw i ing o signa u es, due o hei
simila go e ning neu onal mechanisms. Fo his eason, hey
can be used o au hen ica e an indi idual.
The sys em p esen ed in his wo k applies keys oke
dynamics in dynamic ex , ha is, he analysis occu s o any
ex ha is yped by he use and con inuously. Keys oke
dynamics in s a ic ex equi es less e o o be implemen ed
and i also eached lowe e o a es in he li e a u e [50].
Howe e , a dynamic ex analysis [51] is necessa y o keep
inal s uden passi eness in he au hen ica ion p ocess wi h-
ou bo he ing hem by asking hem o ype a p ede ined
pa ag aph (usually no ela ed o he e-lea ning ac i i ies in
p og ess). This app oach conside s he ac ha he keys oke
dynamics o one pe son may a y in di e en psycho-
emo ional s a es. Fo example, esea ches no iced [52] ha
i ed people usually ype mo e slowly and make mo e mis-
akes, o his eason, e e y yped sample is s o ed o make
he ecogni ion model mo e obus .
Two dis inc i e p ocesses a e in ol ed in he keys oke
dynamics analysis module:
1) Fea u e ex ac ion: The ex ac ed ea u es (de ailed
iming in o ma ion [53]) a e ime di e ences be ween
he ins an s in which:
a) DT: A key is p essed and eleased.
b) PR: A key is p essed and he nex key is eleased.
c) FT: A key is eleased and he nex is p essed.
d) PP: A key is p essed and he nex key is p essed.
e) RR: A key is eleased and he nex key is eleased.
Based on di e en analysis ca ied ou in de elop and
es cycles, DT (dwell ime) and FT ( ligh ime) ea-
u es a e conside ed he mos ele an ones and hey
a e weigh ed acco dingly. In addi ion, a numbe o yp-
ing mis akes (numbe o p esses o such keys such as
‘‘Dele e’’ and ‘‘Backspace’’) a e calcula ed sepa a ely
as auxilia y pa ame e .
2) Classi ica ion o he ex ac ed ea u es: This mod-
ule employs he CNN+RNN model [54] o lea n a
mo e comple e pe sonal keys oke inpu mode o ca y
ou con inuous au hen ica ion. The sequence leng h
o 30 keys oke da a (bes pe o mance) is ec o ized
and hen di ided in o ixed-leng h keys oke ea u e
sequences in o de o enable keys oke sequences o be
inpu in o he RNN ne wo ks. The ac ha he inpu
da a is p e-p ocessed by CNN (ex ac a highe -le el
keys oke ea u e) imp o es he pe o mance o he
ne wo k model.
D. COMPUTER MONITORING
The needs o online p oc o ing ha e e ol ed. In ecen imes,
he ma ke no only seeks o iden i y s uden s, bu also
o e i y ha hey a e no pe o ming any ype o chea -
ing o beha iou ha is no allowed wi h he de ice on which
s uden s pe o m he ac i i y. In o he wo ds, one o he
g ea es changes is wi hou any doub he desi e o moni o
he ac i i y wi hin he de ice o he s uden s who a e doing
e aluable ac i i ies.
The objec i e o his de elopmen is o ob ain an applica-
ion which is able o moni o he ac i i y ca ied ou by he
s uden wi hin hei compu e . This moni o ing will be done
only and exclusi ely when he s uden is doing an ac i i y ha
72406 VOLUME 9, 2021
