Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/4.
1
Consensual Pupillome y o obo ic emo ional
ecogni ion
D . Feke e Róbe Tamás PhD
Budapes Uni e si y o Technology
and Economics, Facul y o
mechanical enginee ing
Depa men o Mecha onics, Op ics
and In o ma ion Enginee ing
Budapes , Hunga y
[email p o ec ed]me.hu
Raj Le en e MSc
Budapes Uni e si y o Technology
and Economics, Facul y o
mechanical enginee ing
Depa men o Mecha onics, Op ics
and In o ma ion Enginee ing
Budapes , Hunga y
[email p o ec ed]me.hu
Tamas Neume BSc
Budapes Uni e si y o Technology
and Economics, Facul y o
mechanical enginee ing
Depa men o Mecha onics, Op ics
and In o ma ion Enginee ing
Budapes , Hunga y
homasneume @gmail.com
Abs ac – The main con ibu ion o his esea ch is o de elop
o a new measu ing de ice o measu ing di ec - and consensual
ligh e lex o human pupil. In his case he pupilla y ligh e lex
means changing o diame e o pupil due o cons an ligh
s imulus as a unc ion o ime. Resea ches ce i y ha he pupil
diame e and i s change e lec s he pe son’s men al o physical
s a e. The e a e se e al “eye acking” de ices a ailable on he
ma ke , howe e , none o hese a e able o pe o m a p ope
consensual measu emen . The basic p inciple o he consensual
pupil measu emen is ha while only one o he eyes is
s imula ed, eac ions o bo h eyes a e being eco ded and
analyzed. The aim o his p ojec was o de elop a so wa e,
applicable o an exis ing p o o ype de ice in o de o ca y ou
“consensual pupilla y e lex” measu emen s.
Keywo ds – human eye, pupil, consensual pupilla y ligh e lex,
image p ocessing, eye acking, obo ics, emo ional ecogni ion
I. INTRODUCTION
The e a e se e al esea ches discussing he pupilla y ligh
e lex. Since he human pupil is con olled by he au oma ic
ne ous sys em, i is adequa e o objec i ely measu e di e en
physical and men al s a es o he pe son. [1][3]
In 1965 Eckha d H. Hess in es iga ed co ela ion be ween
he es pe sons’ men al s a e and hei pupil size. Pic u es o
di e en opics we e shown o he es pe sons while hei
pupil eac ion was eco ded and measu ed. Hess ound ha
men and women showed di e en eac ions o a gi en pic u e.
Fo example, when he opic o he pic u e was a baby, he
eac ions o women showed signi ican ly g ea e in e es han
ha o men’s. Hess concluded ha he e was a co ela ion
be ween people’s emo ional s a e and hei pupil size in ac .
[13]
Co ela ion can be ound no only be ween men al s a es
and he pupil size bu be ween physical a iables and he pupil
size, oo. In 2003 Lucas RJ examined i he e was co ela ion
be ween mice’s he genome and hei pupilla y eac ion o
ligh . [10]
As discussed abo e, many physical and men al a iables
can be bound o he pupilla y e lex, howe e , mos o hese
we e de e mined as a esul o di ec pupilla y measu emen .
E en he mos ou s anding esea che s in his a ea use he
Tobii eye acking de ice o hei wo k, a de ice ha only
allows di ec measu emen and eye- acking. In he ew cases
when consensual measu emen was ca ied ou , he de ice was
non-p o essional. Thus we can conclude ha he e seems o be
no p o essional de ice on he ma ke pe o ming consensual
measu emen igh now. Since ound no such on he ma ke ,
he inal goal would be o de elop a low-cos , bu p o essional
de ice ha pe o ms bo h di ec and consensual measu emen .
[1]
Figu e 1 – Images om ou ins umen s
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/4.
2
II. HARDWARE INTEFACE
Two Logi ech 200 web came a chips eco d he images
ha will be analyzed as he las s ep o he measu emen . An
“NI USB 6008” inpu /ou pu de ice, manu ac u ed by
Na ional Ins umen s, con ols a se o in a and no mal LEDs.
The no mal LEDs a e u ned on and o du ing he
measu emen , changing he amoun o ligh ha eaches he
es pe son’s eyes, hus o cing he pupils o dila e o con ac .
The in a ed LEDs enable o ake pic u es e en in da k
en i onmen , while he no mal LEDs a e u ned o .
Figu e 2 – Consensual pupilla y measu ing ins umen ha dwa e
III. SOFTWARE INTEFACE
The so wa e was w i en in C# language, he Mic oso
.Ne en i onmen was used as de elopmen pla o m.
Al hough he e a e se e al amewo ks o he image
p ocessing and I/O handling ex e nal lib a ies such as he
A o ge.ne , Mic oso Di ec Show, and Na ional Ins umen
DAQmx we e used in ou p ojec in o de o lowe he cos s.
[3]
The use in e ace p o ides op ions o se came a
pa ame e s and o cus omize he measu emen i sel . In a
ypical measu emen , i s he le , hen he igh eye is being
s imula ed. The came as ake images and la e he so wa e
calcula es he pupil di-amme e on hese images. The esul s
a e s o ed in an Excel ile, enabling u he p ocessing and
analysis.
The so wa e de e mines he pupil diame e based on ideo
s eams cap u ed du ing he measu emen . To make i mo e
e icien , i doesn’ calcula e he pupil size in eal ime, bu i
sa es he ames a a use speci ied loca ion.
Figu e 3 - S a /S op came as, Se Image Loca ion
Th oughou he measu emen he pupil size is measu ed
i s in da kness, while i is s imula ed by he ligh emi ed by
LEDs. In o de o ge esul s which a e compa able wi h each
o he , he algo i hm calcula ing he pupil diame e has o be
he same in bo h ligh and da k ci cums ances. Usually web
came as ha e a so called “Au o Balance” unc ion ha se s he
exposu e p ope y o he came a in o de o ge an image wi h
he desi ed luminance. Howe e , his unc ionali y esul s in
d as ic ame pe second (FPS) d op. To a oid ha , he “Au o
Balance” unc ion o he came as had o be u ned o and he
pa ame e adjus men was achie ed manually by applying
di e en gain pa ame e s o he ligh and da k en i onmen s.
A e ha ing he de ice placed in measu ing posi ions, he
p og am calcula es he gain alues ha will be used o ge
images o he same luminance h oughou he whole
measu emen .
Figu e 4 - Au o-Adjus P ope ies
To ully unde s and he Au o-Adjus unc ions, i s he
image p ocessing algo i hm is discussed. The unc ion akes
he image o he eye and applies a h eshold il e o a ce ain
alue so ha he i ele an de ails o he image a e il e ed ou .
Then i c ea es he nega i e o he image and by a “ illing”
algo i hm i ills up he spo s ep esen ing he mi o ed LEDs.
As he las s ep, he shape ecogni ion algo i hm is called o
de ec he ci cle, which is he ou e pe ime e o he pupil.
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/4.
3
Figu e 5 - S eps de ec ion algo i hm
In conclusion, he “Au o-Adjus ” unc ion has go h ee
objec i es: ind a gain alue ha can be applied in da kness,
ind a h eshold alue ha e u ns he pupil, and ind a gain
alue ha makes images o he same luminance, as i was in
da kness.
In a measu emen he p og am akes a gain alue, and
applies i o he came a. The pupil diame e is calcula ed wi h
di e en h eshold alues on he pic u e sho . When he pupil
diame e has been calcula ed on a gi en image wi h he whole
se o h eshold alues, he p og am akes he nex gain alue,
applies i o he came a and epea s he p ocess again. In he
end he p og am ends up wi h a able like he ollowing:
Figu e 6 - Table o he pupil diame e s a di e en gain and h eshold pai s
(The columns a e he di e en gain alues, while he ows ep esen he
di e en h eshold alues)
The algo i hm p ocesses each column and de e mines he
gain alue he pupil was de ec ed wi h he mos ime. (Column
ma ked wi h yellow on Figu e 6). The middle alue o he
column is aken and he co esponding h eshold alue will be
used la e . (Red cell on Figu e 6)
While looking o he gain alue in ligh , he p og am i s
akes an image in da k and calcula es i s luminance – using
la e as a e e ence. Then i shoo s pic u es wi h di e en gain
alues and compa es he luminance o hese images wi h he
luminance o he e e ence. The gain alue o he smalles
de ia ion co esponding will be applied in ligh .
The di e en LED e en s a e se in he ou h s ep. The
ope a o de ines as an e en sequence which LEDs shall be
swi ched on and a which ime. I is also possible o sa e, and
load a gi en e en sequence.
Figu e 7 - Se e en s
The measu emen s a s only a e he amoun o ime
speci ied in he nume ic up-down has elapsed, so ha he
ope a o has ime o place on he de ice.
Figu e 8 - S a Cap u ing
To p ocess he images made by a measu emen un he
image-con aining olde is opened. I he “Measu emen
Co ec ion” is checked an algo i hm co ec s he measu emen
e o s.
Figu e 9 - Analyze images
The p og am loops h ough he images, calcula es he pupil
diame e on he image and w i es he alue in an excel able.
When all images ha e been p ocessed, he co ec ion
algo i hm p ocesses he able, and co ec s he e o s: i checks
i he i s and las pupil diame e s a e 0. I so, i eplaces hem
wi h he i s o he las non 0 alues. (79.5 and 79 on Figu e
10) Then i looks o sequences, when he pupil couldn’ be
de ec ed. (Se ies o 0s.) I ound, hen he p og am applies he
linea eg ession model and eplaces he missing alues.
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/4.
4
Figu e 10 - Example o he co ec ion algo i hm
When he came a pa ame e s ound by he p og am we e
co ec , he e is no signi ican luminance di e ence be ween
pic u es aken in da kness, o LED ligh .
Figu e 11 - Images o same luminance du ing a LED lash
IV. MEASUREMENT RESULTS
As a esul , he p og am calcula es he pupil diame e s on
he images, and ills and excel able wi h he alues.
Figu e 12 - Measu emen esul s o he le eye
Figu e 13 - Measu emen esul s o he igh eye
As seen on Figu e 12 and Figu e 13 he i s (o ange) and
he second (g ay) measu emen s a e loca ed a he om he
o he 3. In e es ingly, hese measu emen s we e made wi h
only 20 FPS, while he o he h ee wi h 30 FPS. The
unde lying eason o his migh be ha some backg ound
p ocesses o he compu e we e using much esou ces and i
was no able o p ocess 30 bu only 20 ames pe second. I
he i s wo measu emen s a e igno ed, he esul s a e mo e
con incing.
Figu e 14 - Measu emen esul s o he le eye
Figu e 15 - Measu emen esul s o he igh eye
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/4.
5
A ano he es , he esul s o 10 measu emen s we e aken
in o accoun . He e he esul s ha e highe s anda d de ia ion.
This may be a esul o he ac ha all he 10 measu emen s
we e aken a e each o he ha migh ha e esul ed in men al
o physical di e en ia ions.
Figu e 16 - Measu emen esul s o he le eye
Figu e 17 - Measu emen esul s o he igh eye
The e a e se e al ways o op imize he de ice in o de o
ge be e esul s. On he ha dwa e side, he wo “comme cial”
web came as could be eplaced by mo e p o essional ones.
These came as could deli e pic u es o highe esolu ion and
F ame pe Second a e. As seen on Figu e 12, he e lec ion o
he LEDs a e p esen on he pupil’s image as iny whi e do s.
Howe e when he pupil con ac s, hese whi e do s o en
in e sec wi h he edge o he pupil, hus making i ha d o he
shape ecogni ion so wa e o ind a co ec ci cle. In o de o
a oid ha he LED ligh s could be eposi ioned, o he whole
concep could be eplaced by he “Pupil Cen e Co neal
Re lec ion”, since he e he pupil is shiny and i ’s easie o
de ec on he pic u e.
On he so wa e side he e could ha e been used o he
objec ecogni ion algo i hms as well such as OpenCV o “C”.
A be e co ec ion algo i hm could also enhance and
smoo hen he esul s. [6]
REFERENCES
[1] Ana Belén Roig, Julián Espinosa, Jo ge Pé ez and Da id Mas (2014):
High-speed ideo-oculog aphy applied o assess pupil ligh e lex. VII
Eu opean / I Wo ld Mee ing in Visual and Physiological Op ics
[2] Ana Lau a A. Mou a (2013): The Pupil Ligh Re lex in Lebe ’s
He edi a y Op ic Neu opa hy, In es Oph halmol Vis Sci. 2013 Jul 2;
54(7):4471-7.
[3] Jianxin Wu, Nini Liu, Ch is ophe Geye , James M. Rehg (2013):
AReal- ime Objec De ec ion F amewo k – IEEE T ansac ions on Image
P o-cessing, doi: 10.1109/TIP.2013.2270111
[4] Malin Fo ne (2012): Physiology as a ool o UX and Usabili y es ing,
S ockholm, Mas e o Science Thesis
[5] Somlai Juid , Ko ács Tibo (2012): Neu o-oph halmologia, NOSZA
Alapí ány, Budapes , 2012
[6] Malin Fo ne (2012): Physiology as a ool o UX and Usabili y es ing,
S ockholm, 2012
[7] Má kus A ila (2012): Neu ológia. Akadémiai Kiadó
[8] Bha i Vidyapee , Bha a i Vidyapee h (2011): Ad ances in image p o-
cessing o de ec ion o plan diseases - Jou nal o Ad anced
Bioin o ma -ics Applica ions and Resea ch, ISSN 0976-2604, pp 135-
141
[9] Ying Gao, A mando Ba e o, Malek Adjouadi (2009): De ec ion o
Sympa- he ic Ac i a ion h ough Measu emen and Adap i e P ocessing
o he Pupil Diame e o A ec i e Assessmen o Compu e Use s -
Ame ican Jou nal o Biomedical Sciences, ISSN: 1937-9080 doi:
10.5099/aj090400283
[10] Lu Yin, Robe G. Smi h, Pe e S e ling, and Da id H. B aina d (2009):
Physiology and Mo phology o Colo -Opponen Ganglion Cells in a
Re i-na Exp essing a Dual G adien o S and M Opsins Responses
Follow a Du-al G adien in Cone Opsin Exp ession - The Jou nal o
Neu oscience doi: 10.1523/JNEUROSCI.5471-08.2009
[11] Lucas RJ, Ha a S, Takao M, Be son DM, Fos e RG, Yau KW (2003):
Di-minished pupilla y ligh e lex a high i adiances in melanopsin-
knockou mice – Science, PMID: 12522249
[12] Csépe Valé ia, Ál alános pszichológia I., OSIRIS Kiadó, Budapes , 2007
[13] Lu Yin, Robe G Smi h, Pe e S e ling, Da id H. B aina d (2006):
Ch o-ma ic P ope ies o Ho izon al and Ganglion Cell Responses
Follow a Du-al G adien in Cone Opsin Exp ession – The Jou nalo
Neu oscience, doi: 10.1523/JNEUROSCI.1071-06.2006
[14] Eckha d H. Hess (1965): A i ude and Pupil Size, Scien i ic Ame ican
no 212, 1965, pp. 46-54.