HumanEYEze 2024: Wo kshop on Eye T acking o Mul imodal
Human-Cen ic Compu ing
Michael Ba z Roman Bedna ik And eas Bulling
michael.ba z@d ki.de [email p o ec ed] [email p o ec ed]s u ga .de
Ge man Resea ch Cen e o A i icial
Uni e si y o Eas e n Finland Uni e si y o S u ga
In elligence (DFKI) Joensuu, Finland S u ga , Ge many
Saa b ücken, Ge many
Uni e si y o Oldenbu g
Oldenbu g, Ge many
C is ina Cona i Daniel Sonn ag
[email p o ec ed] daniel.sonn ag@d ki.de
Uni e si y o B i ish Columbia
Ge man Resea ch Cen e o A i icial
Vancou e , B i ish Columbia, Canada In elligence (DFKI)
Saa b ücken, Ge many
Uni e si y o Oldenbu g
Oldenbu g, Ge many
Abs ac
The HumanEYEze 2024 wo kshop aims o explo e he ole o eye
acking in de eloping human-cen e ed mul imodal AI sys ems.
O e he pas wo decades, eye acking has e ol ed om a diag-
nos ic ool o an impo an inpu modali y o eal- ime in e ac i e
sys ems, d i en by ad ancemen s in ha dwa e ha ha e imp o ed
i s a o dabili y, a ailabili y, and pe o mance. Ini ially used in spe-
cialized applica ions, eye acking now signi ican ly impac s e-
sea ch on gaze-based mul imodal in e ac ion. Recen ly, eye-based
use and con ex modeling has eme ged, u ilizing eye mo emen s
o p o ide ich insigh s in o use beha io and in e ac ion con-
ex s. The wo kshop aims o b ing oge he esea che s om eye
acking, mul imodal human-compu e in e ac ion, and AI. I aims
o enhance unde s anding o in eg a ing eye acking in o mul i-
modal human-cen e ed compu ing. The expec ed ou comes include
os e ing collabo a ions and p omo ing knowledge exchange.
CCS Concep s
• Human-cen e ed compu ing
→
Use models; Human com-
pu e in e ac ion (HCI); • Compu ing me hodologies
→
A i-
icial in elligence; Machine lea ning;
Keywo ds
Eye T acking, Gaze, Mul imodal In e ac ion, Use Modeling, Human-
cen ic Compu ing, Human-cen e ed AI, Wo kshop
Pe mission o make digi al o ha d copies o all o pa o his wo k o pe sonal o
class oom use is g an ed wi hou ee p o ided ha copies a e no made o dis ibu ed
o p o i o comme cial ad an age and ha copies bea his no ice and he ull ci a ion
on he i s page. Copy igh s o componen s o his wo k owned by o he s han he
au ho (s) mus be hono ed. Abs ac ing wi h c edi is pe mi ed. To copy o he wise, o
epublish, o pos on se e s o o edis ibu e o lis s, equi es p io speci ic pe mission
and/o a ee. Reques pe missions om [email p o ec ed].
ICMI ’24, No embe 04–08, 2024, San Jose, Cos a Rica
© 2024 Copy igh held by he owne /au ho (s). Publica ion igh s licensed o ACM.
ACM ISBN 979-8-4007-0462-8/24/11
h ps://doi.o g/10.1145/3678957.3688384
ACM Re e ence Fo ma :
Michael Ba z, Roman Bedna ik, And eas Bulling, C is ina Cona i, and Daniel
Sonn ag. 2024. HumanEYEze 2024: Wo kshop on Eye T acking o Mul i-
modal Human-Cen ic Compu ing. In INTERNATIONAL CONFERENCE ON
MULTIMODAL INTERACTION (ICMI ’24), No embe 04–08, 2024, San Jose,
Cos a Rica. ACM, New Yo k, NY, USA, 2 pages. h ps://doi.o g/10.1145/
3678957.3688384
1 In oduc ion
O e he las 20 yea s, eye acking has e ol ed om being a diag-
nos ic ool o a powe ul inpu modali y o eal- ime in e ac i e
sys ems. This was pa ly d i en by ad ances in eye acking ha d-
wa e conce ning he de ices’ a o dabili y, a ailabili y, pe o mance,
and o m ac o . Eye acking was i s used in niche applica ions
in he ’80s and ’90s and hen ga he ed signi ican a en ion h ough
esea ch on gaze-based in e ac ion and gaze-suppo ed mul imodal
in e ac ion [
8
,
12
,
15
]. In he las 10-15 yea s, a hi d e y p omis-
ing di ec ion has eme ged: eye-based use and con ex modeling,
i.e., seeing he eyes as an addi ional modali y ha p o ides ich
in o ma ion abou use (in e ac i e) beha io and hei (in e ac-
ion) con ex [
3
,
6
,
10
,
13
,
14
]. The eyes e eal in o ma ion abou
isual ac i i ies [
7
], pe sonali y [
9
], use in en s and goals [
4
,
5
],
a en ion [
2
], expe ise and o he cogni i e abili ies [
1
,
11
], and
emo ions [
16
], o name a ew. Wi h ha , eye acking bea s g ea
po en ial o de eloping human-cen e ed mul imodal AI sys ems.
Gaze-based mul imodal use models can be used o, e.g., gene a e
di ec eedback o s ee he aining o AI sys ems o igge ex-
plici eedback eques s (o show model explana ions) i he use
seems o disag ee wi h he ou pu o an AI sys em. The goal o
his wo kshop is o b ing oge he esea che s om eye acking,
mul imodal human-compu e in e ac ion, and a i icial in elligence.
We will welcome con ibu ions on he ollowing opics:
•
Me hods and sys ems o analyze e e yday eye mo emen
beha io
• Real- ime s. pos -hoc analysis and modeling
• Eye acking in human-cen e ed AI sys ems
696
ICMI ’24, No embe 04–08, 2024, San Jose, Cos a Rica Michael Ba z, Roman Bedna ik, And eas Bulling, C is ina Cona i, and Daniel Sonn ag
•
Adap i e gaze-based and gaze-suppo ed mul imodal use
in e aces
• Eye-based use modeling wi h limi ed da a
• Eye acking o mul imodal use modeling
•
Eye-suppo ed mul imodal ac i i y and con ex ecogni ion
•
Compu e ision me hods o gaze es ima ion and mul i-
modal beha io analysis
•
Gaze sensing sys ems - eal-wo ld benchma ks, equi e-
men s, echniques
• P i acy-p ese ing eye acking
• Reposi o ies and da ase s
• Focused e iews and me a-analyses
2 Expec ed Ou come and Impac
Ou goal is o es ablish a unique discussion pla o m o esea che s
in he ield o eye acking wi h a ocus on eye acking in mul i-
modal human-cen ic in e aces and AI as an enabling echnology.
We aim o b ing oge he esea che s and p ac i ione s om he
ields o eye acking, mul imodal in e ac ion, machine lea ning,
human-compu e in e ac ion, psychology, and o he ela ed ields.
This shall os e collabo a ions among esea che s in he ield and
enable a knowledge exchange on he ole o eye acking in mul i-
modal in e ac ion and AI-based sys ems.
3 Wo kshop Con ibu ions
We included ou pape s in he wo kshop p oceedings. The opics
include s udying collabo a i e in e ac ion beha io based on eye
acking in mixed- eali y se ings, p edic ing d i ing decisions us-
ing ision ans o me s and gaze, modeling when use s disag ee
wi h he ou pu o gene a ed image cap ions, and in es iga ing he
impac o ambien illumina ion change on he accu acy o head-
moun ed eye acke s. The accep ed pape s include:
•
3D Gaze T acking o S udying Collabo a i e In e ac ions
in Mixed-Reali y En i onmen s by Edua do Da alos, Yike
Zhang, Ashwin T S, Joyce Ho n Fon eles, Umesh Timalsina,
and Gau am Biswas.
•
Gaze-In o med Vision T ans o me s: P edic ing D i ing Deci-
sions Unde Unce ain y by Sha a h Koo a ho a, Nikolas Pa-
padopoulos, Jia Li Ma, Sh u i Kuma , Xiaoxiao Sun, A unesh
Mi al, Pa ick Adelman, and Paul Sajda.
•
De ec ing when Use s Disag ee wi h Gene a ed Cap ions by
Omai Shahzad Bha i, Ha shinee S i am, Abdul ahman Mo-
hamed Selim, C is ina Cona i, Michael Ba z, and Daniel Son-
n ag.
•
In es iga ing he Impac o Illumina ion Change on he Ac-
cu acy o Head-Moun ed Eye T acke s: A P o ocol and Ini ial
Resul s by Mohammadhossein Sala i and Roman Bedna ik.
Acknowledgmen s
This wo k was unded, in pa , by he Eu opean Union unde g an
numbe 101093079 (MASTER, h ps://www.mas e -x .eu), he Ge -
man Fede al Minis y o Educa ion and Resea ch (BMBF) unde
g an numbe 01IW23002 (No-IDLE), by he Lowe Saxony Minis y
o Science and Cul u e, and he Endowed Chai o Applied A i icial
In elligence o he Uni e si y o Oldenbu g.
Re e ences
[1]
Oswald Ba al, Sébas ien Lallé, G igo ii Guz, Ali eza I anpou , and C is ina
Cona i. 2020. Eye-T acking o P edic Use Cogni i e Abili ies and Pe o mance
o Use -Adap i e Na a i e Visualiza ions. In P oceedings o he 2020 In e na ional
Con e ence on Mul imodal In e ac ion (ICMI ’20). Associa ion o Compu ing Ma-
chine y, New Yo k, NY, USA, 163–173. h ps://doi.o g/10.1145/3382507.3418884
e en -place: Vi ual E en , Ne he lands.
[2]
Michael Ba z, Sebas ian Kapp, Jochen Kuhn, and Daniel Sonn ag. 2021. Au o-
ma ic Recogni ion and Augmen a ion o A ended Objec s in Real- ime using
Eye T acking and a Head-moun ed Display. In ACM Symposium on Eye T ack-
ing Resea ch and Applica ions (ETRA ’21 Adjunc ). Associa ion o Compu ing
Machine y, New Yo k, NY, USA, 1–4. h ps://doi.o g/10.1145/3450341.3458766
[3]
Michael Ba z and Daniel Sonn ag. 2021. Au oma ic Visual A en ion De ec ion
o Mobile Eye T acking Using P e-T ained Compu e Vision Models and Human
Gaze. Senso s 21, 12 (Jan. 2021), 4143. h ps://doi.o g/10.3390/s21124143 Numbe :
12 Publishe : Mul idisciplina y Digi al Publishing Ins i u e.
[4]
Michael Ba z, S en S auden, and Daniel Sonn ag. 2020. Visual Sea ch Ta ge In e -
ence in Na u al In e ac ion Se ings wi h Machine Lea ning. In ACM Symposium
on Eye T acking Resea ch and Applica ions (ETRA ’20 Full Pape s), And eas Bulling,
Anke Huckau , Eak a Jain, Ralph Radach, and Daniel Weiskop (Eds.). Associa ion
o Compu ing Machine y, 1–8. h ps://doi.o g/10.1145/3379155.3391314
[5]
Roman Bedna ik, Hana V zako a, and Michal H adis. 2012. Wha do you wan
o do nex : a no el app oach o in en p edic ion in gaze-based in e ac ion. In
P oceedings o he Symposium on Eye T acking Resea ch and Applica ions (ETRA
’12). Associa ion o Compu ing Machine y, New Yo k, NY, USA, 83–90. h ps:
//doi.o g/10.1145/2168556.2168569 e en -place: San a Ba ba a, Cali o nia.
[6]
And eas Bulling, Daniel Roggen, and Ge ha d T oes e . 2011. Wha ’s in he Eyes
o Con ex -Awa eness? IEEE Pe asi e Compu ing 10, 2 (Ap il 2011), 48–57.
h ps://doi.o g/10.1109/MPRV.2010.49 Place: USA Publishe : IEEE Educa ional
Ac i i ies Depa men .
[7]
And eas Bulling, Jamie A. Wa d, Hans Gelle sen, and Ge ha d T ös e . 2009. Eye
mo emen analysis o ac i i y ecogni ion. In P oceedings o he 11 h In e na ional
Con e ence on Ubiqui ous Compu ing (UbiComp ’09). Associa ion o Compu ing
Machine y, New Yo k, NY, USA, 41–50. h ps://doi.o g/10.1145/1620545.1620552
e en -place: O lando, Flo ida, USA.
[8]
And ew T. Duchowski. 2018. Gaze-based in e ac ion: A 30 yea e ospec i e.
Compu e s & G aphics 73 (2018), 59–69. h ps://doi.o g/10.1016/j.cag.2018.04.002
[9]
Sab ina Hoppe, Tobias Loe sche , S ephanie A. Mo ey, and And eas Bulling. 2018.
Eye Mo emen s Du ing E e yday Beha io P edic Pe sonali y T ai s. F on ie s
in Human Neu oscience 12 (2018). h ps://doi.o g/10.3389/ nhum.2018.00105
[10]
Sébas ien Lallé, De eck Toke , and C is ina Cona i. 2021. Gaze-D i en Adap i e
In e en ions o Magazine-S yle Na a i e Visualiza ions. IEEE T ansac ions on
Visualiza ion and Compu e G aphics 27, 6 (2021), 2941–2952. h ps://doi.o g/10.
1109/TVCG.2019.2958540
[11]
Yan Liu, Pei-Yun Hsueh, Jenni e Lai, Mi weis Sangin, Ma c-An oine Nussli, and
Pie e Dillenbou g. 2009. Who is he expe ? Analyzing gaze da a o p edic
expe ise le el in collabo a i e applica ions. In 2009 IEEE In e na ional Con e ence
on Mul imedia and Expo. 898–901. h ps://doi.o g/10.1109/ICME.2009.5202640
[12]
Päi i Maja an a and And eas Bulling. 2014. Eye T acking and Eye-Based Hu-
man–Compu e In e ac ion. In Ad ances in Physiological Compu ing, S ephen H.
Fai clough and Kiel Gilleade (Eds.). Sp inge , London, 39–65. h ps://doi.o g/10.
1007/978-1-4471-6392-3_3
[13]
Sha on O ia , Bjö n Schulle , Philip Cohen, Daniel Sonn ag, Ge asimos Po ami-
anos, and An onio K üge (Eds.). 2019. The Handbook o Mul imodal-Mul isenso
In e aces: Language P ocessing, So wa e, Comme cializa ion, and Eme ging Di-
ec ions. Associa ion o Compu ing Machine y and Mo gan & Claypool.
h ps://doi.o g/10.1145/3233795
[14]
Sha on. O ia , Bjö n Schulle , Philip R. Cohen, Daniel Sonn ag, Ge asimos
Po amianos, and An onio K üge (Eds.). 2017. The Handbook o Mul imodal-
Mul isenso In e aces: Founda ions, Use Modeling, and Common Modali y Combi-
na ions ( olume 1 ed.). Associa ion o Compu ing Machine y and Mo gan &
Claypool, New Yo k, NY, USA. h ps://doi.o g/10.1145/3015783
[15]
Pe nilla Q a o d . 2017. Gaze-In o med Mul imodal In e ac ion. In The Handbook
o Mul imodal-Mul isenso In e aces: Founda ions, Use Modeling, and Common
Modali y Combina ions - Volume 1. Associa ion o Compu ing Machine y and
Mo gan & Claypool, 365–402. h ps://doi.o g/10.1145/3015783.3015794
[16]
Shane D. Sims and C is ina Cona i. 2020. A Neu al A chi ec u e o De ec ing
Use Con usion in Eye- acking Da a. In P oceedings o he 2020 In e na ional
Con e ence on Mul imodal In e ac ion (ICMI ’20). Associa ion o Compu ing
Machine y, New Yo k, NY, USA, 15–23. h ps://doi.o g/10.1145/3382507.3418828
e en -place: Vi ual E en , Ne he lands.
697
