Au oma ed loca ion o o o acial landma ks o
cha ac e ize ai way mo phology in anaes hesia ia deep
con olu ional neu al ne wo ks
Fe nando Ga c´ıa-Ga c´ıaa,∗, Dae-Jin Leea, F ancisco J. Mendoza-Ga c´esb, So ´ıa
I igoyen-Mi ´ob, Ma ´ıa J. Lega e a-Olaba ie ac, Susana Ga c´ıa-Gu i´e ezc,
Inmaculada A os eguid,a
aBasque Cen e o Applied Ma hema ics (BCAM).
bGaldakao-Usansolo Uni e si y Hospi al, Anaes hesia & Resusci a ion Se ice.
cGaldakao-Usansolo Uni e si y Hospi al, Resea ch Uni .
dUni e si y o he Basque Coun y (UPV/EHU), Depa men o Ma hema ics.
Abs ac
Backg ound: A eliable an icipa ion o a di icul ai way may no ably enhance
sa e y du ing anaes hesia. In cu en p ac ice, clinicians use bedside sc eenings
by manual measu emen s o pa ien s’ mo phology.
Objec i e: To de elop and e alua e algo i hms o he au oma ed ex ac ion
o o o acial landma ks, which cha ac e ize ai way mo phology.
Me hods: We de ined 27 on al + 13 la e al landma ks. We collec ed n=317
pai s o p e-su ge y pho os om pa ien s unde going gene al anaes hesia (140
emales, 177 males). As g ound u h e e ence o supe ised lea ning, land-
ma ks we e independen ly anno a ed by wo anaes hesiologis s.
We ained wo ad-hoc deep con olu ional neu al ne wo k a chi ec u es based
on Incep ionResNe V2 (IRNe ) and MobileNe V2 (MNe ), o p edic simul ane-
ously: a) whe he each landma k is isible o no (occluded, ou o ame), b) i s
2D-coo dina es (x, y). We implemen ed successi e s ages o ans e lea ning,
combined wi h da a augmen a ion. We added cus om op laye s on op o hese
ne wo ks, whose weigh s we e ully uned o ou applica ion. Pe o mance in
landma k ex ac ion was e alua ed by 10- old c oss- alida ion (CV) and com-
pa ed agains 5 s a e-o - he-a de o mable models.
Resul s: Wi h anno a o s’ consensus as he ‘gold s anda d’, ou IRNe -based
ne wo k pe o med compa ably o humans in he on al iew: median CV loss
L= 1.277 ·10−3, in e -qua ile ange (IQR) [1.001, 1.660]; e sus median 1.360,
∗Co esponding au ho . Con ac in o: Alameda de Maza edo, 14–48009 Bilbao, Bizkaia
(Basque Coun y, Spain). Telephone: +34 946 567 842.
Email add esses: [email p o ec ed] (Fe nando Ga c´ıa-Ga c´ıa), [email p o ec ed]
(Dae-Jin Lee), [email p o ec ed] (F ancisco
J. Mendoza-Ga c´es), [email p o ec ed] (So ´ıa I igoyen-Mi ´o),
[email p o ec ed] (Ma ´ıa J. Lega e a-Olaba ie a),
[email p o ec ed] (Susana Ga c´ıa-Gu i´e ez),
[email p o ec ed] (Inmaculada A os egui)
This is he accep ed manusc ip o he a icle ha appea ed in inal o m in Compu e Me hods and P og ams in Biomedicine
232 : (2023) // A icle ID 107428, which has been published in inal o m a h ps://doi.o g/10.1016/j.cmpb.2023.107428. © 2023
Else ie unde CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/)
IQR [1.172, 1.651], and median 1.352, IQR [1.172, 1.619], o each anno a o
agains consensus, espec i ely. MNe yielded sligh ly wo se esul s: median
1.471, IQR [1.139, 1.982].
In he la e al iew, bo h ne wo ks a ained pe o mances s a is ically poo e
han humans: median CV loss L= 2.141 ·10−3, IQR [1.676, 2.915], and median
2.611, IQR [1.898, 3.535], espec i ely; e sus median 1.507, IQR [1.188, 1.988],
and median 1.442, IQR [1.147, 2.010] o bo h anno a o s. Howe e , s anda d-
ized e ec sizes in CV loss we e small: 0.0322 and 0.0235 (non-signi ican ) o
IRNe , 0.1431 and 0.1518 (p <0.05) o MNe ; he e o e quan i a i ely simila
o humans.
The bes pe o ming s a e-o - he-a model (a de o mable egula ized Supe -
ised Descen Me hod, SDM) beha ed compa ably o ou DCNNs in he on al
scena io, bu no o iously wo se in he la e al iew.
Conclusions: We success ully ained wo DCNN models o he ecogni ion
o 27+13 o o acial landma ks pe aining o he ai way. Using ans e lea ning
and da a augmen a ion, hey we e able o gene alize wi hou o e i ing, each-
ing expe -like pe o mances in CV. Ou IRNe -based me hodology achie ed a
sa is ac o y iden i ica ion and loca ion o landma ks: pa icula ly in he on al
iew, a he le el o anaes hesiologis s. In he la e al iew, i s pe o mance
decayed, al hough wi h a non-signi ican e ec size. Independen au ho s had
also epo ed lowe la e al pe o mances; as ce ain landma ks may no be clea
salien poin s, e en o a ained human eye.
Keywo ds: Di icul ai way, Anaes hesia, Deep lea ning, T ans e lea ning,
Facial landma ks.
1. In oduc ion
1.1. Medical mo i a ion
Clinical guidelines p o ide anaes hesiologis s wi h aluable, e idence-based
counselling o he managemen o di icul espi a o y ai ways [1]. Compli-
ca ions –despi e in equen – a e an impo an sou ce o lesions: inadequa e5
oxygena ion migh cause se e e b ain damage o e en dea h [2]. In su gical
planning, when he ai way is an icipa ed as di icul , speci ic sa e y equipmen
and pe sonnel a e equi ed.
Howe e , he p ognosis o di icul ai ways emains a challenging ask, e en
o expe ienced anaes hesiologis s: up o 10% o acheal in uba ions become10
di icul [3], app oxima ely 93% o which we e unan icipa ed [4].
In clinical p ac ice, bedside sc eenings assess ai way di icul y. These exam-
ine he o opha yngeal mo phology: e.g. hy o-/s e no-men al dis ance (TMD,
SMD), mou h opening-in e inciso gap (MO-IIG), o he modi ied Mallampa i
es (MMT) [5]. Howe e , such sc eenings may be ime-consuming and p one15
o epea abili y/ ep oducibili y issues (e.g. due o manual measu emen s), and
sys ema ic e iews epo limi ed disce nmen capabili ies [6, 7].
2
In his con ex , anaes hesiologis s a e gaining g owing in e es in a i icial
in elligence (AI) and machine lea ning (ML) [8, 9, 10].
1.2. Rela ed wo ks20
In li e a u e, one could dis inguish h ee main amilies o AI-ML me hods
concei ed o assis clinicians in p edic ing di icul in uba ion o he ai way. The
i s ype o app oaches consis s o eeding ML algo i hms wi h s uc u ed in o -
ma ion abou he pa ien ’s demog aphics (mos o en: sex, age and body mass
index), along wi h one o mo e mo phological cha ac e iza ions o he ai way25
(e.g. TMD, SMD, MO-IIG o neck ci cum e ence) – hese la e magni udes ha -
ing been measu ed manually, as in he bedside sc eenings om cu en clinical
p ac ise. Wo ks in his ca ego y include Yan e al. [11] (who ained a Suppo
Vec o Machine), Lange on e al. [12] (who employed T ee Boos ing), Kim e
al. [13] (5 ML algo i hms, among which Random Fo es s pe o med bes ), Ya-30
manaka e al. [14] (wi h an ensemble o 7 ML algo i hms), o Zhou e al. [15]
(10 algo i hms, wi h G adien Boos ing as hei bes ).
These s a egies may help o o e come he limi a ions o uni a ia e analyses
(like he bedside sc eenings) and o clinical sco es and logis ic eg essions de i ed
om a ew p edic o a iables [16, 17, 18]. None heless, hey may s ill su e om35
ep oducibili y issues, since hey ely on inpu in o ma ion abou he ai way
mo phology which needs o be measu ed by hand.
A second amily o app oaches comp ises AI-ML compu e ision ools which
analyse p eope a i e pa ien images. Ta ola a e al. [19] de eloped 11 Con o-
lu ional Neu al Ne wo ks (CNN), each one specialized in a di e en acial sub-40
egion. Wi h hese CNNs, he au ho s gene a ed ea u e ensembles which we e
subsequen ly ed o a Mul iple Ins ance Lea ning model. In addi ion, Hayasaka
e al. [20] p oposed a me hodology based on pho os o 16 di e en poses pe pa-
ien (combina ions o on al/la e al iews, supine/si ing posi ion, open/closed
mou h, head ben backwa ds o no ), aining espec i e CNNs o classi y di i-45
cul ai ways. Fu he mo e, Cho e al. [21] used CNNs wi h la e al ce ical spine
X- ay images.
Compa ed o he i s ype o p oposals, hese image-based me hodologies
may be able o exploi iche isual in o ma ion, impossible o cap u e alone
by he adi ional mo phological measu es – pe haps a he cos o no ha ing50
access o he pa ien ’s demog aphic da a. Howe e , hese algo i hms end o
beha e as opaque sys ems lacking human in e p e abili y, which may hinde
hei adop ion by some clinical p ac i ione s.
The hi d ype o me hod can be hough o as a wo-s age p ocess: i s ,
p eope a i e images a e analyzed o iden i y and loca e ele an landma ks55
(i.e. salien key poin s in subjec s’ ana omy); second, mo phological measu e-
men s a e de i ed om hem and ed o ML classi ie s, ained o p edic in-
uba ion di icul y. To ou knowledge, Suzuki e al. [22] we e he i s who
p oposed an algo i hm wi h such a s a egy: he au ho s de ined 84 landma ks
om a on al iew, alongside 39 la e al landma ks. Based on n=32 pa ien s60
3
(16 emales) wi h easy in uba ion, plus n=41 (18 emales) wi h di icul in uba-
ion, hey used ‘mo phing’ echniques o landma k loca ion. A e wa ds, i e
mo phological magni udes we e ex ac ed.
Con e sely, Conno & Seagal [23] op ed o using h ee pho og aphs: one
on al iew, plus le and igh p o iles. The au ho s en olled n=80 subjec s (all65
males), using semi-au oma ic p op ie a y so wa e o acial s uc u e analysis
by ‘eigen ace’ p ojec ion. Based on he landma ks, 61 acial p opo ions we e
compu ed and ed o logis ic eg essions.
Cuende e al. [24] buil a cus om ‘pho o boo h’-like equipmen , wi h wo
high-de ini ion webcams placed o hogonally, 30–40 cm dis an om he pa-70
ien . The au ho s cap u ed ou ypes o pho og aphs: h ee on al iews
(99 landma ks each) and a la e al p o ile (52 landma ks). Thei s udy en olled
n=970 pa ien s (482 emales) bu comp ised 406 se s o pho os manually an-
no a ed o on al de ec ion, alongside 134 o la e al de ec ion. The au ho s
add essed he ask o landma k loca ion by means o a de o mable appea ance75
model, based on scale-in a ian ea u e ans o m (SIFT) ea u es compu ed
om pa ches a ound each landma k. Thei mo phological measu es, alongside
a ious ex u e ea u es, ed a Random Fo es classi ie .
1.3. Ra ionale & Objec i e
In his wo k, we aim a con ibu ing o he cha ac e iza ion o ai way mo -80
phology, in he con ex o p edic ing di icul in uba ions o anaes hesia. Among
he h ee ypes o app oaches ou lined in Sec ion 1.2, we op ed o he la e
wo-s age s a egy. In b ie : au oma ed landma k loca ion o ob ain mo pholog-
ical measu es, ollowed by ML-enabled p edic ion o in uba ion di icul y ( he
la e s age alling beyond he scope o his pape – see Sec ion 5.3). In ou 85
iew, such a wo-s age me hodology a ains a desi able balance be ween in e -
p e abili y o human expe s and he au oma ic, epea able and ep oducible
ex ac ion o ele an in o ma ion om p eope a i e examina ions ia imaging
echniques.
Me hodologically, wi h espec o he ela ed wo ks ound in li e a u e, he e90
we p opose wo models which exploi he ecen algo i hmic ad ances in he
ield o compu e ision: whe e deep con olu ional neu al ne wo ks (DCNNs)
ha e become dominan , hanks o hei la ge exp essi e powe and high le els
o pe o mance achie ed ac oss many di e en asks.
To some ex en , ou wo k was also inspi ed by Cuende e al. [24]. Howe e ,95
he e we made a ious di e en design choices: o example, hei ou -image ac-
quisi ion p o ocol using a ‘pho o boo h’-like equipmen wi h wo high-de ini ion
came as may no be easible o in eg a e in o p e-su gical examina ion wo k lows
in p ac ice. Ins ead, we op ed o a single on al pho og aph cap u ed wi h a
sma phone, wi h pa ien s’ mou h ully open and ongue ou –hence allowing100
o he assessmen o common bedside sc eening magni udes (MO-IIG, MMT)–;
as well as o a single la e al iew, wi h head in e ical ex ension – o TMD
and SMD. Ou anaes hesiology eam de ined wo cus om se s o landma ks (one
on al, one la e al), ca e ully ailo ed o desc ibe he o o acial ana omy pe -
aining o he espi a o y ai way. Fu he mo e, we ob ained wo independen 105
4
se s o high-quali y anno a ions by independen anaes hesiologis s, om which
a consensus g ound u h was de i ed. These wo sou ces o anno a ion allowed
us o s udy in e -human disc epancies: as a ‘gold s anda d’ o he pe o mance
o ou algo i hms, and as a quan i a i e measu e o he in insic complexi y o
his landma k iden i ica ion and loca ion p oblem o he ained human eye.110
2. Backg ound
2.1. ‘Gene al-pu pose’ acial landma ks
In his sec ion, we b ie ly o e iew he s a e-o - he-a in acial landma k
ecogni ion. Fo a comp ehensi e li e a u e e iew, he in e es ed eade may
conside Wu & Ji [25]. No e ha , al hough he models o ‘gene al-pu pose’ a-115
cial landma ks canno be applied di ec ly o ou anaes hesia scena io (gi en he
subs an ial isual di e ences in head poses, pho o aming, ana omical con en ,
e c. and e en in he de ini ion o landma ks hemsel es – see Sec ion 3), a ious
me hodological p inciples can indeed be sha ed.
The p oblem o iden i ying and loca ing landma ks in human ace images120
has igge ed majo esea ch e o s in he compu e ision communi y, as i
can be applied o a ious con ex s: subjec iden i ica ion [26, 27], ecogni ion
o acial exp essions [28, 29], sen imen analysis [30], de ec ion o gene ic syn-
d omes [31], e c. Ac i e appea ance models (AAM) [32] and cons ained local
models (CLM) [33] we e among he i s me hods o achie e sa is ac o y gene -125
aliza ion capabili ies, e en wi h ew aining images [34]. Wi h he a ailabili y
o la ge acial da ase s and imp o ed desc ip i e ea u es (such as his og am
o o ien ed g adien s – HOG, o scale-in a ian ea u e ans o ms – SIFT) [35],
disc imina i e models like cascade eg ession me hods (in pa icula , supe ised
descen me hods – SDM) [36] became he echnique o e e ence.130
Cu en ly, Deep Lea ning (DL) has achie ed majo ele ance and ema kable
pe o mance. This end o g ound-b eaking algo i hmic and echnical ad ances
by DL is common o many ields in compu e ision, also o he subdomain o
acial landma ks [25]. In pa icula , a ious DCNN a chi ec u es ha e been ap-
plied success ully o acial landma k localiza ion [37, 38, 39], mo i a ing us o135
explo e his DCNN app oach in he con ex o ai way assessmen and anaes he-
sia.
2.2. Deep Lea ning & T ans e Lea ning
Al hough DL has consis en ly ou pe o med o he compu e ision ech-
niques in a wide a ie y o applica ions, including biomedical imaging [40, 41];140
i s ‘da a hunge ’ phenomenon [42] poses an impo an challenge in scena ios
whe e he a ailabili y o da a (and/o o g ound u h) is sca ce o cos ly o ex-
ac [43] – as in ou case he e. T ans e Lea ning (TL) may alle ia e such ‘da a
hunge ’ issue: a DL ne wo k’s millions o neu on weigh s may no need o be i -
ed om sc a ch, bu ins ead euse knowledge ob ained om o he asks [44, 45].145
In his sense, empi ical s udies ha e shown ha knowledge ans e abili y in TL
– ypically ia weigh sha ing– is mo e a ou able he close he o iginal and
5
a ge domains a e, bu low-le el gene ic ea u es lea n om dis an asks can
s ill be be e han andom weigh ini ializa ion, and a boos o he ne wo k’s
gene aliza ion p ope ies [46].150
He e we explo e his TL s a egy on he basis o wo public la ge-scale
da ase s o ‘gene al-pu pose’ acial landma k ecogni ion.
3. Ma e ials and me hods
In his wo k, we p opose wo ad-hoc DCNNs o iden i y a se o landma ks,
which ou anaes hesiologis s es ablished as ele an o cha ac e ize pa ien s’155
o o acial mo phology when assessing di icul ai ways o acheal in uba ion.
Fo each landma k, he DCNNs will p edic : a) i s isibili y (occlusions may
occu ); as well as b) i s 2D coo dina es (x, y) in he image.
3.1. Da a
3.1.1. T ans e Lea ning da ase s160
Ou s a ing poin consis ed o DCNNs p e- ained on ImageNe : a la ge-
scale collec ion wi h >3.2 million images [47, 48]. This da ase is an essen ial
e e ence o compu e ision applica ions, and almos all DL p og amming
amewo ks include ne wo ks p e- i ed on i : hence, al eady sui able o TL.
In addi ion, we used wo o he public da ase s o ‘gene al-pu pose’ acial165
landma k ecogni ion: ‘CelebFaces A ibu es Da ase ’ (CelebA) and ‘Anno a ed
Facial Landma ks in he Wild’ (AFLW ). To he bes o ou knowledge, we a e
unawa e o he exis ence o public da ase s pe aining o he assessmen o
espi a o y ai ways o anaes hesia, which would indeed be a close domain o
ou scena io. Ne e heless, based on he p inciples in oduced in Sec ion 2.2 [44,170
45, 46], we deemed he isual con en s o hese ‘gene al-pu pose’ acial da ase s
o be simila enough o be p omising sou ces o knowledge ex ac ion ia TL.
CelebA [49] is a non-comme cial, la ge-scale da ase con aining 202,599 hu-
man ace images (58.3% emales) om 10,177 dis inc subjec s. Faces we e
p e-aligned and c opped o cons an size ( ec angles wi h 218×178 pixels). Fo 175
each ace, he g ound u h 2D coo dina es o 5 acial landma ks (eye cen es,
nose ip, and mou h co ne s) a e supplied. The da ase encompasses a no able
a ie y in head pose and pho o pe spec i es, as well as o he a ibu es (age,
sex, e hnici y, glasses, e c.).
AFLW [50] is a non-comme cial, esea ch-only da ase which comp ises180
24,384 aces (58.8% emales) om 21,121 dis inc images. The au ho s p o ide
each ace’s squa e bounding box, alongside he isibili y and 2D coo dina es o
21 landma ks (b ows, eyes, ea s, nose, mou h, chin). AFLW is epo ed o con-
ain subs an ial a ie y in e ms o sex and e hnici y. Mos no ably, app ox. 66%
o images co espond o non- on al poses: o he au ho s’ knowledge, a highe 185
a io han in any o he public acial landma ks da ase .
6
3.1.2. Di Ai W da ase
We conduc ed an obse a ional, p ospec i e, coho s udy a he Uni e si y
Hospi als o Galdakao-Usansolo and Basu o (Biscay, Basque Coun y, Spain),
app o ed by he ‘E hics Commi ee o Resea ch wi h medica ion’ o he Basque190
Coun y (CEIm-E). We en olled pa icipan s who ga e in o med w i en consen
in acco dance wi h he Decla a ion o Helsinki. Inclusion c i e ia we e: adul -
hood (age≥18) and unde going acheal in uba ion ia di ec la yngoscopy o
gene al anaes hesia, ega dless o whe he he in e en ion was p og ammed o
eme gency. Exclusion c i e ia we e: o ola yngeal diseases wi h a ec ed s uc-195
u es, adio he apy, any p e ious su ge y o ana omical al e a ions in he ai way,
obs e ic pa ien s and/o a known di icul ai way equi ing ibe op ic in uba-
ion.
F om Ma ch 2018 o Janua y 2020, n=317 pai s o p eope a i e pho os we e
collec ed: one on al, and one la e al. In he on al iew, pa ien s we e in-200
s uc ed o ully open hei mou hs, and o s ick hei ongues ou ; hence acil-
i a ing a simul aneous judgmen o MMT and MO-IIG. In he la e al iew, he
pose was wi h he head in e ical ex ension: he s anda d p ocedu e o measu e
TMD and SMD. Sma phones wi h gene al-pu pose came as we e used o ake
he pho og aphs, and a cue ca d was added o he scene (ci cle wi h 25 mm205
diame e ), as a e e ence o physical dimensions.
We eco ded each pa ien ’s demog aphics (sex, age, weigh , heigh , body
mass index), MMT and ope a i e ou comes, including: he numbe o in uba ion
a emp s and/o ex a de ices, du a ion o he manoeu e, Co mack-Lehane
sco e o di ec la yngoscopy [51], e c. Wi h hese, we de e mined objec i ely210
he in uba ion di icul y, acco ding o he IDS [52] and SFAR c i e ia [53] (Ta-
ble 1). In ou coho , women we e unde - ep esen ed (signi ican ly al hough
by a na ow ma gin, p=0.0430 – Table 1): 177 males e sus 140 emales; a
p opo ion o 44.16%, wi h a 95% con idence in e al [38.63%–49.82%].
Ou anaes hesiologis s de ined wo se s o landma ks ele an o o o acial215
mo phology (Figu e 1): N=27 poin s o he on al iew, alongside N=13 o
he la e al iew. Two anaes hesiologis s anno a ed hese landma ks manually
(Sec ion 3.2.1).
3.2. Landma k loca ion
Le ( i, xi, yi) be he iple which cha ac e izes g ound u h anno a ions220
o he i- h ou o Nlandma ks; whe e N=5 o CelebA,N=21 o AFLW,
whe eas N=27 o Di Ai W[F on al], and N=13 o Di Ai W[La e al]. He e
i ep esen s a landma k’s isibili y (i could become occluded by skin issue,
bea d, clo hes, e c., o e en le ou o he pho o ame); whe eas (xi, yi) deno e
i s 2D posi ion: i.e. ho izon al and e ical coo dina es, being (0,0) he op-le 225
o he image and (1,1) i s bo om- igh co ne , by con en ion.
The e o e, he ask o add ess he e can be desc ibed as a combina ion o
mul iple bina y classi ica ions (N isibili ies i) and 2N eg essions o coo -
dina es (xi, yi). To compa e iple s {(ˆ i,ˆxi,ˆyi)}N
i=1 p edic ed by he DCNNs
7
Table 1: Cha ac e is ics o ou Di Ai W coho
All By sex
Females Males Binomial
n=317 n=140 n=177 0.0430
Mann-Whi ney
Age 63.0 (51.8, 72.0) 58.0 (46.0, 69.3) 65.0 (54.0, 73.0) <0.001
Weigh [kg] 74.0 (64.8, 88.0) 68.0 (58.5, 81.3) 78.0 (70.0, 91.3) <0.001
Heigh [m] 1.66 (1.60, 1.73) 1.60 (1.56, 1.65) 1.71 (1.66, 1.76) <0.001
BMI [kg m−2] 26.71 (23.95, 30.65) 26.13 (23.02, 31.23) 26.94 (24.22, 30.37) 0.4463
Mallampa i χ2
G ade I 152 (47.95%) 72 (51.43%) 80 (45.20%)
0.5842
G ade II 108 (34.07%) 47 (33.57%) 61 (34.46%)
G ade III 48 (15.14%) 18 (12.86%) 30 (16.95%)
G ade IV 9 (2.84%) 3 (2.14%) 6 (3.39%)
Co mack-Lehane χ2
Classes I–II 300 (94.64%) 135 (96.43%) 165 (93.22%) 0.2080
Classes III–IV 17 (5.36%) 5 (3.57%) 12 (6.78%)
IDS di icul y χ2
No 301 (94.95%) 136 (97.14%) 165 (93.22%) 0.1132
Yes 16 (5.05%) 4 (2.86%) 12 (6.78%)
SFAR di icul y χ2
No 302 (95.27%) 137 (97.86%) 165 (93.22%) 0.0535
Yes 15 (4.73%) 3 (2.14%) 12 (6.78%)
Values a e shown as median (in e -qua ile ange), o numbe (pe cen age) whe e app op ia e.
The igh -mos column con ains p- alues o popula ion di e ences wi h espec o sex, and he s a is ical es used.
BMI: body-mass index, IDS: In uba ion Di icul y Scale
SFAR: Soci´e ´e F an¸caise d’Anes h´esie e de R´eanima ion [F ench Socie y o Anaes hesia and Resusci a ion].
8
Figu e 1: De ini ion o o o acial landma ks o cha ac e ize ai way mo phology in he con ex
o anaes hesia. Le panel – F on al iew: Landma ks F01–F27 (nose, lips, ee h, ongue,
chin, mandible, neck, hy oid ca ilage, s e nal manub ium). Righ panel – La e al iew:
Landma ks L01–L13 (nose, lips, chin, mandible, hy oid ca ilage, s e nal manub ium, nape,
occipu ).
agains g ound u h {( i, xi, yi)}N
i=1, we de ined he ollowing loss unc ion:
L(·|γ) = γ1
N
N
X
i=1
BCE( i,ˆ i)+ 1
PN
i=1 i
N
X
i=1
i(xi−ˆxi)2+AR2(yi−ˆyi)2(1)
being BCE he bina y c oss-en opy loss:
BCE( , ˆ ) = − log(ˆ )−(1 − ) log(1 −ˆ ) (2)
and AR he aspec a io (heigh /wid h) o he o iginal image.
The i s summa ion e m in Eq. [1] penalizes he disag eemen be ween p e-
dic ed ˆ iand g ound u h i isibili ies; whe eas he second summa ion e m
co esponds o he mean squa ed poin - o-poin e o in coo dina es. When-230
e e a ce ain landma k was se as non- isible in he g ound u h anno a ions
( i=0), he mul iplica i e e m by iin he second summa ion o Eq. [1] nulli-
ied any con ibu ion o loss wi h ega d o he ue/es ima ed loca ions (xi, yi)
o (ˆxi,ˆyi). Hence, wi h i=0, he es ima ed (ˆxi,ˆyi) become i ele an o he
o al loss and do no con ibu e ei he o compu ing he g adien o L(·) in he235
aining o he ne wo k by backp opaga ion. Besides, he weigh pa ame e γ
balances he con ibu ion o each e m owa ds he o e all loss L(·|γ).
Eq. [1] is simila o Ranjan e al.’s p oposal [38]. Howe e , in ou sec-
ond e m, we a e age only by he numbe o isible landma ks PN
i=1 i,
9
espec i e HOG-SVM de ec o s [66, 67] o ou on al and la e al o o acial
s uc u es in Di Ai W. In u n, hese de ec o s we e used wi h he images in
he es s age, o ind sui able bounding boxes o hem, om which o s a
i ing he ained de o mable models.385
Likewise in Sec ion 3.3.1, we e alua ed hese models’ pe o mance ia 10-
old CV, again s a i ied by sex. In he case o SDM, we conduc ed p elimina y
CV hype pa ame e uning expe imen s o de e mine egula iza ion s eng h λ,
wi h alues om 10−3 o 109in ac o s o 10. Since such a uning p ocedu e
epea edly selec ed λ=106as op imal, we decided o ix λalways o ha alue.390
Figu e 5: Schema ic lowcha o he a ious s ages o aining he de o mable models.
4. Resul s
4.1. T ans e Lea ning om he ‘gene al-pu pose’ acial da ase s
S a ing wi h p e- i ed weigh s om ImageNe , we applied successi e TL
s ages in o de o inc easing isual simila i y wi h ou a ge Di Ai W do-
main [46] (also in dec easing da ase size): i s on CelebA, hen on AFLW395
(Figu e 4).
As depic ed in Figu e 6, ou ad-hoc a chi ec u es wi h he Incep ionRes-
Ne V2 co e (deno ed onwa ds IRNe Ai W – uppe ow) and MobileNe V2
co e (MNe Ai W – lowe ow), bo h lea n sa is ac o ily: con e ging success-
ully wi hin he es ablished aining epochs and achie ing no iceable dec eases400
in loss L, bo h o CelebA da a (le column in Figu e 6) and o AFLW ( igh
column). Losses om he es se s (plo ed in o ange in Figu e 6) we e epea -
edly lowe han om he aining se s (blue), due o he absence o augmen a ion
ans o ma ions du ing he es ing s age. The e was a consis en end by IR-
Ne Ai W o yield lowe es losses han MNe Ai W o he same da ase and405
he same aining epoch. On he o he hand, he la ge size –hence, inc eased
exp essi e powe – o IRNe Ai W may explain he sudden peaks obse ed in
aining loss L, om which i was none heless able o eco e wi hin a single
epoch.
Figu e 7 shows he wo-dimensional UMAP p ojec ion/embedding [68] o 410
he neu on ac i a ions (a e aged pe con olu ional channel), a di e en ne wo k
dep hs: aw inpu pixel alues ( i s ow), as well as he ac i a ion ou pu by he
sec ion o he ne wo ks ozen a each TL s age (50% o 100% o he ‘s anda d’
DCNN a chi ec u es, Figu e 4). In hese plo s, bo h IRNe Ai W (le column)
and MNe Ai W ( igh ) showed he empi ically expec ed beha iou [46], by415
16
Figu e 6: T ans e lea ning (TL), in e media e esul s – Con e gence o he aining and
es losses L(·|γop ) ac oss epochs, plo ed in loga i hmic scale [o dina es]. A e ob aining
he al eady p e- ained weigh s om ImageNe , he i s subsequen TL s age was ca ied
ou on he CelebA da ase (le column), and hen on AFLW ( igh column). Uppe ow:
IRNe Ai W a chi ec u e, lowe ow: MNe Ai W.
which he bo om laye s ex ac gene ic ea u es: aw pixel embeddings we e
no sepa able (panels a–b), bu hei compu ed ea u es we e (c–d onwa ds); so
he neu ons in he uppe laye s could ocus on specializa ion.
4.2. Pe o mance o landma k loca ion in Di Ai W
Table 2 and Figu e 8 summa ize he losses L(·|γop ) incu ed by each human420
agains consensus [‘gold s anda d’], as well as by he DCNNs – compu ed om
iple s (ˆ i,ˆxi,ˆyi) ou pu in he alida ion spli o ou 10- old CV.
Figu e 9 displays he DCNN esul s o eigh pa ien s (4 on al, 4 la e al; 4
by IRNe Ai W, 4 by MNe Ai W ). Fo he sake o ai epo ing and illus a-
ion o he ne wo ks’ pe o mance, hese examples we e no selec ed among he425
mos a ou able cases (i.e. lowes losses), bu ins ead a ound he o e all median
CV loss L(·|γop ). Quali a i ely, he o e all esul s we e sa is ac o y o ou
anaes hesiology eam h ough isual explo a ion. None heless, in he on al
iew, he s e nal manub ium (F25–F27) was o en he mos di icul ana omical
s uc u e o cap u e –as i was also o he human anno a o s–. In his e-430
ga d, he la e al iew should be be e sui ed o iden i y he s e nal manub ium
(landma k L11): he la e al and on al pho os complemen each o he o an
enhanced desc ip ion o pa ien s’ ana omy. The base o he neck (F23–F24)
17
(a) Raw RGB pixels: 299×299×3, as in he
Incep ionResNe V2 inpu o ma .
(b) Raw RGB pixels: 224×224×3, as in he
MobileNe V2 inpu o ma .
(c) IRNe Ai W : Uppe mos ozen laye , a e
TL om ImageNe .
(d) MNe Ai W : Uppe mos ozen laye , a e
TL om ImageNe .
(e) IRNe Ai W : Uppe mos ozen laye a e
TL om CelebA.
( ) MNe Ai W : Uppe mos ozen laye a e
TL om CelebA.
(g) IRNe Ai W : Uppe mos ozen laye a e
TL om AFLW.
(h) MNe Ai W : Uppe mos ozen laye a e
TL om AFLW.
Figu e 7: Two-dimensional UMAP p ojec ion o he a e age neu on ac i a ions, a di e en
ne wo k dep hs and TL s ages, o bo h DCNNs: IRNe Ai W (le column) and MNe Ai W
( igh column). In pu ple, 2,000 andomly sampled images om ImageNe ( wo om each o
he C=1000 exis ing classes); in o ange, 2,000 andomly sampled images om CelebA’s es
se (size 20,000); in blue, 2,000 andomly sampled om AFLW ’s (size 2,500); in ed and g een,
each o he n=317 images om ou Di Ai W da ase ( on al and la e al iews, espec i ely).
18
was some imes also challenging, o o e weigh pa ien s wi h a hick neck o o
hose si ua ions in which, gi en he ela i e posi ion o he came a wi h espec 435
o he pa ien (dis ance, inclina ion), he pe spec i e iew o he neck could
become obs uc ed by he mandible.
In he la e al iew, he nape, occipu and mandible co ne and hy oid ca -
ilage (L12–L13, L08–L09, L10) we e he landma ks wi h in e -anno a ion dis-
c epancies abo e he a e age. Howe e , some d i in he a ea a ound he mou h440
and he chin could also be encoun e ed some imes, pa icula ly when he image
backg ound had a ied isual con en , ins ead o being la .
IRNe Ai W signi ican ly ou pe o med MNe Ai W in he on al and la -
e al iews (Table 2, Figu e 8), bo h o he en i e coho and disagg ega -
ing by sex. Only MNe Ai W[F on al] exhibi ed s a is ically di e en pe o -445
mances wi h espec o sex: pe o ming be e o women –despi e hei ce ain
unde ep esen a ion– han o men ( o u he de ails, see Sec ion 4.3).
Figu e 10 depic s ou di ec compa isons (2 women, 2 men) be ween IR-
Ne Ai W and MNe Ai W, wi h he ne wo ks being applied o exac ly he
same inpu images. Quali a i ely, bo h DCNNs a ained simila localiza ions;450
quan i a i ely, IRNe Ai W ou pe o med MNe Ai W, again as shown by Ta-
ble 2 and Figu e 8. In addi ion, he e we can obse e e y simila beha iou s
wi h he speci ic landma ks o hose epo ed in Figu e 9.
Figu e 8: Cumula i e E o Dis ibu ions (CED) o he CV losses L(·|γop ) wi h espec o
consensus in Di Ai W, in landma k iden i ica ion and loca ion, as de ined in Eq. [1]: expe
human anno a o s and ou DCNNs. Le panel, on al iew scena io; igh panel, la e al
iew. [Abscissae in loga i hmic scale].
Fu he mo e, we disagg ega ed CV losses by landma k and by con ibu ion:
i.e. whe he om disc epancy i∼ˆ i(BCE e ms in Eq. [1]), o om squa ed455
e o s in (xi, yi)∼(ˆxi,ˆyi) coo dina e loca ions, whene e posi i e g ound u h
landma k isibili y ( i=1). In he case o occlusions ( i=0), such con ibu ion
o he o e all loss is null (Eq. [1]).
19
Table 2: Losses L(·|γop ) in landma k iden i ica ion and loca ion o ou Di Ai W da ase . In he uppe pa , losses incu ed by each human
anno a o , when compa ed agains consensus. On he lowe pa , CV losses by each DCNN a chi ec u e.
Loss L(·|γop )
L(·|γop )
L(·|γop ) [10−3]F on al iew La e al iew
Humans
Anno #1 s.
Consensus LA1|C
All 1.360 (1.172, 1.651) 1.507 (1.188, 1.988)
Females 1.314 (1.154, 1.528) p=0.0049 1.603 (1.270, 2.011) p=0.0364
Males 1.424 (1.201, 1.732) 1.467 (1.117, 1.948)
Anno #2 s.
Consensus LA2|C
All 1.352 (1.172, 1.619) 1.442 (1.147, 2.010)
Females 1.296 (1.140, 1.504) p=0.0017 1.516 (1.205, 2.011) p=0.1906
Males 1.412 (1.227, 1.686) 1.419 (1.110, 2.008)
In e -anno a o
di e ence (p- alues)
All 0.0135 <0.001
Females 0.0061 <0.001
Males 0.4095 0.4458
DCNNs
IRNe Ai W s.
Consensus LIRNe |C
All 1.277 (1.001, 1.660) 2.141 (1.676, 2.915)
Females 1.300 (0.986, 1.586) p=0.7043 2.080 (1.754, 2.887) p=0.9297
Males 1.266 (1.035, 1.696) 2.158 (1.651, 2.953)
MNe Ai W s.
Consensus LMNe |C
All 1.471 (1.139, 1.982) 2.611 (1.898, 3.535)
Females 1.391 (1.104, 1.803) p=0.0229 2.752 (1.925, 3.661) p=0.3713
Males 1.568 (1.210, 2.094) 2.561 (1.877, 3.455)
In e -ne wo k
di e ence (p- alues)
All <0.001 <0.001
Females <0.001 <0.001
Males <0.001 <0.001
Values a e shown as median (IQR: in e -qua ile ange).
Popula ion di e ences wi h espec o sex we e analysed ia Mann-Whi ney wo-sided U es s.
In e -anno a o In e -ne wo k di e ences (i.e. bo om ows) we e analysed ia pai ed Wilcoxon signed- ank es s.
20
Figu e 9: Resul s o eigh example pa ien s in Di Ai W : ou on al (le ), ou la e al ( igh ). DCNN landma k ou pu s a e depic ed as o e lay on
hei co esponding inpu image. These 8 indi idual cases co e in e media e pe o mances, wi h CV losses app oxima ely a ound he o e all median
CV loss L(·|γop ).
21
Figu e 10: Resul s o ou comple e pa ien cases in Di Ai W : on al iew (uppe ow), and la e al iew (lowe ow). Di ec compa ison be ween he
ou pu s by IRNe Ai W (blue) and by MNe Ai W ( ed), agains consensus as g ound u h e e ence (g een). These examples illus a e g aphically
he supe io a e age pe o mance by IRNe Ai W o e MNe Ai W.
22
Figu e 11: CV losses o Di Ai W, disagg ega ed by landma k and by con ibu ion e m. F on al (up) and la e al (down) scena ios. E o s due
o disc epancies in coo dina e es ima ions (‘coo ds’ in he legend) we e compu ed exclusi ely o hose cases wi h posi i e g ound u h landma k
isibili y i=1, as o he wise ( i=0) such con ibu ion e m becomes i ele an o L. The ed dash-do ed line ma ks he co esponding o e all median
loss L(·|γop ). Fo isual cla i y in he diag ams, ou lie s we e no plo ed he e.
23
Losses ela ed o ˆ iwe e negligible, excep o speci ic landma ks (Figu e 11).
F01 (nose ip) was o en le ou o he pho o ame, and he ne wo ks ended o460
misjudge i s isibili y. Ne e heless, he loss e m o F01 loca ion emained in
anges simila o o he landma ks. The ˆ i ecogni ion o F22–F27 also exhibi ed
a ce ain deg ee o inaccu acy. This may be explained, a leas in pa , due o
occlusions and di icul isual dis inguishabili y e en o humans ( a , clo hes,
e c.). Fo ins ance, Figu e 2 illus a es some common cases wi h non-negligible465
disag eemen be ween anno a o s. L01, L02 and L13 also su e ed om ou -
o - ame issues. Dis inguishing L10 ( hy oid ca ilage) was a challenge o he
human anno a o s, and also o he DCNNs.
4.3. Compa isons e sus human pe o mance in Di Ai W
We compa ed he pe o mances achie ed by ou DCNNs (in e ms o CV470
losses Lagains consensus), wi h espec o he losses incu ed by he human
anno a o s. We ca ied ou a epea ed-measu es, wo-way ANOVA analysis
wi h ac o s: anno a o ( ou le els – wo humans, wo DCNNs) and sex. These
analyses we e pe o med wi h he s a is ical so wa e R, and lib a ies lme4 [69],
lme Tes [70].475
Table 3 and Figu e 12 show ha he main e ec o anno a o is always signi -
ican (p < 10−12), poin ing ou di e ences ac oss a leas some o he ou . Sex
is signi ican in he on al iew (p=0.0169), bu no in he la e al (p=0.8611).
Such signi icance o ac o sex is a ibu able pa ly o MNe Ai W, bu im-
po an ly, also o in e -human di e ences (see Figu e 12a): anno a o s incu ed480
la ge losses wi h men – a he han wi h women, despi e being unde ep e-
sen ed. In bo h cases, he in e ac ion sex:anno is no signi ican .
F on al iew SS MS DoFnum DoFdenom F-s a is ic p- alue
sex 1.03 1.03 1 315 5.77 0.0169
anno 10.41 3.47 3 945 19.50 <1e-12
sex:anno 1.08 0.36 3 945 2.02 0.1092
La e al iew SS MS DoFnum DoFdenom F-s a is ic p- alue
sex 0.04 0.04 1 315 0.03 0.8611
anno 389.68 129.89 3 945 107.76 <1e-12
sex:anno 2.63 0.88 3 945 0.73 0.5350
Table 3: ANOVA summa y able o CV losses o ou Di Ai W da ase , in he on al and
la e al iews – SS: sum o squa es; MS: mean squa e; DoFnum: deg ees o eedom in he
nume a o ; DoFdenom: deg ees o eedom in he denomina o .
4.4. Pos -hoc analyses and e ec sizes in Di Ai W
The ANOVA F- es s in Table 3 e ealed signi ican o e all di e ences ac oss
anno a o s in gene al. To ind di e ences be ween speci ic anno a o s, we pe -485
o med pos -hoc es s o assess he signi icance o di e ences be ween pai s o
g oup means using R’s emmeans lib a y [71] (Table 4).
24
Figu e 12: Compa ison o CV losses L[10−3] in ou Di Ai W da ase : Mean and 95%
con idence in e al. F on al (a) and la e al (b) iews.
IRNe Ai W[F on al] beha ed compa ably o bo h humans (Figu e 12a),
wi h e ec sizes non-signi ican ly di e en om ze o (see anno 1 - incep ,an-
no 2 - incep ows in Table 4); whe eas MNe Ai W[F on al] exhibi ed lowe 490
pe o mance, al hough wi h small s anda dized e ec sizes: 0.2245, 0.2382. IR-
Ne Ai W[La e al] pe o med on a e age wo se han humans (Figu e 12b), ye
be e han MNe Ai W[La e al]. Besides, IRNe Ai W[La e al]’s e ec sizes
we e non-signi ican ly di e en om ze o (Table 4); whe eas MNe Ai W[La e al]’s
e ec sizes we e small: 0.1431, 0.1518.495
4.5. Compa ison wi h s a e-o - he-a me hods
A ending o he Cumula i e E o Dis ibu ions (CED) o losses o he
i e p oposed Menpo de o mable models (Figu e 13), he one achie ing he bes
o e all pe o mance was he egula ized SDM (λ=106). No e ha , since he
de o mable models canno es ima e isibili ies ˆ i, all he losses he e co espond500
only o he di e ence be ween g ound u h (xi, yi) and es ima ed coo dina es
(ˆxi,ˆyi) o he isible landma ks ( i=1), i.e. he second summa ion e m in
Eq. [1]. In he on al iew scena io, SDM pe o med compa ably o ou DCNNs
o up o 80–90% o he images. Howe e , i s beha iou deg aded no o iously
o he la e al iew: i s median CV loss was app oxima ely 10 imes highe han505
o ou bes DCNN.
Figu e 14 depic s wo a ou able and wo in e media e SDM landma k de ec-
ion esul s. The i s wo pa ien cases (panels on he le -hand side) co espond
o sa is ac o y ou pu s –losses in he in e io 10% pe cen ile–; whe eas he wo
la e cases ( igh panels) co espond o in e media e pe o mances –a ound he510
median loss–. In he la e al iew, one case depic s a la ge loss owing o a majo
e o in a single landma k (s e nal manub ium, L11); whe eas he o he case
illus a es a no iceable o e all d i in loca ion. No e ha his ype o d i is
he p edominan beha iou in he cases wi h he mos se e e losses, whe e he
SDM model ails o a ain sa is ac o y de ec ion, con e ging ins ead o a noisy515
solu ion.
25
The DCNNs we e e alua ed ia 10- old CV s a i ied by sex, o gua an ee675
a ai and obus assessmen o pe o mance in images unseen du ing aining.
Resul s we e compa ed o in e -anno a o disc epancies as a ‘gold s anda d’ and
o 5 s a e-o - he-a de o mable models. O e all, IRNe Ai W[F on al] yielded
sa is ac o y gene aliza ion capabili ies (achie ing pe o mances a he le el o
human expe s), whe eas MNe Ai W[F on al] expe ienced only sligh deg ada-680
ion. In he la e al iew, bo h DCNNs pe o med s a is ically wo se han hu-
mans, al hough wi h small e ec sizes (non-signi ican o IRNe Ai W[La e al]).
This issue o lowe la e al pe o mance had al eady been desc ibed and discussed
in li e a u e by independen au ho s, who used di e en echniques. A guably,
i may be ela ed o in insic isual dis inguishabili y challenges, e en o ained685
human eyes. Fo ou bes model, no bias in pe o mance was ound ega ding
sex.
Acknowledgemen s
We would like o acknowledge he pa ien s who pa icipa ed in his esea ch,
he s a a he Uni e si y Hospi als o Galdakao-Usansolo and Basu o om he690
public Basque heal hca e sys em (Osakide za), as well as he ‘E hics Commi ee
o Resea ch wi h medica ion’ o he Basque Coun y (CEIm-E).
The au ho s hank also D . Amani Taha o he wo k deploying ou DCNNs
in sma phone de ices.
CRediT au ho ship s a emen 695
Fe nando Ga c´ıa-Ga c´ıa: Concep ualiza ion, me hodology, so wa e, in-
es iga ion, alida ion, o mal analysis, isualiza ion, w i ing - o iginal d a .
Dae-Jin Lee: Fo mal analysis, isualiza ion, supe ision, w i ing - e iew &
edi ing, p ojec adminis a ion, unding acquisi ion. F ancisco J. Mendoza-700
Ga c´es: Concep ualiza ion, me hodology, esou ces, da a cu a ion, w i ing -
e iew & edi ing, unding acquisi ion. So ´ıa I igoyen-Mi ´o: Resou ces, da a
cu a ion. Ma ´ıa J. Lega e a: Resou ces, da a cu a ion. Susana Ga c´ıa-
Gu i´e ez: Supe ision, p ojec adminis a ion, unding acquisi ion. Inmac-
ulada A os egui: W i ing - e iew & edi ing, supe ision, p ojec adminis a-705
ion, unding acquisi ion.
Funding
This esea ch is suppo ed by he Spanish S a e Resea ch Agency AEI unde
he p ojec S3M1P4R (PID2020-115882RB-I00), as well as by he Basque Go -
e nmen EJ-GV unde he g an ‘A i icial In elligence in BCAM’ 2019/00432,710
unde he s a egy ‘Ma hema ical Modelling Applied o Heal h’, and unde he
BERC 2018–2021 and 2022–2025 p og ammes, and also by he Spanish Min-
is y o Science and Inno a ion: BCAM Se e o Ochoa acc edi a ion CEX2021-
001142-S / MICIN / AEI / 10.13039/501100011033.
32
The unding sou ces had no ole in his wo k: nei he in he design and715
conduc o he s udy, collec ion, managemen , analysis and in e p e a ion o
he da a, no in he p epa a ion, e iew, app o al o he manusc ip , no in he
decision o submi he manusc ip o publica ion.
Con lic o in e es s a emen
The au ho s decla e ha hey ha e no known compe ing inancial in e -720
es s o pe sonal ela ionships ha could ha e appea ed o in luence he wo k
epo ed in his pape .
To maximize he impac o his s udy, he Basque Cen e o Applied Ma h-
ema ics (BCAM) submi ed a p o isional applica ion o in ellec ual p ope y
o he Te i o ial O ice o In ellec ual P ope y o he Basque Au onomous725
Communi y, Spain.
33
Re e ences
[1] J. L. Ap elbaum, C. A. Hagbe g, R. A. Caplan, C. D. Bli , R. T. Connis,
D. G. Nickino ich, C. A. Hagbe g, e al., P ac ice guidelines o manage-
men o he di icul ai way: An upda ed epo by he Ame ican Socie y o 730
Anes hesiologis s Task Fo ce on Managemen o he Di icul Ai way, Anes-
hesiology 118 (2) (2013) 251–270. doi:10.1097/ALN.0b013e31827773b2.
[2] C. F e k, V. S. Mi chell, A. F. McNa y, C. Mendonca, R. Bhag a h, A. Pa-
el, E. P. O’Sulli an, N. M. Woodall, I. Ahmad, Di icul Ai way Socie y
2015 guidelines o managemen o unan icipa ed di icul in uba ion in735
adul s, B J Anaes h 115 (6) (2015) 827–848. doi:10.1093/bja/ae 371.
[3] M. E. De sky, N. Ji aj, N. K. Adhika i, J. O. F ied ich, R. Pin o, D. L.
Simel, D. N. Wijeysunde a, D. C. Scales, Will his pa ien be di icul o
in uba e? The a ional clinical examina ion sys ema ic e iew, J Am Med
Assoc 321 (5) (2019) 493–503. doi:10.1001/jama.2018.21413.740
[4] A. K. Nø sko , C. V. Rosens ock, J. We e sle , G. As up, A. A sha i,
L. H. Lunds øm, Diagnos ic accu acy o anaes hesiologis s’ p edic ion o
di icul ai way managemen in daily clinical p ac ice: A coho s udy o 188
064 pa ien s egis e ed in he Danish Anaes hesia Da abase, Anaes hesia
70 (3) (2015) 272–281. doi:10.1111/anae.12955.745
[5] S. R. Mallampa i, S. P. Ga , L. D. Gugino, S. P. Desai, B. Wa aksa,
D. F eibe ge , P. L. Liu, A clinical sign o p edic di icul acheal in u-
ba ion; a p ospec i e s udy, Can Anaes h Soc J 32 (4) (1985) 429–434.
doi:10.1007/BF03011357.
[6] A. Vannucci, L. F. Ca allone, Bedside p edic o s o di icul in uba ion: A750
sys ema ic e iew, Mine a Anes esiol 82 (1) (2016) 69–83.
[7] D. Ro h, N. L. Pace, A. Lee, K. Ho hannisyan, A. M. Wa eni s, J. A ich,
H. He kne , Bedside es s o p edic ing di icul ai ways: An ab idged
Coch ane diagnos ic es accu acy sys ema ic e iew, Anaes hesia 74 (7)
(2019) 915–928. doi:10.1111/anae.14608.755
[8] C. Ma a a, E. Panki , L. Ahumada, B. Weinga en, A. Simpao, A i icial
in elligence, machine lea ning and he pedia ic ai way, Paedia Anaes h
30 (3) (2020) 264–268. doi:10.1111/pan.13792.
[9] J. C. Alexande , B. T. Romi o, M. C. C¸ obano˘glu, The p esen and
u u e ole o a i icial in elligence and machine lea ning in anes he-760
siology, In Anes hesiol Clin 58 (4) (2020) 7–16. doi:10.1097/AIA.
0000000000000294.
[10] D. A. Hashimo o, E. Wi kowski, L. Gao, O. Mei eles, G. Rosman, A i i-
cial in elligence in anes hesiology: Cu en echniques, clinical applica ions,
and limi a ions, Anes hesiology 132 (2020) 379–394. doi:10.1097/ALN.765
0000000000002960.
34
[11] Q. Yan, H. Yan, F. Han, X. Wei, T. Zhu, SVM-based decision sup-
po sys em o clinic aided acheal in uba ion p edica ion wi h mul i-
ple ea u es, Expe Sys Appl 36 (3 PART 2) (2009) 6588–6592. doi:
10.1016/j.eswa.2008.07.076.770
[12] O. Lange on, P. Cu illon, C. Iba˜nez-Es e e, F. Len an , B. Riou, Y. Le
Manach, P edic ion o di icul acheal in uba ion: Time o a pa adigm
change, Anes hesiology 117 (6) (2012) 1223–1233. doi:10.1097/ALN.
0b013e31827537cb.
[13] J. H. Kim, H. Kim, J. S. Jang, S. M. Hwang, S. Y. Lim, J. J. Lee,775
Y. S. Kwon, De elopmen and alida ion o a di icul la yngoscopy p e-
dic ion model using machine lea ning o neck ci cum e ence and hy-
omen al heigh , BMC Anes hesiol 21 (1) (2021) 125. doi:10.1186/
s12871-021-01343-4.
[14] S. Yamanaka, T. Go o, K. Mo ikawa, H. Wa ase, H. Okamo o, Y. Hagi-780
wa a, K. Hasegawa, Machine lea ning app oaches o p edic ing di icul
ai way and i s -pass success in he eme gency depa men : Mul icen e
p ospec i e obse a ional s udy, In e ac J Med Res 11 (1) (2022) e28366.
doi:10.2196/28366.
[15] M. Zhou, W. Y. Xu, S. Xu, Q. L. Zang, Q. Li, L. Tan, Y. C. Hu, N. Ma,785
e al., P edic ing di icul ai way in uba ion in hy oid su ge y using mul-
iple machine lea ning and deep lea ning algo i hms, F on Public Heal 10
(2022) 1–14. doi:10.3389/ pubh.2022.937471.
[16] M. Wilson, D. Spiegelhal e , J. Robe son, P. Lesse , P edic ing di icul
in uba ion, B J Anaes h 61 (2) (1988) 211–216. doi:10.1093/bja/61.2.790
211.
[17] M. Naguib, F. L. Scamman, C. O’Sulli an, J. Ake , A. F. Ross, S. Kosmach,
J. E. Enso , P edic i e pe o mance o h ee mul i a ia e di icul acheal
in uba ion models: A double-blind, case-con olled s udy, Anes h Analg
102 (3) (2006) 818–824. doi:10.1213/01.ane.0000196507.19771.b2.795
[18] A. K. Chhina, R. Jain, P. L. Gau am, J. Ga g, N. Singh, A. G ewal, Fo mu-
la ion o a mul i a ia e p edic i e model o di icul in uba ion: A double
blinded p ospec i e s udy, J Anaes hesiol Clin Pha macol 34 (1) (2018)
62–67. doi:10.4103/joacp.JOACP_230_16.
[19] T. E. Ta ola a, M. N. Gu can, S. Segal, M. K. K. Niazi, Iden i ica ion o 800
di icul o in uba e pa ien s om on al ace images using an ensemble o
deep lea ning models, Compu Biol Med 136 (2021) 104737. doi:10.1016/
j.compbiomed.2021.104737.
[20] T. Hayasaka, K. Kawano, K. Ku iha a, H. Suzuki, M. Nakane, K. Kawa-
mae, C ea ion o an a i icial in elligence model o in uba ion di icul y805
classi ica ion by deep lea ning (con olu ional neu al ne wo k) using ace
35
images: an obse a ional s udy, J In ensi e Ca e 9 (1) (2021) 1–14.
doi:10.1186/s40560-021-00551-x.
[21] H. Y. Cho, K. Lee, H. J. Kong, H. L. Yang, C. W. Jung, H. P. Pa k,
J. Y. Hwang, H. C. Lee, Deep-Lea ning model associa ing la e al ce i-810
cal adiog aphic ea u es wi h Co mack–Lehane g ade 3 o 4 glo ic iew,
Anaes hesia 78 (1) (2023) 64–72. doi:10.1111/anae.15874.
[22] N. Suzuki, S. Isono, T. Ishikawa, Y. Ki amu a, Y. Takai, T. Nishino,
Submandible angle in nonobese pa ien s wi h di icul acheal in uba-
ion, Anes hesiology 106 (5) (2007) 916–923. doi:10.1097/01.anes.815
0000265150.71319.91.
[23] C. W. Conno , S. Segal, Accu a e classi ica ion o di icul in uba ion by
compu e ized acial analysis, Anes h Analg 112 (1) (2011) 84–93. doi:
10.1213/ANE.0b013e31820098d6.
[24] G. L. Cuende , P. Schoe ke , A. Y¨uce, M. So ci, H. Gao, C. Pe uchoud,820
J. P. Thi an, Facial image analysis o ully au oma ic p edic ion o di icul
endo acheal in uba ion, IEEE T ans Biomed Eng 63 (2) (2016) 328–329.
doi:10.1109/TBME.2015.2457032.
[25] Y. Wu, Q. Ji, Facial landma k de ec ion: A li e a u e su ey, In J Compu
Vis 127 (2) (2019) 115–142. doi:10.1007/s11263-018-1097-z.825
[26] Y. Sun, X. Wang, X. Tang, Deep lea ning ace ep esen a ion om p e-
dic ing 10,000 classes, in: P oc IEEE Con Compu Vis Pa e n Recogni ,
2014, pp. 1891–1898. doi:10.1109/c p .2014.244.
[27] G. Guo, N. Zhang, A su ey on deep lea ning based ace ecogni ion,
Compu Vision Image Unde s anding 189 (2019) 102805. doi:10.1016/830
j.c iu.2019.102805.
[28] M. Pan ic, L. J. Ro hk an z, Au oma ic analysis o acial exp essions: The
s a e o he a , IEEE T ans Pa e n Anal Mach In ell 22 (12) (2000) 1424–
1445. doi:10.1109/34.895976.
[29] S. Li, W. Deng, Deep acial exp ession ecogni ion: A su ey, IEEE T ans835
A ec i e Compu 13 (3) (2022) 1195–1215. doi:10.1109/ a c.2020.
2981446.
[30] K. Pa el, D. Meh a, C. Mis y, R. Gup a, S. Tanwa , N. Kuma , M. Alazab,
Facial sen imen analysis using AI echniques: S a e-o - he-a , axonomies,
and challenges, IEEE Access 8 (2020) 90495–90519. doi:10.1109/access.840
2020.2993803.
[31] L. Tu, A. R. Po as, A. Mo ales, D. A. Pe ez, G. Piella, F. Sukno, M. G.
Lingu a u, Th ee-dimensional ace econs uc ion om uncalib a ed pho-
og aphs: Applica ion o ea ly de ec ion o gene ic synd omes, in: P oc In
Wo k Unce ain Sa e U il Mach Lea n Med Imaging, 2019, pp. 182–189.845
doi:10.1007/978-3-030-32689-0_19.
36
[32] X. Gao, Y. Su, X. Li, D. Tao, A e iew o ac i e appea ance models,
IEEE T ans Sys Man Cybe n Pa C Appl Re 40 (2) (2010) 145–158.
doi:10.1109/TSMCC.2009.2035631.
[33] D. C is inacce, T. F. Coo es, e al., Fea u e de ec ion and acking wi h850
cons ained local models, in: BMVC, Vol. 1 (2), 2006, p. 3.
[34] J. Deng, A. Roussos, G. Ch ysos, E. Ve e as, I. Ko sia, J. Shen,
S. Za ei iou, The Menpo benchma k o mul i-pose 2D and 3D acial land-
ma k localisa ion and acking, In J Compu Vis 127 (6-7) (2019) 599–624.
doi:10.1007/s11263-018-1134-y.855
[35] E. An onakos, J. Alabo -I-Medina, G. Tzimi opoulos, S. P. Za ei iou,
Fea u e-based Lucas-Kanade and ac i e appea ance models, IEEE T ans
Image P ocess 24 (9) (2015) 2617–2632. doi:10.1109/TIP.2015.2431445.
[36] X. Xiong, F. De La To e, Supe ised descen me hod and i s applica-
ions o ace alignmen , P oc IEEE Compu Soc Con Compu Vis Pa e n860
Recogni (2013) 532–539doi:10.1109/CVPR.2013.75.
[37] L. Ale, X. Fang, D. Chen, Y. Wang, N. Zhang, Ligh weigh deep lea ning
model o acial exp ession ecogni ion, in: P oc IEEE In Con T us Se-
cu P i Compu Commun, 2019, pp. 707–712. doi:10.1109/T us Com/
BigDa aSE.2019.00100.865
[38] R. Ranjan, V. M. Pa el, R. Chellappa, Hype Face: A deep mul i- ask lea n-
ing amewo k o ace de ec ion, landma k localiza ion, pose es ima ion,
and gende ecogni ion, IEEE T ans Pa e n Anal Mach In ell 41 (1) (2019)
121–135. doi:10.1109/TPAMI.2017.2781233.
[39] X. Zhu, X. Liu, Z. Lei, S. Z. Li, Face alignmen in ull pose ange: A 3D870
o al solu ion, IEEE T ans Pa e n Anal Mach In ell 41 (1) (2019) 78–92.
doi:10.1109/TPAMI.2017.2778152.
[40] G. Li jens, T. Kooi, B. E. Bejno di, A. A. A. Se io, F. Ciompi, M. Gha oo-
ian, J. A. an de Laak, B. an Ginneken, C. I. S´anchez, A su ey on
deep lea ning in medical image analysis, Med Image Anal 42 (1995) (2017)875
60–88. doi:10.1016/j.media.2017.07.005.
[41] E. Gibson, W. Li, C. Sud e, L. Fidon, D. I. Shaki , G. Wang, Z. Ea on-
Rosen, R. G ay, e al., Ni yNe : a deep-lea ning pla o m o medical
imaging, Compu Me hods P og ams Biomed 158 (2018) 113–122. doi:
10.1016/j.cmpb.2018.01.025.880
[42] C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, C. Liu, A su ey on deep
ans e lea ning, in: Lec No es Compu Sci, 2018, pp. 270–279. doi:
10.1007/978-3-030-01424-7_27.
37
[43] N. Tajbakhsh, J. Y. Shin, S. R. Gu udu, R. T. Hu s , C. B. Kendall,
M. B. Go way, J. Liang, Con olu ional neu al ne wo ks o medical image885
analysis: Full aining o ine uning?, IEEE T ans Med Imaging 35 (5)
(2016) 1299–1312. doi:10.1109/TMI.2016.2535302.
[44] I. Good ellow, Y. Bengio, A. Cou ille, Deep lea ning, Camb idge MIT
p ess, 2016.
[45] Q. Yang, Y. Zhang, W. Dai, S. J. Pan, T ans e lea ning, Camb idge Uni-890
e si y P ess, 2020. doi:10.1017/9781139061773.
[46] J. Yosinski, J. Clune, Y. Bengio, H. Lipson, How ans e able a e ea u es
in deep neu al ne wo ks?, in: Ad Neu al In P ocess Sys , 2014, pp. 3320–
3328.
[47] J. Deng, W. Dong, R. Soche , L.-J. Li, Kai Li, Li Fei-Fei, ImageNe : A895
la ge-scale hie a chical image da abase, in: P oc IEEE Con Compu Vis
Pa e n Recogni , 2009, pp. 248–255. doi:10.1109/CVPR.2009.5206848.
[48] O. Russako sky, J. Deng, H. Su, J. K ause, S. Sa heesh, S. Ma, Z. Huang,
A. Ka pa hy, e al., ImageNe la ge scale isual ecogni ion challenge, In J
Compu Vis 115 (3) (2015) 211–252. doi:10.1007/s11263-015-0816-y.900
[49] Z. Liu, P. Luo, X. Wang, X. Tang, Deep lea ning ace a ibu es in he wild,
in: P oc IEEE Con Compu Vis Pa e n Recogni , 2015, pp. 3730–3738.
a Xi :1411.7766,doi:10.1109/ICCV.2015.425.
[50] M. K¨os inge , P. Wohlha , P. M. Ro h, H. Bischo , Anno a ed acial land-
ma ks in he wild: A la ge-scale, eal-wo ld da abase o acial landma k905
localiza ion, in: P oc IEEE In Wo k Benchma king Facial Image Anal
Technol, 2011, pp. 2144–2151. doi:10.1109/ICCVW.2011.6130513.
[51] R. S. Co mack, J. Lehane, Di icul acheal in uba ion in obs e ics,
Anaes hesia 39 (11) (1984) 1105–1111. doi:10.1111/j.1365-2044.1984.
b08932.x.910
[52] F. Adne , S. W. Bo on, S. X. Racine, J. L. Clemessy, J. L. Fou nie ,
P. Plaisance, C. Lapand y, The in uba ion di icul y scale (IDS): P o-
posal and e alua ion o a new sco e cha ac e izing he complexi y o
endo acheal in uba ion, Anes hesiology 87 (6) (1997) 1290–1297. doi:
10.1097/00000542-199712000-00005.915
[53] D. Boisson-Be and, J. L. Bou gain, J. Cambouli es, V. C inque e, A. M.
C os, M. Dub euil, B. Eu in, J. P. Habe e , T. Po eche , D. Tho in,
P. Ra ussin, B. Riou, In uba ion di icile: Soci´e ´e F an¸caise d’Anes h´esie e
de R´eanima ion, Expe ise collec i e, Ann F Anes h Reanim 15 (2) (1996)
207–214. doi:10.1016/0750-7658(96)85047-7.920
38
[54] C. Szegedy, S. Io e, V. Vanhoucke, A. A. Alemi, Incep ion-V4, Incep ion-
ResNe and he impac o esidual connec ions on lea ning, in: P oc AAAI
Con A i In ell, 2017, pp. 4278–4284. a Xi :1602.07261.
[55] C. A. Fe ei a, T. Melo, P. Sousa, M. I. Meye , E. Shakibapou , P. Cos a,
A. Campilho, Classi ica ion o b eas cance his ology images h ough925
ans e lea ning using a p e- ained Incep ion Resne V2, in: Lec No es
Compu Sci, 2018, pp. 763–770. doi:10.1007/978-3-319-93000-8_86.
[56] O. Pelka, F. Nensa, C. M. F ied ich, Anno a ion o enhanced adiog aphs
o medical image e ie al wi h deep con olu ional neu al ne wo ks, PLoS
One 13 (11) (2018) 1–18. doi:10.1371/jou nal.pone.0206229.930
[57] L. D. Nguyen, D. Lin, Z. Lin, J. Cao, Deep CNNs o mic oscopic image
classi ica ion by exploi ing ans e lea ning and ea u e conca ena ion, in:
P oc IEEE In Symp Ci cui s Sys , Vol. 2018-May, 2018, pp. 3–7. doi:
10.1109/ISCAS.2018.8351550.
[58] M. Sandle , A. Howa d, M. Zhu, A. Zhmogino , L. C. Chen, MobileNe V2:935
In e ed esiduals and linea bo lenecks, in: P oc IEEE Con Compu Vis
Pa e n Recogni , 2018, pp. 4510–4520. doi:10.1109/CVPR.2018.00474.
[59] K. Aguila , G. H. Al ´e ez, C. Aguila , De ec ion o di icul ai way us-
ing deep lea ning, Mach Vis Appl 31 (1) (2020) 4. doi:10.1007/
s00138-019-01055-3.940
[60] C. Ma a a, E. Panki , S. Raisbeck, M. Caldei a, F. Alam, A con olu ional
neu al ne wo k o eal ime classi ica ion, iden i ica ion, and labelling o
ocal co d and acheal using la yngoscopy and b onchoscopy ideo, J Med
Sys 44 (2) (2020) 44. doi:10.1007/s10916-019-1481-4.
[61] R. Koha i, A s udy o c oss- alida ion and boo s ap o accu acy es ima-945
ion and model selec ion, in: P oc 14 h In Join Con A i In ellig, 1995,
pp. 1137–1143.
[62] D. P. Kingma, J. Ba, Adam: A me hod o s ochas ic op imiza ion, Tech.
ep., a Xi (2014). doi:10.48550/a Xi .1412.6980.
[63] J. Alabo -i Medina, S. Za ei iou, Uni ying holis ic and pa s-based de-950
o mable model i ing, in: P oc IEEE Con Compu Vis Pa e n Recogn,
2015, pp. 3679–3688. doi:10.1109/CVPR.2015.7298991.
[64] J. Alabo -I-Medina, E. An onakos, J. Boo h, P. Snape, S. Za ei iou,
Menpo: A comp ehensi e pla o m o pa ame ic image alignmen and
isual de o mable models, in: P oc ACM Con Mul imed, 2014, pp. 679–955
682. doi:10.1145/2647868.2654890.
[65] S. an Buu en, K. G oo huis-Oudshoo n, mice: Mul i a ia e impu a ion
by chained equa ions in R, Jou nal o S a is ical So wa e 45 (3) (2011)
1–67. doi:10.18637/jss. 045.i03.
39
[66] V. Kazemi, J. Sulli an, One millisecond ace alignmen wi h an ensemble960
o eg ession ees, in: P oc IEEE Compu Soc Con Compu Vis Pa e n
Recogni , 2014, pp. 1867–1874. doi:10.1109/CVPR.2014.241.
[67] N. Dalal, B. T iggs, C. Schmid, Human de ec ion using o ien ed his og ams
o low and appea ance, Lec u e No es in Compu e Science (2006) 428–
441doi:10.1007/11744047_33.965
[68] L. McInnes, J. Healy, J. Mel ille, UMAP: Uni o m mani old app oxima ion
and p ojec ion o dimension educ ion, a Xi 1802.03426 (2018) 1–63.
[69] D. Ba es, M. M¨achle , B. Bolke , S. Walke , Fi ing linea mixed-e ec s
models using lme4, J S a So w 67 (1) (2015) 1–48. doi:10.18637/jss.
067.i01.970
[70] A. Kuzne so a, P. B. B ockho , R. H. B. Ch is ensen, lme Tes package:
Tes s in linea mixed e ec s models, J S a So w 82 (13) (2017) 1–26.
doi:10.18637/jss. 082.i13.
[71] R. V. Len h, emmeans: Es ima ed ma ginal means, aka leas -squa es
means, package e sion 1.5.4 (2021).975
[72] A. T an, C. Nguyen, T. Hassne , T ans e abili y and ha dness o supe ised
classi ica ion asks, in: P oc IEEE In Con Compu Vis, 2019, pp. 1395–
1405. doi:10.1109/ICCV.2019.00148.
[73] C. V. Nguyen, T. Hassne , M. Seege , C. A chambeau, LEEP: A new mea-
su e o e alua e ans e abili y o lea ned ep esen a ions (2020). doi:980
10.48550/a Xi .2002.12462.
[74] B. F. Kla e, M. J. Bu ge, J. C. Klon z, R. W. Vo de B uegge, A. K. Jain,
Face ecogni ion pe o mance: Role o demog aphic in o ma ion, IEEE
T ans In Fo ensics Secu 7 (6) (2012) 1789–1801. doi:10.1109/TIFS.
2012.2214212.985
[75] L. B iceno, G. Paul, MakeHuman: A e iew o he modelling amewo k,
in: P oc 20 h Cong In E gonom Assoc – Human Simula ion and Vi ual
En i onmen s, 2019, pp. 224–232.
[76] L. Quan, Z.-D. Lan, Linea n-poin came a pose de e mina ion, IEEE T ans
Pa e n Anal Mach In ell 21 (8) (1999) 774–780. doi:10.1109/34.784291.990
[77] E. Ma chand, H. Uchiyama, F. Spindle , Pose es ima ion o augmen ed
eali y: A hands-on su ey, IEEE T ans Visual Compu G aphics 22 (12)
(2016) 2633–2651. doi:10.1109/ cg.2015.2513408.
40
