Towa d G aph-based Semi-supe ised Face Beau y
P edic ion
Fadi Do naika 1,2
1Uni e si y o he Basque Coun y (UPV/EHU), Spain
2IKERBASQUE, Basque Founda ion o Science, Spain
EMAIL: [email p o ec ed] TEL: 0034 943018034
Kunwei Wang 1,3
1Uni e si y o he Basque Coun y (UPV/EHU), Spain
3No hwes e n Poly echnic Uni e si y, Xian, China
EMAIL: wkwke[email p o ec ed]du.cn
Ignacio A ganda-Ca e as 1,2
1Uni e si y o he Basque Coun y (UPV/EHU), Spain
2IKERBASQUE, Basque Founda ion o Science, Spain
EMAIL: ignacio.a [email p o ec ed]
Anne Elo za 1
1Uni e si y o he Basque Coun y (UPV/EHU), Spain
EMAIL: [email p o ec ed]
Abdelmalik Moujahid 1
1Uni e si y o he Basque Coun y (UPV/EHU), Spain
EMAIL: ab[email p o ec ed]
1
This is he accep ed manusc ip o he a icle ha appea ed in inal o m in Expe Sys ems wi h Applica ions 142 : (2020) //
A icle ID 112990, which has been published in inal o m a h ps://doi.o g/10.1016/j.eswa.2019.112990. © 2019 Else ie unde
CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/by-nc-nd/4.0/)
(?;?). The gene ic semi-supe ised lea ning me hods using g aph-based label p opa-
ga ion a ac ed much a en ion in he las decade. All o hem impose ha samples wi h
high simila i y should sha e simila labels. They di e by he egula iza ion e m as well as
by he loss unc ion used o i ing label in o ma ion associa ed wi h he labeled samples.
All o hese me hods use he g aph simila i y ma ix and he ini ial labels o some samples.
Some ecen label p opaga ion algo i hms ( hey can also be called classi ie s (Sousa e al.,
2013)) a e: Gaussian Fields and Ha monic Func ions (GFHF) (Zhu e al., 2003), Local
and Global Consis ency (LGC) (Zhou e al., 2004), Laplacian Regula ized Leas Squa e
(LapRLS) (Belkin e al., 2006), Robus Mul i-class G aph T ansduc ion (RMGT) (Liu
and Chang, 2009), Flexible Mani old Embedding (FME) (Nie e al., 2010). These ech-
niques can be ei he ansduc i e (de ined o aining samples only) o induc i e (de ined
o bo h aining and unseen samples). The me hod p oposed in (?), lea ns a uni ied g aph
ia a s uc u al egula iza ion e m. Ins ead o weigh egula iza ion which is adop ed by
p e ious wo ks, he wo k p esen ed in (?) lea ns a uni ied g aph and weigh s om a p io i
indi idual g aphs.
Thinking abou beau y, i seems qui e easonable o assume ha when wo aces e-
semble each o he hey should ha e simila a ac i eness sco es. This abs ac idea can be
ma e ialized by cons uc ing a weigh ed g aph, in which nodes a e images (o hei desc ip-
o s) and he weigh s be ween each pai o nodes ep esen hei simila i ies. The e o e,
in ou case, we exploi mani old s uc u e o ace images (bo h labeled and unlabeled) ia
g aphs. In his assump ion, simila images should sha e simila beau y sco es.
In his pape , we in oduce he semi-supe ised pa adigm o ace beau y p edic ion
ield. We explo e some mani old based semi-supe ised algo i hms o he speci ic p ob-
lem o au oma ic acial beau y assessmen . Mo eo e , we p opose a non-linea Flexible
Mani old Embedding o ca ying ou he sco e p opaga ion. This p oposed me hod can
achie e s a e-o - he-a esul s.
0.1 No a ions and p elimina ies
In he sequel, capi al bold le e s deno e ma ices and bold le e s deno e ec o s. Assume
ha x1,x2,...,xla e llabeled ace images (o hei desc ip o s) and ha xl+1,xl+2,...,xN
a e he uunlabeled ace images. He e, he ec o xi e e s o he i h ace image.
The da a ma ix Xis de ined by X= [x1,x2,...,xN]∈RD×N. The o al numbe o
aining images is N=l+u. The semi-supe ised algo i hms we a e using we e o iginally
de eloped o classi ica ion asks, whe e he g ound u h labels Yand he p edic ed labels
Fa e ma ices in RN×C, whe e Yij = 1 i sample ibelongs o class jand Yij = 0 o he wise.
Cdeno es he numbe o classes.
Since ou p oblem is essen ially a eg ession p oblem, he labels a e eal numbe s ha
can be ep esen ed as a column ec o y∈RN. The i s l ows o ywill con ain he sco es
o he llabeled images, while he las u ows will gene ally be 0, since hey co espond o
he uunlabeled images. In addi ion o he ini ial label (o sco e) ec o y, we conside
he unknown label ec o ∈RN ha should be es ima ed.
As al eady s a ed, he way o exploi ing he in o ma ion con ained in unlabeled da a
is o conside a simila i y g aph ha encodes he pai wise simila i y be ween images. To
his end, we ha e o in oduce a simila i y ma ix S∈RN×N(which will be symme ic, so
ou g aph has o be undi ec ed). Each elemen Sij o Sis he simila i y be ween samples
2
iand j(i.e., ace iand ace j). This g aph is assumed o cap u e much in o ma ion abou
he da a mani old. In ou wo k, wi hou loss o gene ali y, he a ini y ma ix Sis se o
he KNN g aph simila i y ma ix as i o e s a simple and e y e icien me hod o g aph
cons uc ion. I p oceeds as ollows. Fi s , he adjacency ma ix is cons uc ed ( he edges
a e se ). Second, he weigh s o he edges a e es ima ed.
Fo adjacency ma ix cons uc ion, K-Nea es Neighbo can be used in o de o ind
he neighbo s o a da um. The e is a unc ion ha de ines he dis ance (simila i y) o one
inpu wi h espec o he o he s.
In he second phase, a weigh should be assigned o each cons uc ed edge. In gene al,
his weigh should quan i y he simila i y be ween wo connec ed nodes. Le sim(xi,xj)
be he simila i y sco e be ween neighbo s xiand xj, hen he elemen s o he g aph weigh
ma ix Sa e gi en by Eq. (1).
Sij =sim(xi,xj) i xiand xja e neighbou s
0 o he wise (1)
The e a e se e al choices o sim(xi,xj). Fo ins ance, in (Belkin and Niyogi, 2003) he
au ho s use he hea ke nel sim(xi,xj) = e−
kxi−xjk2
whe e can be se o he a e age o
squa ed dis ances in he aining se . We adop he abo e simila i y and se he neighbo -
hood size o he KNN g aph o 10 as in many s udies (Do naika and El T aboulsi, 2016)).
We emphasize ha mo e sophis ica ed g aph cons uc ion me hods can be used in o de
o es ima e he simila i y ma ix S(e.g., (Cheng e al., 2010; He e al., 2011; Do naika
e al., 2013; Do naika and Bosaghzadeh, 2015; Do naika e al., 2016; Nie e al., 2016)).
The Laplacian ma ix o Sis gi en by L=D−S, whe e Dis he diagonal ma ix
whose elemen s a e he ow sums o S. The no malized Laplacian ma ix is gi en by
ˆ
L=I−D−1/2SD−1/2, whe e Iis he iden i y ma ix o size N. Finally, 1,0∈RNdeno e
ec o s wi h all elemen s as 1 and 0 espec i ely. The no m ||·|| deno es he Euclidean
no m.
0.2 P oblem s a emen
The inpu da a a e gi en by a se o ace images o hei desc ip o s x1,...,xl,xl+1,xl+2,...,xN.
land u=N−l ep esen he numbe s o labeled and unlabeled ace images, espec i ely.
The labels a e gi en by eal sco es y1, y2, . . . , yl. The goal is o in e he sco es o he
unlabeled ace images. Fo illus a ion, Figu e 1 shows a oy example ha demons a es
he p inciple o g aph-based sco e p opaga ion. In his example, we ha e se en ace images,
om which only 4 images a e labeled wi h ace beau y sco e. The emaining images a e
unlabeled. The objec i e is o eco e he ace beau y sco e o all unlabeled images by
pe o ming sco e p opaga ion o e he g aph. The simila i y ma ix o he g aph, S, is
cons uc ed using all aining images. Unless s a ed o he wise, he pape a ge s lea ning
om single- alued sco es. Thus, lea ning om disc e e classes o om label dis ibu ion
is beyond he scope o he pape .
3
Figu e 1: P inciple o g aph-based beau y sco e p opaga ion.
4
1 G aph-based sco e p opaga ion schemes
In his sec ion, we will desc ibe some exis ing label p opaga ion me hods ha we e de el-
oped o disc e e classi ica ion. We also show hei adap a ion o he sco e p opaga ion,
whe e he sco e is a con inuous a iable. All exis ing g aph-based label p opaga ion scheme
use ei he a non no malized g aph o a no malized g aph. We emphasize ha he h ee
semi-supe ised schemes ha a e p esen ed and de eloped in Sec ions 3 and 4 use a no -
malized g aph. In o he wo ds, he objec i e unc ional o each me hod uses a no malized
Laplacian ma ix.
1.1 Local and Global Consis ency
The Local and Global Consis ency (LGC) me hod was in oduced in (Zhou e al., 2004).
I aims a p edic ing he disc e e labels o all labeled and unlabeled ins ances, F, by
minimizing he ollowing unc ion:
q(F) =
N
X
i,j=1
Sij 




i
√Dii − j
pDjj 




2
+µ
N
X
i=1 k i−yik2,(2)
whe e iis he i- h ow o Fand Dii is he sum o he i- h ow o So , in o he wo ds,
he sum o he simila i ies o sample iwi h all he o he images. The i s e m is he
smoo hness cons ain . The second e m is he i ing cons ain and µis he pa ame e
which con ols he ade-o be ween hem. The op imal solu ion o his p oblem can be
ound analy ically by anishing he i s de i a i es w. . . he unknown. I is gi en by
F= (I+ˆ
L/µ)−1Y, whe e Iis he iden i y ma ix o Ndimensions. Once he ma ix F
is es ima ed, he p edic ed class o an ins ance iwill be he maximum index jo he i- h
ow o F.
Ou e isi ed LGC should p edic he sco es o all images, namely he ec o . I
minimizes he ollowing:
q( ) =
N
X
i,j=1
ˆ
Sij( i− j)2+µk −yk2,(3)
whe e he ma ix ˆ
Sis gi en by ˆ
S=D−1S. As i can be seen, he c i e ion is simila o he
one p esen ed in (2). Howe e , he e a e wo signi ican di e ences. Fi s , he dis ance in
sco es be ween any pai o nodes xiand xjis no any mo e depending on he deg ee o
hese wo nodes. Indeed, empi ically, we ound ha keeping he deg ees in he pai wise
dis ance lead o wo se esul s. This is due o he ac ha ou p oblem is sco e es ima ion
ha should i some ixed alues p o ided by he labeled images. The e o e, whene e he
deg ees a e di e en he ec o s i
√Dii
and j
pDjj
will no be equally signi ican in he
dis ance depic ed in he i s e m o Eq. ( (2)). By ecalling he de ini ion o Dii (i is
he sum o he i- h ow in S), a simple analysis o his e m i
√Dii − j
pDjj
will show ha
he mos in luen ial samples a e he ones being he leas simila o he es .
Second, unlike disc e e label p opaga ion whe e i is sa e o se he label dis ibu ion
Yij associa ed wi h he unlabeled images o ze o ec o s, in ou case hese alues (i.e.,
5

yi, i = 1, ..., u) a e se o he a e age sco es ha can be easily known om he labeled
images. The solu ion o is again simila o he one p o ided by he LGC disc e e label
p opaga ion in which he Laplacian ma ix is now associa ed wi h he g aph ˆ
S+ˆ
ST
2:
= (I+ˆ
L/µ)−1y
1.2 Flexible Mani old Embedding
Simila ly o he LGC me hod, he Flexible Mani old Embedding (FME) me hod (Nie
e al., 2010) es ima es he labels Fby minimizing he ollowing cos unc ion:
g(F,W,b) = (FTL F) + β [(F−Y)TU(F−Y)] +
µ(||W||2+γ||XTW+1bT−F||2),(4)
whe e Uis an indica o ma ix, ha is a diagonal ma ix, wi h i s i s ldiagonal ele-
men s, co esponding o labeled ins ances, equal o 1, while he las udiagonal elemen s,
co esponding o unlabeled ins ances, a e equal o 0. This assumes ha he llabeled
images a e he i s lsamples in he da a ma ix Xand in he simila i y ma ix S.W
and bdeno e he unknown linea eg esso which maps he o iginal samples o he label
space.
As wi h he LGC me hod, a closed- o m solu ion can be ound by se ing he de i a i es
o gwi h espec o W,band Fas 0 (Nie e al., 2010). The solu ion is gi en by:
b=1
l+uFT1−WTX1(5)
W=γ(γXHcXT+I)−1XHcF(6)
F=β(βU+L+µγHc−µγ2Q)−1UY,(7)
whe e Q=XT
cXc(γXT
cXc+I)−1,Xc=X Hcand Hc=I−(1/(l+u))11T.
Flexible Mani old Embedding and eg ession As wi h he LGC me hod, his algo-
i hm was o iginally designed o classi ica ion asks. None heless, i can easily be adap ed
o wo k as a eg esso . I and ya e eal- alued ec o s ep esen ing he p edic ed labels
and he g ound- u h sco es, he cos unc ion becomes
g( ,w, b) = TL +β( −y)TU( −y) + µ(||w||2+γ||XTw+b1− ||2).(8)
The i s e m con ols he label smoo hness, he second one he label i ness and he
las e m i s a linea eg ession be ween ea u es and labels, whe e ||w||2is a egula iza ion
e m con olling he complexi y o he model ( hus, a oiding o e - i ing). β,µand γa e
he pa ame e s con olling he ade-o be ween all he e ms. The solu ion is gi en by:
b=1
N T1−wTX1(9)
w=γ(γXHcXT+I)−1XHc (10)
=β(βU+L+µγHc−µγ2Q)−1U y.(11)
6
FME and unseen da a One ad an age o he FME me hod, which dis inguishes i om
many p oposed label p opaga ion me hods, e.g., om he LGC me hod, is i s capaci y o
dealing wi h unseen da a. Apa om p edic ing he labels o he unlabeled da a in ec o
, i can p edic unseen da a using he eg ession model ha is al eady lea ned in (8).
Gi en he ea u es o he unseen da a Xuns, i s sco es would be:
uns =XT
unsw+b1.
2 P oposed app oach: Non-linea Flexible Mani old Em-
bedding
The FME model wo ks di ec ly on he da a samples. In many ecen machine lea ning
wo ks, i was shown ha wo king wi h a non-linea ep esen a ion o he da a can imp o e
he inal pe o mance o he lea ne . In ou wo k, we p opose o use he column gene a ion
ick in o de o ge he non-linea ep esen a ion o he o iginal da a ma ix X.
Column gene a ion eplaces each sample xiby a ec o o simila i ies o ha sample
wi h he samples con ained in a ixed se o samples (Kla e and Jain, 2013). Ve y o en,
he la e se is gi en by he aining samples o a subse o hem (Zhang e al., 2018).
In ou wo k, we use all aining samples as e e ence samples. The da a ma ix X
is hus eplaced by he ma ix G= [g1,g2,...,gN], whe e each ec o giis o med by
he simila i ies, i.e. gi=sim(xi,x1), ..., sim(xi,xN)∈RN×N. In ou case, we use he
Gaussian simila i y. This means ha
Gij =e
−||xi−xj||2
2 0σ2,
whe e σ2is a measu e o he a iabili y o he da a. Conc e ely, i is se o he mean o
he squa es o he dis ances be ween all pai s o samples. 0is a ke nel pa ame e ha can
con ol he simila i y unc ion. The e o e, he Non-linea Flexible Mani old Embedding
(NFME) can be o mula ed as ollows:
g( ,w, b) = TL +β( −y)TU( −y) + µ(||w||2+γ||GTw+b1− ||2).(12)
The NFME me hod can cope wi h non-linea da a when a linea eg ession may ha e a
poo pe o mance in he FME. A closed- o m solu ion can be ound again by se ing he
de i a i es o gwi h espec o ,w, and bas 0. The solu ion is gi en by:
b=1
N T1−wTG 1(13)
w=γ(γG HcGT+I)−1G Hc (14)
=β(βU+L+µ γ Hc−µ γ2Q)−1U y (15)
whe e Q=GT
cGc(γGT
cGc+I)−1,Gc=G Hc, and Hc=I−(1/(l+u))11T.
7
NFME and unseen da a Simila ly o he FME me hod, in he NFME me hod one
can easily handle unseen da a using he eg ession e m in (12). To do so, gi en a se
o unseen samples xuns
i|m
i=1, one has o build he simila i y ma ix o he unseen samples
Guns ∈Rm×N, whe e he elemen (i, j) is he simila i y unc ion o he unseen sample i,
xuns
i, and he aining sample j,xj. Then, he p edic ed sco es, uns ∈Rm, would be
uns =GT
unsw+b1.(16)
3 Expe imen al se up
3.1 Da ase s
Th ee da ase s a e used in his wo k: he SCUT-FBP da ase (Xie e al., 2015), he
Mul i Modali y-Beau y (M2B) da ase (Nguyen e al., 2013), and he SCUT-FBP5500
da ase (Liang e al., 2018). The i s was speci ically designed o au oma ic acial beau y
pe cep ion and con ains high esolu ion on -on ace po ai s o Asian emales. Mo eo e ,
he second was de eloped o e alua e beau y ia a ace, d essing and/o oice on bo h
Eas e n and Wes e n emales and each ins ance in he da ase con ains in o ma ion abou
he h ee modali ies. Howe e , we a e only ocusing on he acial images, which unlike
he ones in he SCUT-FBP da ase , show e y di e en poses and exp essions. This
complica es, in consequence, he beau y assessmen , which could be ound di icul e en
by a human a e .
Figu e 2: Examples o ace po ai s o SCUT-FBP da ase om (Xie e al., 2015).
SCUT-FBP da ase : The SCUT-FBP da ase con ains 500 high esolu ion on -on
ace po ai s o Asian emales wi h neu al exp essions, simple backg ound and minimal
occlusion, as can be seen in Figu e 2. These cha ac e is ics p e en om aking in o
accoun i ele an ac o s in he beau y classi ica ion ask. The beau y ankings (sco es)
lie in he in e al (1, 5) and a e he esul o a e aging a ious a ings. The a ings we e
collec ed among 75 indi iduals using a web-based ool wi h an a e age numbe o 70 a e s
pe image. The sco es app oxima ely ollow a no mal dis ibu ion (Figu e 3) wi h a small
peak a ound 4.5.
Ra e s’ consis ency and sel -consis ency a e checked in di e en ways by he au ho s
o he pape . Fo ins ance, low s anda d de ia ions in he a ings o each image indica e
8
Figu e 3: His og am o he a ing dis ibu ion om (Xie e al., 2015).
Figu e 4: Use in e ace o he a ac i eness anking ool om (Nguyen e al., 2013).
a e ’s ag eemen in he pe cep ion o beau y.
M2B da ase : The Mul i-Modali y Beau y da ase has been de eloped o s udy beau y
pe cep ion in h ee di e en modali ies, in d essing, in he ace and in he oice, as well
as he global beau y pe cei ed when any o hese h ee aspec s a e combined. The e o e,
he da ase con ains one ace pho o, one ull body pho o and one oice snippe o 1240
emales belonging o wo e hnic g oups: wes e ne s and eas e ne s (620 indi iduals in each
g oup). In addi ion, each o he emales o he da ase is a ed, in he di e en modali ies
and hei combina ions, wi h a ious sco es in he in e al [1, 10].
The a ings we e collec ed among 40 pa icipan s, which we e spli in o wo g oups
depending on hei e hnici y, so ha each o he pa icipan s a ed emales o hei own
e hnic g oup. The web ool used o his pu pose can be seen in Figu e 4. The a ings
we e ob ained using k-wise compa ison, which means ha he a e s a e asked o so
k emales acco ding o hei beau y, and hen hese k-wise a ings we e con e ed in o
global a ings in he in e al [1, 10] by sol ing an op imiza ion p oblem o p ese e as
many pai wise p e e ences as possible. The d awback o his me hod o collec ing he
labels is ha , unlike he SCUT-FBP da ase , whe e we had he a ings o a ious a e s
pe image, he e we ha e a unique a ing. Thus, we canno eally measu e he unce ain y
o each o he labels, e en i i seems o be impo an , since beau y is no an absolu e
concep .
9
Table 5: Summa y o he pe o mances o he supe ised and semi-supe ised me hods on
SCUT-FBP wi h a 10%–90% da a pa i ion.
Me hod MAE ↓RMSE ↓PC %↑-e o ↓
1-NN 0.0956 0.1248 51.59 0.2423
Ridge Reg ession 0.0946 0.1310 23.89 0.2328
Gaussian -SVR 0.0735 0.0955 71.51 0.1761
LGC 0.0983 0.1336 17.91 0.2470
FME 0.0707 0.0923 72.91 0.1682
NFME 0.0724 0.0946 72.90 0.1711
4.2 M2B da ase
In his sec ion, we p esen he esul s ob ained on M2B da ase , wi h a con igu a ion o
50% o he samples as labeled da a and he o he 50% o he samples as he unlabeled/ es
da a. All expe imen s a e ca ied ou doing 10 s a i ied spli s o he da a.
Table 6 shows he esul s o applying he h ee di e en g aph-based label p opaga ion
schemes on M2B da ase . In his expe imen , we conside h ee se s: he i s se con ains
620 images o Eas e n subjec s, he second se con ains 620 images o Wes e n subjec s.
The hi d one con ains he whole M2B da ase . The bes pe o mances a e shown in bold.
As i can be seen, he FME and NFME me hods achie ed he bes pe o mances. We
can also obse e ha when he wo ypes o aces Eas e n and Wes e n a e mixed, he
pe o mance o all schemes d opped. This sugges s ha o a compu a ional model he
ea u es abou beau y a e no he same o e e y e hnici y. Recall ha , in M2B da ase ,
he eas e n aces we e a ed by eas e n subjec s, and wes e n aces we e a ed by wes e n
subjec s. Since machine lea ning ies o imi a e human expe ise, using ace images
belonging o mixed e hnici ies and a ed by mo e han one e hnici y will be mo e di icul
han using ace images belonging o one e hnici y and a ed by ha e hnici y.
We can also obse e ha he NFME me hod ga e he bes pe o mances o he mixed
case. Compa ed o he SCUT esul s, he pe o mances ob ained on M2B da ase a e
wo se han hose ob ained wi h SCUT-FBP da ase . This is due o wo main easons:
(i) he images in M2B da ase a e mo e challenging (some aces co espond o mannequin
aces), (ii) he a ing p ocess is based on o de ing en images a a ime using 40 a e s.
On he o he hand, in SCUT-FBP da ase e e y ace image go he opinion o 70 a e s
on a e age.
Table 7 depic s he compa ison be ween he bes MAEs o some supe ised me hods
(Nguyen e al., 2013), which we e ca ied ou wi h 2- old c oss- alida ion, and ou s, which
has he same da a p opo ions, i.e., ain/ es is 50%/50%. The FAT me hod is a cascaded
es ima ion o he eg ession de ined he DFAT me hod. Ou MAEs a e mul iplied by 10,
because we used he no malized labels (di iding he o iginal sco e by 10). As i can be
seen, he NFME me hod has p o ided he bes MAE.
16

Table 6: A e age pe o mances ob ained wi h h ee g aph-based sco e p opaga ion
schemes: LGC, FME, and NFME. The da ase used is M2B.
Se Me hod MAE ↓RMSE ↓PC %↑
Eas e n
LGC 0.1502 0.1827 20.51
FME 0.1352 0.1668 44.82
NFME 0.1358 0.1671 44.55
Wes e n
LGC 0.1438 0.1742 34.99
FME 0.1141 0.1422 63.38
NFME 0.1132 0.1424 63.22
Bo h
LGC 0.1484 0.1801 22.90
FME 0.1346 0.1665 43.58
NFME 0.1303 0.1624 48.05
Table 7: MAEs ob ained wi h supe ised schemes and he p oposed h ee g aph-based
sco e p opaga ion schemes: LGC, FME, and NFME. The da ase used is M2B.
Me hod Eas e n Wes e n
S a e-o - he a
1-NN 2.11 1.92
Ridge Reg ession 1.95 1.87
Neu al Ne wo k 1.80 1.76
F-A-T 1.80 1.69
DFAT 1.77 1.66
Ou schemes
LGC 1.50 1.42
FME 1.35 1.14
NFME 1.35 1.13
17
4.3 SCUT-FBP5500 da ase
In his sec ion, we p esen he esul s ob ained on he SCUT-FBP5500 da ase . Table 8
shows he esul s o applying six di e en me hods on he SCUT-FBP5500 da ase . This
able depic s he pe o mance ob ained wi h i e di e en expe imen s. The i s ou
expe imen s co espond o lea ning om a gi en gende and e hnici y. These expe imen s
co espond o Asian Female da a (2,000 images), Asian Male da a (750 images), Caucasian
Female da a (2,000 images), and Caucasian Male da a (750 images), espec i ely. The i h
expe imen co esponds o he use o he whole da ase (5,500 images). All expe imen s
we e conduc ed using he i e- old c oss alida ion scheme in which 80% o images a e
used o aining and he emaining 20% a e used o es ing. The ea u es a e gi en by
he laye c6 o he VGG-Face ne .
F om he esul s depic ed in Table 8, we can obse e ha he bes supe ised me hod
was he non-linea -SVR me hod, and he bes semi-supe ised me hod was he NFME
me hod. The bes PCs we e ob ained o Asian Female and Caucasian Female models.
This can be explained by he ac ha hese wo da ase s ha e 2000 images each, making
he size o labeled images equal o 1600 images. On he o he hand, he size o he labeled
images o he Asian Male and Caucasian Male cases is 600 images. I is in e es ing
o no e ha he model lea ned on a mix u e o gende s and e hnici ies ha e p o ided a
pe o mance ha can be sligh ly wo se han ha ob ained in he o he cases. Ne e heless,
his pe o mance has no been signi ican ly de e io a ed. This can be plausible since
e hnici y and gende ha e di e en ep esen a ions.
4.4 Sco e-based p opaga ion e sus label dis ibu ion based p opaga ion
Fo some da ase s, he ace pho os may ha e label dis ibu ions ins ead o a single sco e.
In ou case, hese label dis ibu ions a e a ailable o SCUT-FBP and SCUT-FBP5500 in
which he dis ibu ion is o e i e le els o beau y. We compa e he pe o mance o he
NFME me hod when i p opaga es single alued sco es and when i p opaga es he label
dis ibu ion o e he same g aph. The NFME was chosen since i ga e he bes esul s
among he semi-supe ised me hods and since i can pe o m a basic label dis ibu ion
p opaga ion. When he NFME me hod is used o label dis ibu ion p opaga ion, he
label dis ibu ions o labeled images a e gi en by he label ma ix Y, and he unknown
so label dis ibu ions a e gi en by he ma ix F. In hese cases, each o hese ma ices
has i e columns. In o de o compa e he pe o mances o he NFME ha ou pu s single
alued p edic ed sco es and he NFME ha ou pu s label dis ibu ions, he la e a e
con e ed o a single alued sco e. This is achie ed by a e aging he le els using he
ob ained p obabili y dis ibu ion.
Table 9 summa izes he pe o mance o he wo NFME a ian s on he Asian-Female
da a o he SCUT-FBP5500 da ase . These esul s we e ob ained using he i e- old c oss
alida ion scheme. As i can be seen, he pe o mance o he a ian ha es ima es label
dis ibu ion is wo se han he a ian ha p opaga es a single alued sco e. One plausible
explana ion is ha NFME es ima es so class label dis ibu ion ha can be mo e sui able
o disc e e classi ica ion. While label dis ibu ion can be use ul o unce ain y es ima ion,
i is s ill no clea wha can be gained when a single alued sco e should be used.
18
Table 8: A e age pe o mances ob ained wi h h ee g aph-based sco e p opaga ion
schemes: LGC, FME, and NFME. The da ase used is SCUT-FBP5500.
Subse Me hod MAE ↓RMSE ↓PC %↑-e o ↓
Asian Female
1-NN 0.0923 0.1206 63.09 0.2610
Ridge Reg ession 0.0580 0.0742 85.41 0.1362
Gaussian -SVR 0.0550 0.0714 86.57 0.1264
LGC 0.1202 0.1427 58.52 0.3554
FME 0.0564 0.0707 86.09 0.1309
NFME 0.0550 0.0713 86.72 0.1256
Asian Male
1-NN 0.0863 0.1182 59.29 0.2248
Ridge Reg ession 0.0569 0.0739 82.88 0.1249
Gaussian -SVR 0.0538 0.0701 84.68 0.1136
LGC 0.1032 0.1316 53.08 0.2768
FME 0.0557 0.0722 83.72 0.1202
NFME 0.0540 0.0704 84.80 0.1150
Caucasian Female
1-NN 0.0825 0.1080 70.71 0.2133
Ridge Reg ession 0.0624 0.0778 83.98 0.1395
Gaussian -SVR 0.0550 0.0701 87.27 0.1161
LGC 0.1226 0.1424 62.64 0.3553
FME 0.0554 0.0701 87.07 0.1172
NFME 0.0553 0.0702 87.52 0.1181
Caucasian Male
1-NN 0.0752 0.0978 68.71 0.2003
Ridge Reg ession 0.0567 0.0733 81.32 0.1316
Gaussian -SVR 0.0505 0.0650 85.53 0.1064
LGC 0.0983 0.1232 59.50 0.2721
FME 0.0506 0.0650 85.46 0.1071
NFME 0.0525 0.0663 85.67 0.1173
Whole da ase
1-NN 0.0864 0.1153 64.52 0.2330
Ridge Reg ession 0.0570 0.0736 84.50 0.1293
Gaussian -SVR 0.0539 0.0693 86.41 0.1167
LGC 0.1137 0.1376 56.45 0.3239
FME 0.0570 0.0734 84.60 0.1283
NFME 0.0535 0.0691 86.60 0.1151
Table 9: Pe o mance o wo NFME a ian s on Asian-Female da a o he SCUT-FBP5500
da ase .
MAE ↓RMSE ↓PC ↑-e o ↓
NFME (sco e) 0.0550 0.0713 86.72 0.1256
NFME (label dis .) 0.0750 0.0926 85.35 0.1714
19
5 Conclusions and discussions
Cu en me hods o ace beau y assessmen a e ully supe ised. A limi a ion o hese
app oaches is he sca ci y o labeled ace images. The pape has in oduced wo main
con ibu ions. Fi s ly, semi-supe ised pa adigms a e p oposed o he ace beau y p e-
dic ion p oblem. We exploi g aph-based sco e p opaga ion me hods in o de o en ich
model lea ning wi hou he need o addi ional labeled ace images. Secondly, a non-linea
lexible mani old embedding o sol ing he sco e p opaga ion was p oposed.
The p oposals we e es ed on h ee public da ase s o ace beau y analysis: SCUT-
FBP, M2B, and SCUT-FBP5500. These expe imen s as well as many compa isons wi h
supe ised schemes show ha he non-linea semi-supe ised scheme compa es a o ably
wi h he bes supe ised scheme.
Ob iously, he p oposed semi-supe ised schemes ha e he limi a ion ha he simi-
la i y g aph should be compu ed p io o he es ima ion o he beau y p edic ion model,
namely he semi-supe ised lea ning model. In o he wo ds, i he g aph quali y is bad
(e.g., he g aph is e y dense), he p edic ion accu acy migh be a ec ed. The e o e, as
a u u e wo k we en ision o o e come his limi a ion by deploying a join es ima ion o
he pai wise simila i y g aph and he unknown p edic ion model. The o he limi a ion is
ela ed o he da a hemsel es. By na u e he ace beau y p edic ion p oblem su e s om
imbalanced da a. Fo ins ance, i is well known ha he numbe o aces wi h a e age
a ac i eness is e y high. A emedy o his limi a ion is o use s a e-o - he-a da a
augmen a ion echniques in o de o inc ease he he leas and mos a ac i e aces.
The p oposed semi-supe ised sco ing amewo k opens he doo o o he image-based
applica ions. I pa es he way o i ually all applica ions o adop con inuous sco es
ins ead o he usual disc e e labels ha a e usually used. These applica ions can be
pain le el es ima ion, d i e d owsiness de ec ion, subjec concen a ion de ec ion, acial
exp ession ecogni ion, e c. In addi ion, all applica ions whose na u al ou pu is a numbe
(age es ima ion, numbe o p esen objec s, e c.) can di ec ly bene i om he p oposed
amewo k.
Fu u e wo k may en ision explo ing he ollowing esea ch di ec ions. Fi s ly, we may
exploi image simila i ies in o de o ec i y some human gene a ed g ound- u h beau y
sco es. Secondly, we may a ge he join es ima ion o he pai wise simila i y g aph
and he unknown p edic ion model. Thi dly, we would use mul iple iews as inpu s (i.e.,
mul iple image desc ip o s) o he objec i e unc ional ha es ima es he p edic ion model
in o de o imp o e he p edic ion esul s o e he use o one single desc ip o . In his case,
he unknown model will be es ima ed om a used g aph whe e he usion is ca ied ou by
au o-weigh ed schemes. Fou hly, we in end o use mul i- ask es ima ion o sol ing he
ace beau y p edic ion p oblem in a mo e accu a e way. In his kind o schemes, se e al
ou pu s a e simul aneously es ima ed. Fo ins ance, he ou pu can be he gende , he
e hnici y and he ace beau y sco e.
20
Re e ences
da a ang. u l: h p://da a ang.com/.
Bainb idge, W., Isola, P., and Oli a, A. (2013). The in insic memo abili y o ace images.
Jou nal o Expe imen al Psychology: Gene al, 142(4):1323–1334.
Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps o dimensionali y educ ion and
da a ep esen a ion. Neu al Compu ., 15(6):1373–1396.
Belkin, M., Niyogi, P., and Sindhwani, V. (2006). Mani old egula iza ion: A geome ic
amewo k o lea ning om labeled and unlabeled examples. The Jou nal o Machine
Lea ning Resea ch, 7:2399–2434.
Cheng, B., Yang, J., Yan, S., Fu, Y., and Huang, T. (2010). Lea ning wi h l1-g aph o
image analysis. T ans. Img. P oc., 19(4):858–866.
Do naika, F. and Bosaghzadeh, A. (2015). Adap i e g aph cons uc ion using da a sel -
ep esen a i eness o pa e n classi ica ion. In o ma ion Sciences, 325:118–139.
Do naika, F., Dhabi, R., Bosaghzadeh, A., and Ruichek, Y. (2016). Dynamic adap i e
g aph cons uc ion: Applica ion o g aph-based mul i-obse a ion classi ica ion. In
IEEE In e na ional Con e ence on Pa e n Recogni ion.
Do naika, F. and El T aboulsi, Y. (2016). Lea ning lexible g aph-based semi-supe ised
embedding. IEEE ansac ions on cybe ne ics, 46(1):206–218.
Do naika, F., T aboulsi, Y. E., and Assoum, A. (2013). Adap i e wo phase spa se ep-
esen a ion classi ie o ace ecogni ion. In LNCS 8192. Ad anced Concep s o
In elligen Vision Sys ems.
He, R., Zheng, W.-S., Hu, B.-G., and Kong, X.-W. (2011). Nonnega i e spa se coding o
disc imina i e semi-supe ised lea ning. In Compu e Vision and Pa e n Recogni ion
(CVPR), 2011 IEEE Con e ence on, pages 2849–2856.
Kla e, B. and Jain, A. (2013). He e ogeneous ace ecogni ion using ke nel p o o ype simi-
la i ies. IEEE T ansac ions on Pa e n Analysis and Machine In elligence, 2(6):1410–
1422.
Liang, L., Lin, L., Jin, L., Xie, D., and Li, M. (2018). SCUT-FBP5500: A di e se
benchma k da ase o mul i-pa adigm acial beau y p edic ion. a Xi :1801.06345 1
[cs.CV] 19 Jan 2018.
Liu, W. and Chang, S.-F. (2009). Robus mul i-class ansduc i e lea ning wi h g aphs.
In Compu e Vision and Pa e n Recogni ion, 2009. CVPR 2009. IEEE Con e ence
on, pages 381–388. IEEE.
Nguyen, T. V., Liu, S., Ni, B., Tan, J., Rui, Y., and Yan, S. (2013). Towa ds dec yp ing
a ac i eness ia mul i-modali y cues. ACM T ansac ions on Mul imedia Compu ing,
Communica ions, and Applica ions (TOMM), 9(4):28.
21

Nie, F., Wang, X., Jo dan, M. I., and Huang, H. (2016). The cons ained laplacian ank
algo i hm o g aph-based clus e ing. In AAAI Con e ence on A i icial In elligence.
Nie, F., Xu, D., Tsang, I. W.-H., and Zhang, C. (2010). Flexible mani old embedding: A
amewo k o semi-supe ised and unsupe ised dimension educ ion. IEEE T ans-
ac ions on Image P ocessing, 19(7):1921–1932.
Pa khi, O. M., Vedaldi, A., Zisse man, A., e al. (2015). Deep ace ecogni ion. In BMVC,
olume 1, page 6.
Raducanu, B. and Do naika, F. (2014). Embedding new obse a ions ia spa se-coding
o non-linea mani old lea ning. Pa e n Recogni ion, 47(1):480–492.
Sha i Raza ian, A., Azizpou , H., Sulli an, J., and Ca lsson, S. (2014). Cnn ea u es
o - he-shel : an as ounding baseline o ecogni ion. In P oceedings o he IEEE
con e ence on compu e ision and pa e n ecogni ion wo kshops, pages 806–813.
Sousa, C., Rezende, S., and Ba is a, G. (2013). In luence o g aph cons uc ion on semi-
supe ised lea ning. In Eu opean Con e ene on Machine Lea ning, pages 160–175.
Xie, D., Liang, L., Jin, L., Xu, J., and Li, M. (2015). Scu - bp: A benchma k da ase
o acial beau y pe cep ion. In Sys ems, Man, and Cybe ne ics (SMC), 2015 IEEE
In e na ional Con e ence on, pages 1821–1826. IEEE.
Zhang, Z., Liu, L., Shen, F., Shen, H. T., and Shao, L. (2018). Bina y mul i- iew clus e -
ing. IEEE T ansac ions on Pa e n Analysis and Machine In elligence.
Zhou, D., Bousque , O., Lal, T. N., Wes on, J., and Sch¨olkop , B. (2004). Lea ning wi h
local and global consis ency. In Ad ances in neu al in o ma ion p ocessing sys ems,
pages 321–328.
Zhu, X., Ghah amani, Z., and La e y, J. (2003). Semi-supe ised lea ning using gaussian
ields and ha monic unc ions. pages 912–919.
22