Design o a Realis ic Robo ic Head Based on
Ac ion Coding Sys em
Samuel Ma cos1, Robe o Pinillos1, Jaime G´omez Ga c´ıa-Be mejo2, and
Edua do Zalama2
1Fundaci´on Ca i .
Pa que Tecnol´ogico de Boecillo Pa cela 205.
47011 Boecillo, Valladolid, Spain
[email p o ec ed],[email p o ec ed]
2Ins i u o de las Tecnolog´ıas de la P oducci´on, Uni e si y o Valladolid.
Paseo del Cauce 59, 47011 Valladolid, Spain
[email p o ec ed],[email p o ec ed]
Abs ac . In his pape , he de elopmen o a obo ic head able o mo e
and show di e en emo ions is add essed. The mo emen and emo ion
gene a ion sys em has been designed ollowing he human acial muscu-
la u e. S a ing om he Facial Ac ion Coding Sys em (FACS), we ha e
buil a 26 ac ions uni s model ha is able o p oduce he mos ele an
mo emen s and emo ions o a eal human head. The whole wo k has
been ca ied ou in wo s eps. In he i s s ep, a mechanical skele on
has been designed and buil , in which he di e en ac ua o s ha e been
inse ed. In he second s ep, a wo-laye ed silicon skin has been manu-
ac u ed, on which he di e en ac ua o s ha e been inse ed ollowing
he eal muscle-inse ions, o pe o ming he di e en mo emen s and
ges u es. The de eloped head has been in eg a ed in a high le el be-
ha iou al a chi ec u e, and pilo expe imen s wi h 10 use s ega ding
emo ion ecogni ion and mimicking ha e been ca ied ou .
Keywo ds: FACS, mecha onic head, emo ional exp essions, pilo s udy
1 INTRODUCTION
The de elopmen o obo s wi h social skills is one o he challenges o which
mo e e o s has been de o ed in he esea ch cen e s, du ing he las yea s. These
obo s should be cha ac e ized by he abili y o ansmi ing isual in o ma ion
h ough ges u es, in o de o achie e a na u al human- o- obo communica ion
simila o he human- o-human one.
Howe e , esea che s ha e no eached a gene al consensus abou wha a
obo ’s appea ance should be. The e is a la ge numbe o pa icula complica-
ions associa ed o he design o mo e ealis ic social obo s [2], and al hough a
human appea ance may imp o e he in e ac ion so o o e come he limi a ions
imposed by he Uncanny Valley Theo y [3], he e a e always incon eniences asso-
cia ed wi h he beha io al expec a ions ha an an h opomo phic obo c ea es
in he use s [4].
2 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama
Rega ding he deg ee o ealism expec ed o a humanoid obo , in [6] he
need o use mo e sophis ica ed ways o exp ess emo ions in i ual en i onmen s
was deno ed. Also, he esul s p esen ed in [7] show ha li le exp essi e agen s
had mo e di icul ies o a ain an e ec i e in e ac ion. On he o he hand, in
[8] simple d awings o aces we e able o c ea e emo ional eelings on he use .
The ace is a highly exp essi e elemen ha humans end o in e p e in ace
o ace communica ion, so highly ealis ic a i icial aces may con use he use i
hey a e no designed p ope ly [7]. Acco ding o [9], he e is e idence ha he
use o exagge a ed ca oonish aces can exp ess emo ions mo e accu a ely han
ealis ic ones.
Since he eme ge o humanoid obo s, he use o un ealis ic o ca oonish-like
humanoid obo s has been jus i ied in e ms o a oiding well-known uncanny al-
ley. Howe e , di e en s udies show ha use p e e ences in e ms o appea ance
and beha io a e no uni e sal and may be e ol ing as he echnology sp eads.
These p e e ences may di e depending on cul u al and psychological aspec s
[10] [11] [12], as well as on age o gende (men s. women, elde ly s. young
people).
The i s de elopmen s o exp essi e humanoid obo s ha e been designed
using a ca oonish simple ap oach [16]. One o he i s de elopmen s is he
Kisme obo [17]. This obo is equipped whi h an un ealis ic ace wi h la ge
exp essi e eyes, eyeb ows and mou h. Ano he example is he Robo WE-4R
Waseda [18] which can ou pu ich emo ional exp essions and beha io by using
a simplis ic ace, wais , a ms and neck. Howe e , al hough hese obo s exhibi
la ge exp essi eness, hey lack o he abili y o exp ess de ailed emo ional cues.
I is necessa y o inco po a e mo emen s ha mimic human ges u es in g ea e
de ail, as is e iden in he de elopmen o obo s wi h humanoid appea ance,
like he and oid HRP-4C [19] p esen ed by he Na ional Ins i u e o Ad anced
Indus ial Science and Technology (AIST) o Japan, o he DER2 and DER3
p ojec s cons uc ed by Osaka Uni e si y.
The e exis o he app oaches ha y o emula e he human ace s uc u e
a a g ea e de ail. Fo ins ance, in he Geminoid obo [13] and [20] hey use
pneuma ic ac ua o s o display acial exp essions. In [21] an in e es ing emo ional
a chi ec u e is p esen ed, and in [22] he muscula s uc u e o he human ace
is imi a ed wi h g ea de ail. Howe e , he ob ained esul s ei he do no ha e a
na u al appea ance o a e e y ha d o implemen .
The p oposed app oach desc ibed in his pape also looks o ob aining a
ealis ic mecha onic ace. We ollow he biomechanic p inciples o he human
ace based on mic oexp essions. Ou goal is o ob ain isually ealis ic esul s, so
he ana omical ea u es o a eal ace a e aken in o accoun in he design o he
obo ic head. Ou app oach is simila o Hansons wo k [14],[15] as he gene a ion
ealis ic emo ional exp essions is based on he human muscula model, bu while
Hansons model uses 32 se o mo o s coupled o a mask o map he acial muscles,
we ha e de eloped a simpli ied model using 22 se omo o s ha can be engaged
and disengaged easily om he acial mask, hus allowing o use di e en masks
Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 3
Table 1. Rele an muscles o he human ace.
a ea Mos signi ican muscles
Fo ehead and eyeb ows Occipi alis and on alis
Eyes and eyelids Ele a o palpeb ae supe io is, o -
bicula is o is, co uga o supe cilii
Lips and mou h Quad a us labii in e io is, Quad a-
us labii supe io is, iangula is, zy-
goma icus majo and mino , bucci-
na o , o bicula is o is, caninus
Nose Py amidalis ans e se nos il di-
la a o , ele a o muscles, ans e se
Ea s Supe io and pos e io au icula is
o he same c anial s uc u e. Ou app oach also eases he in eg a ion o he
obo ic head wi h a high le el con ol a chi ec u e de eloped in ou lab.
2 The human acial muscula u e
The human skull includes 14 bones o gi e suppo o he ace. The acial mus-
cula u e, in u n, is ela ed di ec ly o he di e en exp essions he ace can
show. The main ace muscles in ol ed in o acial ges u es, along wi h he ace
a eas hey belong o, a e shown in able 1. The synch onized mo emen o hese
muscles allows he di e en ges u es and emo ions o be pe o med.
2.1 FACS - Facial Ac ion Code Sys em
The ela ionship be ween a gi en muscle con ac ion and he co esponding
shape o he ace su ace can be desc ibed upon he muscula ana omy. How-
e e , om he iewpoin o social in e ac ion, he exis ing ela ionship among
he acial exp essions, wha hose exp essions mean and wha o he s can in e -
p e om hem should be es ablished. To his end, social psychology esea che s
ha e add essed he de elopmen o s anda ds o coding and pa ame e izing he
acial mo emen s objec i ely. This way, emo ions and acial exp essions can be
linked upon speci ic pa e ns o acial mo emen s.
Some ecognised s anda s a e: EMFACS [23], Monadic Phases [24], he Max-
imally Disc imina i e Facial Mo emen Coding Sys em [25], and he Facial Ac-
ion Coding Sys em [26].
The (Facial Ac ion Coding Sys em, FACS) [26] is one o he mos widesp ead
s anda d used o measu ing and desc ibing he ace beha io .
The FACS is s uc u ed in o Ac ion Uni s, AU. An AU is a con ac ion o
elaxa ion o a gi en se o muscles ha esul s in o a change in he ace appea -
ance. The FACS is no based on he ac i i y o isola ed muscles because ce ain
muscles may ac ua e mo e han once du ing he p oduc ion o a gi en ges u e.
Mo eo e , he ace appea ance changes a e weakly dependen on he mo emen
4 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama
Table 2. Common AUs in ol ed in he uni e sal exp essions.
Uni e sal Exp ession AUs
Joy 6+12
Sadness 1+4+15
Su p ise 1+2+5+26
Fea 1+2+4+5+20+26
Ange 4+5+7+23
Disgus 9+15+16
o an isola ed muscle. The e a e mo e han 70 AUs in ol ed in he mo emen o
a human head and ace. 46 AUs a e linked speci ically o he con ac ion o acial
muscles, and 30 o hem a e linked o he con ac ion o speci ic ace muscles:
12 o he uppe ace and 18 o he lowe ace.
Acco ding o Ekman, he e a e 6 uni e sal exp esions: happiness, disgus ,
sadness, ange , ea and su p ise. Each exp ession is de ined (al hough no com-
ple ely, as we will discuss la e ) by a gi en numbe o AUs. The uni e sal exp e-
sions and he mos common AUs associa ed a e lis ed in able 2.
3 Mechanical design
3.1 Full model
Figu e 1 shows side and on iews o he ull model. A silicon skin is hen added
o his model, o comple e he ealis ic look o he obo ic head. Fo a comple e
desc ip ion o he mechanical s uc u e o he head please see [28].
Fig. 1. Side and on iew o he head.
Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 5
(a) (b)
Fig. 2. (a). Head wi h mask unde laye and silicon mask. (b) Molds employed o he
mask cons uc ion.
3.2 Face mask
Fi s , a plas e model o he desi ed head has been c ea ed. As ou obo is
in ended o in e ac wi h aged people, a ma u e aged ace has been selec ed. In
addi ion, he w inkles ease pe o ming he di e en ges u es. Then, a nega i e
mold has been gene a ed ha will be used o p oducing he ou e su ace o
he mask. A mold co esponding o he inne su ace o he mask has been also
gene a ed. This mold p o ides de equi ed hickness o he di e en a eas o
he mask acco ding o he en isaged beha io o he skin du ing he se omo o
ac ions. Bo h ou e and inne molds can be seen in igu e 2.
Figu e 3 shows he inal look o he head, wi h he skin on, a e he pain ing
he ou e su ace and including addi ional elemen s such as eyeb ows, eyelashes
and acial hai . An in e media e oam laye has been added be ween he me-
chanical s uc u e and he mask in o de o ill he s uc u e gaps and acili a e
he skin mo emen s (see igu e 2).
Fig. 3. Final look o he head wi h he silicon mask.

6 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama
4 Emo ion gene a ion
The p ominen biologis Cha les Da win was he i s o sugges ha he ex-
p essions co esponding o he main p ima y emo ions can be easily pe cei ed
and ca ego ized in o di e en ges u e pa e ns pe o med by he human ace [1].
Following his wo k, Ekman [26] pos ula ed ha he e a e six uni e sal exp es-
sions co esponding o he six p ima y emo ions exp essions: happiness, disgus ,
sadness, ange , ea and su p ise. In he p esen wo k we ha e modeled he six
emo ional exp essions as a linea combina ion o he in ol ed AUs. Each AU j
is gi en a weigh alue ha de ines i s le el o ac i a ion in he cu en ges u e,
acco ding o he desi ed in ensi y o his ges u e. Mo eo e , a gain e m Kiis
added which de ines he in ensi y o he o e all ges u e.
ComplexExp essioni=KiX(weigh j∗AUj) (1)
(a) (b)
(c) (d)
(e) ( )
Fig. 4. Example o AUs
(a) Neu al (b) AU4 (c) Neu al (d) AU12 (e) Neu al ( ) AU5
Figu e 4 shows he basic emo ional exp essions pe o med by he obo . The
gene a ion o he happiness exp ession in ol es he coo dina ed ac i a ion o
AUs UA6+UA12+UA25+UA26 ha co espond o li ing he cheeks, pulling
up he mou h co ne s, sepa a ing he lips and lowe ing he jaw. The inal esul
can be seen in igu e 5.a.
Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 7
Fo ins ance, he sadness exp ession in ol es he ac i a ion o AU1+AU17
+AU54+AU55+AU63 ha co espond o aising he inne pa o he eyeb ows,
aising he chin and il ing o wa d he head and u ning i le and mo ing he
eyes up. (Figu e 5.b).
The ang iness exp ession shown in ol es he ac i a ion o AU4+AU7+AU10
+AU16+AU55+AU63 ha co espond o lowe ing he eyeb ows, na owing he
eyelid ape u e, aising he uppe lip, dep essing he lowe lip, il ing o wa d
he head and sligh ly mo ing he eyes up. (Figu e 5.c).
(a) (b)
(c) (d)
(e) ( )
Fig. 5. Basic Emo ions
(a) Joy (b) Sadness (c) Ange (d) Fea (e) Disgus ( ) Su p ise
8 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama
5 In eg a ion in a mul imodal a chi ec u e
Cu en ly he e is a g owing in e es in modula a chi ec u es and componen
amewo ks ha encou age he c ea ion o easily modi iable and scalable open
sys ems. This kind o ini ia i e is o pa icula in e es in he ield o mecha on-
ics and se ice obo ics, whe e sys ems a e speci ically designed o high le el
in e ac ions and sough o be easily modi ied and imp o ed o mee new equi e-
men s. The main end hese days in e ms o de elopmen o obo ic sys ems
is he design o a chi ec u es based on componen s o amewo ks. Fo his pu -
pose ules o combining componen s and he in e ac ion pa e ns be ween hem
a e de ined in o de o achie e a lexible sys em. This lexibili y is unde s ood
mainly in he ollowing aspec s:
- Easy eplacemen o componen s wi h di e en unc ionali y.
- Reuse o di e en modules in o he p ojec s whe e he unc ionali y o hose
pa icula modules is equi ed.
The po en ial o he open modula a chi ec u es is huge, and leads o econ-
igu able, po able and ex ensible sys ems. This philosophy has been adop ed o
de elop he con ol so wa e o di e en social obo s in ou labo a o y. Based
in he ROS amewo k, i consis s o a se o laye s wi h di e en in e change-
able modules (see Figu e 6). The in e ac ion be ween laye s and modules is well
de ined, so a new module can easily be in eg a ed. I has o be no ed ha his
a chi ec u e has no only been used in obo s, bu also in o he agen s such
as anima ed aces [5]. Thanks o he AU based emo ional gene a ion app oach,
bo h anima ed and obo ic aces can be con olled using he same uppe le el
componen s.
Fig. 6. Modula mul imodal a chi ec u e
Making use o his a chi ec u e, he mecha oinc head and i s low le el con-
ol d i e s ha e been in eg a ed wi hin he down laye , aking ad an age o
Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 9
he uppe laye se ices such as oice ecogni ion, exp ession ecogni ion o be-
ha iou al planne .
As can be seen in he ideo linked o his a icle (see Sec ion 7 below), as
an example he obo ic head has been easily connec ed o ou p e iously de el-
oped acial exp ession ecogni ion sys em. This sys em si able o pe o m bo h:
acial exp ession ecogni ion and acial emo ion ecogni ion. Exp ession ecog-
ni ion e e s o he de ec ion and analysis o acial mo emen s, and changes in
acial ea u es om isual in o ma ion. On he o he hand, emo ional exp ession
ecogni ion also associa es hese acial displacemen s o hei emo ional signi -
icance, gi en ha he e will be ce ain mo emen s associa ed wi h mo e han
one emo ion. The p oposed me hod [5] is di ided in o wo di e en s ages: he
i s one ocuses on he de ec ion and ecogni ion o small g oups o acial muscle
ac ions using a combina ion o Gabo il e s and Ac i e Shape Models. Based on
he ou pu o his i s s age, he second de e mines he associa ed emo ional ex-
p ession using an habi ua ion based ne wo k plus a compe i i e based ne wo k.
Thanks o he implemen ed con ol a chi ec u e, i can be obse ed in he ideo
how he mecha onic head is able o mimic he emo ional exp essions o he use
using a egula webcam.
6 Pilo expe imen s in emo ion ecogni ion
The mecha onic head exp ession gene a ion capabili ies we e es ed in a pilo
s udy wi h 10 pa icipan s which ages anged be ween 28 and 40 yea s old. In
o de o see he in luence o he in ensi y pa ame e ki(see ecua ion 1) in he
gene a ion o ecognisable exp essions, use s we e asked o ecognize he six
uni e sal exp essions unde wo condi ions: hal - in ensi y (ki= 0.5), and ull-
in ensi y (ki= 1). Exp essions we e andomly gene a ed o each condi ion and
use s we e asked o ano a e which exp essions hey we e able o ecognize, as well
as he o de he mecha onic head had displayed hem. The subjec s answe s in
labeling an exp ession we e sco ed as co ec o w ong and esul s a e shown in
igu e 7.
F om he esul s i can be obse ed ha he in ensi y has a g ea impac
in he ecogni ion o he majo i y o he exp essions, specially in he ange ex-
p ession. Apa om he joy exp ession, a g ea e in ensi y esul s in a highe
ecogni ion a e, al hough in exp essions such as sadness o disgus he di e -
ence is no highly signi ican . I has o be no ed ha igu e 7 jus shows he
co ec answe a io and no hose answe s in which use s doub ed be ween wo
exp essions. This happened be ween he ange and disgus exp essions and be-
ween he ea and disgus exp essions. Use s ound i di icul o dis inguish
be ween hem, and labeled he shown exp essions as bo h ange and disgus o
ea and su p ise. I can be seen ha his con usion is e en mo e accen ua ed in
he case o he hal -in ensi y exp essions. I has o be no ed, howe e , ha he
small numbe o pa icipan s make some o he ob ained esul s a om being
signi ican . Fo ins ance, ecogni ion a es o 3 ou o 10 a e close o andomness
meaning ha use s we e no able o ecognize he exp ession. In o de o ha