Design of a Realistic Robotic Head based on Action Coding System

Design o a Realis ic Robo ic Head Based on Ac ion Coding Sys em Samuel Ma cos1, Robe o Pinillos1, Jaime G´omez Ga c´ıa-Be mejo2, and Edua do Zalama2 1Fundaci´on Ca i . Pa que Tecnol´ogico de Boecillo Pa cela 205. 47011 Boecillo, Valladolid, Spain [email p o ec ed],[email p o ec ed] 2Ins i u o de las Tecnolog´ıas de la P oducci´on, Uni e si y o Valladolid. Paseo del Cauce 59, 47011 Valladolid, Spain [email p o ec ed],[email p o ec ed] Abs ac . In his pape , he de elopmen o a obo ic head able o mo e and show di e en emo ions is add essed. The mo emen and emo ion gene a ion sys em has been designed ollowing he human acial muscu- la u e. S a ing om he Facial Ac ion Coding Sys em (FACS), we ha e buil a 26 ac ions uni s model ha is able o p oduce he mos ele an mo emen s and emo ions o a eal human head. The whole wo k has been ca ied ou in wo s eps. In he i s s ep, a mechanical skele on has been designed and buil , in which he di e en ac ua o s ha e been inse ed. In he second s ep, a wo-laye ed silicon skin has been manu- ac u ed, on which he di e en ac ua o s ha e been inse ed ollowing he eal muscle-inse ions, o pe o ming he di e en mo emen s and ges u es. The de eloped head has been in eg a ed in a high le el be- ha iou al a chi ec u e, and pilo expe imen s wi h 10 use s ega ding emo ion ecogni ion and mimicking ha e been ca ied ou . Keywo ds: FACS, mecha onic head, emo ional exp essions, pilo s udy 1 INTRODUCTION The de elopmen o obo s wi h social skills is one o he challenges o which mo e e o s has been de o ed in he esea ch cen e s, du ing he las yea s. These obo s should be cha ac e ized by he abili y o ansmi ing isual in o ma ion h ough ges u es, in o de o achie e a na u al human- o- obo communica ion simila o he human- o-human one. Howe e , esea che s ha e no eached a gene al consensus abou wha a obo ’s appea ance should be. The e is a la ge numbe o pa icula complica- ions associa ed o he design o mo e ealis ic social obo s [2], and al hough a human appea ance may imp o e he in e ac ion so o o e come he limi a ions imposed by he Uncanny Valley Theo y [3], he e a e always incon eniences asso- cia ed wi h he beha io al expec a ions ha an an h opomo phic obo c ea es in he use s [4]. 2 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama Rega ding he deg ee o ealism expec ed o a humanoid obo , in [6] he need o use mo e sophis ica ed ways o exp ess emo ions in i ual en i onmen s was deno ed. Also, he esul s p esen ed in [7] show ha li le exp essi e agen s had mo e di icul ies o a ain an e ec i e in e ac ion. On he o he hand, in [8] simple d awings o aces we e able o c ea e emo ional eelings on he use . The ace is a highly exp essi e elemen ha humans end o in e p e in ace o ace communica ion, so highly ealis ic a i icial aces may con use he use i hey a e no designed p ope ly [7]. Acco ding o [9], he e is e idence ha he use o exagge a ed ca oonish aces can exp ess emo ions mo e accu a ely han ealis ic ones. Since he eme ge o humanoid obo s, he use o un ealis ic o ca oonish-like humanoid obo s has been jus i ied in e ms o a oiding well-known uncanny al- ley. Howe e , di e en s udies show ha use p e e ences in e ms o appea ance and beha io a e no uni e sal and may be e ol ing as he echnology sp eads. These p e e ences may di e depending on cul u al and psychological aspec s [10] [11] [12], as well as on age o gende (men s. women, elde ly s. young people). The i s de elopmen s o exp essi e humanoid obo s ha e been designed using a ca oonish simple ap oach [16]. One o he i s de elopmen s is he Kisme obo [17]. This obo is equipped whi h an un ealis ic ace wi h la ge exp essi e eyes, eyeb ows and mou h. Ano he example is he Robo WE-4R Waseda [18] which can ou pu ich emo ional exp essions and beha io by using a simplis ic ace, wais , a ms and neck. Howe e , al hough hese obo s exhibi la ge exp essi eness, hey lack o he abili y o exp ess de ailed emo ional cues. I is necessa y o inco po a e mo emen s ha mimic human ges u es in g ea e de ail, as is e iden in he de elopmen o obo s wi h humanoid appea ance, like he and oid HRP-4C [19] p esen ed by he Na ional Ins i u e o Ad anced Indus ial Science and Technology (AIST) o Japan, o he DER2 and DER3 p ojec s cons uc ed by Osaka Uni e si y. The e exis o he app oaches ha y o emula e he human ace s uc u e a a g ea e de ail. Fo ins ance, in he Geminoid obo [13] and [20] hey use pneuma ic ac ua o s o display acial exp essions. In [21] an in e es ing emo ional a chi ec u e is p esen ed, and in [22] he muscula s uc u e o he human ace is imi a ed wi h g ea de ail. Howe e , he ob ained esul s ei he do no ha e a na u al appea ance o a e e y ha d o implemen . The p oposed app oach desc ibed in his pape also looks o ob aining a ealis ic mecha onic ace. We ollow he biomechanic p inciples o he human ace based on mic oexp essions. Ou goal is o ob ain isually ealis ic esul s, so he ana omical ea u es o a eal ace a e aken in o accoun in he design o he obo ic head. Ou app oach is simila o Hansons wo k [14],[15] as he gene a ion ealis ic emo ional exp essions is based on he human muscula model, bu while Hansons model uses 32 se o mo o s coupled o a mask o map he acial muscles, we ha e de eloped a simpli ied model using 22 se omo o s ha can be engaged and disengaged easily om he acial mask, hus allowing o use di e en masks Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 3 Table 1. Rele an muscles o he human ace. a ea Mos signi ican muscles Fo ehead and eyeb ows Occipi alis and on alis Eyes and eyelids Ele a o palpeb ae supe io is, o - bicula is o is, co uga o supe cilii Lips and mou h Quad a us labii in e io is, Quad a- us labii supe io is, iangula is, zy- goma icus majo and mino , bucci- na o , o bicula is o is, caninus Nose Py amidalis ans e se nos il di- la a o , ele a o muscles, ans e se Ea s Supe io and pos e io au icula is o he same c anial s uc u e. Ou app oach also eases he in eg a ion o he obo ic head wi h a high le el con ol a chi ec u e de eloped in ou lab. 2 The human acial muscula u e The human skull includes 14 bones o gi e suppo o he ace. The acial mus- cula u e, in u n, is ela ed di ec ly o he di e en exp essions he ace can show. The main ace muscles in ol ed in o acial ges u es, along wi h he ace a eas hey belong o, a e shown in able 1. The synch onized mo emen o hese muscles allows he di e en ges u es and emo ions o be pe o med. 2.1 FACS - Facial Ac ion Code Sys em The ela ionship be ween a gi en muscle con ac ion and he co esponding shape o he ace su ace can be desc ibed upon he muscula ana omy. How- e e , om he iewpoin o social in e ac ion, he exis ing ela ionship among he acial exp essions, wha hose exp essions mean and wha o he s can in e - p e om hem should be es ablished. To his end, social psychology esea che s ha e add essed he de elopmen o s anda ds o coding and pa ame e izing he acial mo emen s objec i ely. This way, emo ions and acial exp essions can be linked upon speci ic pa e ns o acial mo emen s. Some ecognised s anda s a e: EMFACS [23], Monadic Phases [24], he Max- imally Disc imina i e Facial Mo emen Coding Sys em [25], and he Facial Ac- ion Coding Sys em [26]. The (Facial Ac ion Coding Sys em, FACS) [26] is one o he mos widesp ead s anda d used o measu ing and desc ibing he ace beha io . The FACS is s uc u ed in o Ac ion Uni s, AU. An AU is a con ac ion o elaxa ion o a gi en se o muscles ha esul s in o a change in he ace appea - ance. The FACS is no based on he ac i i y o isola ed muscles because ce ain muscles may ac ua e mo e han once du ing he p oduc ion o a gi en ges u e. Mo eo e , he ace appea ance changes a e weakly dependen on he mo emen 4 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama Table 2. Common AUs in ol ed in he uni e sal exp essions. Uni e sal Exp ession AUs Joy 6+12 Sadness 1+4+15 Su p ise 1+2+5+26 Fea 1+2+4+5+20+26 Ange 4+5+7+23 Disgus 9+15+16 o an isola ed muscle. The e a e mo e han 70 AUs in ol ed in he mo emen o a human head and ace. 46 AUs a e linked speci ically o he con ac ion o acial muscles, and 30 o hem a e linked o he con ac ion o speci ic ace muscles: 12 o he uppe ace and 18 o he lowe ace. Acco ding o Ekman, he e a e 6 uni e sal exp esions: happiness, disgus , sadness, ange , ea and su p ise. Each exp ession is de ined (al hough no com- ple ely, as we will discuss la e ) by a gi en numbe o AUs. The uni e sal exp e- sions and he mos common AUs associa ed a e lis ed in able 2. 3 Mechanical design 3.1 Full model Figu e 1 shows side and on iews o he ull model. A silicon skin is hen added o his model, o comple e he ealis ic look o he obo ic head. Fo a comple e desc ip ion o he mechanical s uc u e o he head please see [28]. Fig. 1. Side and on iew o he head. Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 5 (a) (b) Fig. 2. (a). Head wi h mask unde laye and silicon mask. (b) Molds employed o he mask cons uc ion. 3.2 Face mask Fi s , a plas e model o he desi ed head has been c ea ed. As ou obo is in ended o in e ac wi h aged people, a ma u e aged ace has been selec ed. In addi ion, he w inkles ease pe o ming he di e en ges u es. Then, a nega i e mold has been gene a ed ha will be used o p oducing he ou e su ace o he mask. A mold co esponding o he inne su ace o he mask has been also gene a ed. This mold p o ides de equi ed hickness o he di e en a eas o he mask acco ding o he en isaged beha io o he skin du ing he se omo o ac ions. Bo h ou e and inne molds can be seen in igu e 2. Figu e 3 shows he inal look o he head, wi h he skin on, a e he pain ing he ou e su ace and including addi ional elemen s such as eyeb ows, eyelashes and acial hai . An in e media e oam laye has been added be ween he me- chanical s uc u e and he mask in o de o ill he s uc u e gaps and acili a e he skin mo emen s (see igu e 2). Fig. 3. Final look o he head wi h he silicon mask. 6 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama 4 Emo ion gene a ion The p ominen biologis Cha les Da win was he i s o sugges ha he ex- p essions co esponding o he main p ima y emo ions can be easily pe cei ed and ca ego ized in o di e en ges u e pa e ns pe o med by he human ace [1]. Following his wo k, Ekman [26] pos ula ed ha he e a e six uni e sal exp es- sions co esponding o he six p ima y emo ions exp essions: happiness, disgus , sadness, ange , ea and su p ise. In he p esen wo k we ha e modeled he six emo ional exp essions as a linea combina ion o he in ol ed AUs. Each AU j is gi en a weigh alue ha de ines i s le el o ac i a ion in he cu en ges u e, acco ding o he desi ed in ensi y o his ges u e. Mo eo e , a gain e m Kiis added which de ines he in ensi y o he o e all ges u e. ComplexExp essioni=KiX(weigh j∗AUj) (1) (a) (b) (c) (d) (e) ( ) Fig. 4. Example o AUs (a) Neu al (b) AU4 (c) Neu al (d) AU12 (e) Neu al ( ) AU5 Figu e 4 shows he basic emo ional exp essions pe o med by he obo . The gene a ion o he happiness exp ession in ol es he coo dina ed ac i a ion o AUs UA6+UA12+UA25+UA26 ha co espond o li ing he cheeks, pulling up he mou h co ne s, sepa a ing he lips and lowe ing he jaw. The inal esul can be seen in igu e 5.a. Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 7 Fo ins ance, he sadness exp ession in ol es he ac i a ion o AU1+AU17 +AU54+AU55+AU63 ha co espond o aising he inne pa o he eyeb ows, aising he chin and il ing o wa d he head and u ning i le and mo ing he eyes up. (Figu e 5.b). The ang iness exp ession shown in ol es he ac i a ion o AU4+AU7+AU10 +AU16+AU55+AU63 ha co espond o lowe ing he eyeb ows, na owing he eyelid ape u e, aising he uppe lip, dep essing he lowe lip, il ing o wa d he head and sligh ly mo ing he eyes up. (Figu e 5.c). (a) (b) (c) (d) (e) ( ) Fig. 5. Basic Emo ions (a) Joy (b) Sadness (c) Ange (d) Fea (e) Disgus ( ) Su p ise 8 S. Ma cos, R. Pinillos, J. G´omez-Ga c´ıa-Be mejo, E. Zalama 5 In eg a ion in a mul imodal a chi ec u e Cu en ly he e is a g owing in e es in modula a chi ec u es and componen amewo ks ha encou age he c ea ion o easily modi iable and scalable open sys ems. This kind o ini ia i e is o pa icula in e es in he ield o mecha on- ics and se ice obo ics, whe e sys ems a e speci ically designed o high le el in e ac ions and sough o be easily modi ied and imp o ed o mee new equi e- men s. The main end hese days in e ms o de elopmen o obo ic sys ems is he design o a chi ec u es based on componen s o amewo ks. Fo his pu - pose ules o combining componen s and he in e ac ion pa e ns be ween hem a e de ined in o de o achie e a lexible sys em. This lexibili y is unde s ood mainly in he ollowing aspec s: - Easy eplacemen o componen s wi h di e en unc ionali y. - Reuse o di e en modules in o he p ojec s whe e he unc ionali y o hose pa icula modules is equi ed. The po en ial o he open modula a chi ec u es is huge, and leads o econ- igu able, po able and ex ensible sys ems. This philosophy has been adop ed o de elop he con ol so wa e o di e en social obo s in ou labo a o y. Based in he ROS amewo k, i consis s o a se o laye s wi h di e en in e change- able modules (see Figu e 6). The in e ac ion be ween laye s and modules is well de ined, so a new module can easily be in eg a ed. I has o be no ed ha his a chi ec u e has no only been used in obo s, bu also in o he agen s such as anima ed aces [5]. Thanks o he AU based emo ional gene a ion app oach, bo h anima ed and obo ic aces can be con olled using he same uppe le el componen s. Fig. 6. Modula mul imodal a chi ec u e Making use o his a chi ec u e, he mecha oinc head and i s low le el con- ol d i e s ha e been in eg a ed wi hin he down laye , aking ad an age o Design o a Realis ic Robo ic Head based on Ac ion Coding Sys em 9 he uppe laye se ices such as oice ecogni ion, exp ession ecogni ion o be- ha iou al planne . As can be seen in he ideo linked o his a icle (see Sec ion 7 below), as an example he obo ic head has been easily connec ed o ou p e iously de el- oped acial exp ession ecogni ion sys em. This sys em si able o pe o m bo h: acial exp ession ecogni ion and acial emo ion ecogni ion. Exp ession ecog- ni ion e e s o he de ec ion and analysis o acial mo emen s, and changes in acial ea u es om isual in o ma ion. On he o he hand, emo ional exp ession ecogni ion also associa es hese acial displacemen s o hei emo ional signi - icance, gi en ha he e will be ce ain mo emen s associa ed wi h mo e han one emo ion. The p oposed me hod [5] is di ided in o wo di e en s ages: he i s one ocuses on he de ec ion and ecogni ion o small g oups o acial muscle ac ions using a combina ion o Gabo il e s and Ac i e Shape Models. Based on he ou pu o his i s s age, he second de e mines he associa ed emo ional ex- p ession using an habi ua ion based ne wo k plus a compe i i e based ne wo k. Thanks o he implemen ed con ol a chi ec u e, i can be obse ed in he ideo how he mecha onic head is able o mimic he emo ional exp essions o he use using a egula webcam. 6 Pilo expe imen s in emo ion ecogni ion The mecha onic head exp ession gene a ion capabili ies we e es ed in a pilo s udy wi h 10 pa icipan s which ages anged be ween 28 and 40 yea s old. In o de o see he in luence o he in ensi y pa ame e ki(see ecua ion 1) in he gene a ion o ecognisable exp essions, use s we e asked o ecognize he six uni e sal exp essions unde wo condi ions: hal - in ensi y (ki= 0.5), and ull- in ensi y (ki= 1). Exp essions we e andomly gene a ed o each condi ion and use s we e asked o ano a e which exp essions hey we e able o ecognize, as well as he o de he mecha onic head had displayed hem. The subjec s answe s in labeling an exp ession we e sco ed as co ec o w ong and esul s a e shown in igu e 7. F om he esul s i can be obse ed ha he in ensi y has a g ea impac in he ecogni ion o he majo i y o he exp essions, specially in he ange ex- p ession. Apa om he joy exp ession, a g ea e in ensi y esul s in a highe ecogni ion a e, al hough in exp essions such as sadness o disgus he di e - ence is no highly signi ican . I has o be no ed ha igu e 7 jus shows he co ec answe a io and no hose answe s in which use s doub ed be ween wo exp essions. This happened be ween he ange and disgus exp essions and be- ween he ea and disgus exp essions. Use s ound i di icul o dis inguish be ween hem, and labeled he shown exp essions as bo h ange and disgus o ea and su p ise. I can be seen ha his con usion is e en mo e accen ua ed in he case o he hal -in ensi y exp essions. I has o be no ed, howe e , ha he small numbe o pa icipan s make some o he ob ained esul s a om being signi ican . Fo ins ance, ecogni ion a es o 3 ou o 10 a e close o andomness meaning ha use s we e no able o ecognize he exp ession. In o de o ha