Knowledge representation and reasoning for perception-based manipulation planning

Uni e si a Poli ècnica de Ca alunya Ph.D. P og am: AUTOMATIC CONTROL, ROBOTICS AND COMPUTER VISION Ph.D. Thesis Knowledge Rep esen a ion and Reasoning o Pe cep ion-based Manipula ion Planning Mohammed Diab Thesis Ad iso : Jan Rosell G a acòs No embe 2020 Knowledge Rep esen a ion and Reasoning o Pe cep ion-based Manipula ion Planning Submi ed in pa ial ul illmen o he equi emen s o he deg ee o Ph.D. in Au oma ic Con ol, Robo ics and Compu e Vision Supe ised by Jan Rosell G a acòs Ins i u d’O gani zació i Con ol de Sis emes Indus ials Uni e si a Poli ècnica de Ca alunya No embe 2020 This hesis is dedica ed o he spi i o my a he , my child en, and my sis e s’ child en. Acknowledgmen s I would like o hank he Responsible o he Doc o al p og amme in Au oma ic Con ol, Robo ics and Compu e Vision P o . Raúl Suá ez o his e o s and guidance du ing he Ph.D. My ex e nal collabo a o s P o . Micheal Bee z, P o . John Ba eman, D . Mihai Poma lan and Daniel Beßle om B emen Uni e si y, P o . Flo en in Wö gö e , D . Tomas Kul icius om Geo g-Augus - Uni e si ä Gö ingen, and inally P o . S e ano Bo go om Labo a o y o Applied On ology (LOA), ISTC CNR, T en o, I aly, o hei guidance du ing my esea ch s ays. A e y special hanks should be p esen ed o my supe iso P o . Jan Rosell o his guidance, mo i a ion, pa ience, discussion, and help in all aspec s o comple e his hesis. I would also like o hank D . Ali Akba i and D . Muhayy Ud Din o hei help and use ul long discussion. I would also like o hank all he membe s o he Se ice o Indus ial Robo ics esea ch g oup o hei help. I would like o hank Leopold Palomo o his guidance in he implemen a ion s u . A e y special hanks o my mo he , Wahida El-Husseiny, my wi e, Fa ema Mohamed, my sis e s, and he es o my amily o he e in ini e suppo , i was no possible o comple e he hesis wi hou hei suppo in all aspec s. All p aise and glo y be o God who is he g ea es bene ac o , and Whose helps enabled me o comple e his hesis. ii Con en s Acknowledgmen s ii Abs ac xix 1 In oduc ion 1 1.1 P oblems a emen ................................... 1 1.2 P oposedsolu ions ................................... 2 1.3 Con ibu ions...................................... 3 1.4 Thesis oadmap..................................... 6 1.5 Mo i a ionexamples.................................. 8 1.6 Lis o Publica ions................................... 10 1.7 Publica ionNo e .................................... 11 2 Rela ed Wo k 13 2.1 Manipula ionplanning................................. 13 2.1.1 Taskplanning.................................. 14 2.1.2 Mo ionplanning ................................ 15 2.1.3 Combina ion o ask and mo ion planning . . . . . . . . . . . . . . . . . . 17 2.2 Knowledge ep esen a ion using on ologies . . . . . . . . . . . . . . . . . . . . . 19 2.3 The use o knowledge in di e en domains . . . . . . . . . . . . . . . . . . . . . . 20 2.4 Knowledge uppe -le el ounda ions e o s . . . . . . . . . . . . . . . . . . . . . . 21 2.5 Knowledge-based seman ic pe cep ion in obo ics . . . . . . . . . . . . . . . . . . 23 2.6 Use o on ologies o inc ease obo au onomy . . . . . . . . . . . . . . . . . . . . 24 2.7 Logic-basedplanning.................................. 25 3 Knowledge Guidance o Task and Mo ion Planning 29 3.1 In oduc ion....................................... 29 3.2 P oblem s a emen and p oposed solu ion . . . . . . . . . . . . . . . . . . . . . . 29 3.2.1 P oblem o maliza ion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.2.2 P oposed F amewo k o e iew . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3 A Knowledge P ocessing F amewo k o Physics-based Manipula ion Planning . . 31 3.3.1 The P oposed F amewo k o Physics-based Manipula ion . . . . . . . . . 34 ix LIST OF FIGURES 5.12 a) he obo checks he simila i y o he cu en si ua ion, i inds he same skill has been used wi h he same objec (i.e., a d awe in he ile cabine ), hen adap s he skill wi h he same mo ion used in he da abase; b) he obo igu es ou he op-g asp is no easible o pou ing ac ion, he op o he ile cabine is used as a placemen oom o change he g asp ype; c) he obo changes he g asp ype om he op-g asp o he side-g asp; d) he obo se es he con en s o he can in he cup o a cus ome , he se e mo ion is adap ed om he expe ience, acco ding o he cu en pose o he obo and loca ion o he cup. Video URL: h ps://www.you ube.com/wa ch? =bTmWAkjC93c ................116 5.13 (a) A e age ime o openD awe , pickUp om he d awe and se ing skills using adap a ion and planning wi h and wi hou expe ien ial knowledge. (b) Success a e o heeachskill. .................................118 5.14 Adap a ion p ocess o se ing skill wi h di e en poses. . . . . . . . . . . . . . . 120 5.15 The sequence o snapsho s om planning p ocess. The i s image shows he manipula ion example whe e he goal is o ans e he g ay cylinde (labeled as A) o one o he ays w. . i s colo . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.16 The condi ional plan esul s om he planning p ocess. a) lowcha desc ibing he plan ob ained by he con ingen FF. b) lowcha added when execu ing he plan o moni o ing and epai i necessa y he ac ion ou comes shown in ed in heplan..........................................124 5.17Theexecu ableplan...................................124 5.18 The in eg a ion o con ingency plan wi h eco e y knowledge. . . . . . . . . . . . 125 Page x i o 151 Lis o Tables 3.1 Spa ial easoning. ................................... 56 3.2 Lis o ele an e ms o he au onomous obo ics domain, and hei co e age in he di e en chosen wo ks. Yes and No s a e o when he e m is o no co e ed by he on ology o he speci ic amewo k. No e ha in he cases when he e m is needed and aken om he uppe on ology used wi hin he amewo k, and/o when he knowledge is cap u ed using a simila e m, i is conside ed ha he e m is co e ed. I he uppe on ology con ains he e m bu i is no used, we conside ha he e m is no included. . . . . . . . . . . . . . . . . . . . . . . . . 64 3.3 Lis o cogni i e capabili ies o he au onomous obo ics domain and hei co e age in he di e en chosen amewo ks/on ologies. I is possible o ind he e e ence o he a icles in which he di e en easoning capabili ies a e add essed using he on ologies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 4.1 Modeling he ailu e on ology unde he DUL and SUMO ounda ions. . . . . . . 73 5.1 Tes he skill openD awe , pickUp and se ing using adap a ion me hod s he planning sys em wi h and wi hou expe ien ial knowledge. . . . . . . . . . . . . . 117 B.1 DLno a ion.......................................138 x ii Abs ac This hesis add esses he pe cep ion-based knowledge ep esen a ion and easoning o a combina ion o ask and mo ion planning o deal wi h di e en ypes o obo ic manipula ion p oblems, anging om single o mul iple collabo a i e mobile obo s na iga ing among mo able obs acles o complex highe -dimensional able- op manipula ion p oblems ca ied ou by dual-a m obo s o mobile manipula o s. Fo hose p oblems, besides he combina ion o ask and mo ion le els o planning, he in eg a ion o pe cep ion models wi h knowledge o guide bo h planning le els, esul ing in a sequence o ac ions o skills which, acco ding o he cu en knowledge o he wo ld, may be execu ed, is necessa y. This combina ion pu sues he ob en ion o a geome ically easible manipula ion plan h ough a symbolic and geome ic sea ch space. I has eme ged as a challenging issue as he ailu es due o geome ic cons ain s lead obo s o dead-end asks. Manipula ion asks in which he e may be in e ac ions be ween obo s and objec s a e conside ed. To cope wi h hem, knowledge abou he physics o he en i onmen is in eg a ed wi h a combina ion o ask planning and physics-based mo ion planning, allowing o deal wi h push and pull ac ions and also o look o low-cos plans in e ms o powe . This in eg a ion en iches he planning p ocess and o aid in p o iding ways o execu ing symbolic ac ions. P oblems wi h unce ain in o ma ion (in he ini ial s a e o he obo wo ld o in he esul o symbolic ac ions) and p oblems whe e humans and obo s in e ac a e also conside ed. To deal wi h such issues, a combina ion o con ingen -based ask and mo ion planne wi h knowledge o ailu e in e p e a ion and eco e y is p oposed, which assumes he a ailabili y o a pe cep ion sys em ( o e alua e he ac ual s a e o he en i onmen ) and he collabo a ion o he human ope a o ( obo s can ask humans o hose asks which a e di icul o in easible o hem). Fo e e y-day asks, expe ien ial knowledge can play a signi ican ole o make he obo capable o lea n om i s expe ience ins ead o epea edly planning he same ask wi h he same gi ens wi hin he planning sys em, which could be compu a ional and ime-consuming p ocess. To deal wi h such issues, a obo ics amewo k is p oposed which is equipped wi h a module wi h expe ien ial knowledge (lea ned om i s expe ience o gi en by he use ) on how o execu e a se o ac ions, like pick-up, pu -down, o open a d awe , using wo k lows as well as obo ajec o ies. An implemen a ion amewo k o combine di e en ypes o ask and mo ion planne s is p esen ed. All he equi ed modules and ools a e illus a ed, including he explana ions on he low o in o ma ion be ween he di e en languages used, P olog and C++. xix Chap e 1 In oduc ion This chap e desc ibes he s uc u e and con en s o his hesis. I con ains he p oblems ha his hesis deals wi h, as depic ed in Sec. 1.1, p oposed solu ions, as depic ed in Sec. 1.2, and he hesis con ibu ions, Sec. 1.3. The p oposed solu ions equi e app oaches o be used such as planning, knowledge ep esen a ion, easoning, pe cep ion, as well as lea ning, as desc ibed in Sec. 1.4. These app oaches ha e been used and in eg a ed oge he in o de o inc ease obo au onomy. Se e al challenges ha e been se o es he p oposed solu ions, as depic ed in 1.5. These solu ions ha e been published in obo ics and A i icial in elligence (AI) jou nals, as well as con e ences, hey a e men ioned in Sec. 1.6. These wo ks ha e been done wi h collabo a ion wi h o he esea ch cen e s, as men ioned in Sec. 1.7. 1.1 P oblem s a emen Indoo obo s wi h au onomy, mobili y and manipula ion capabili ies ha e he po en ial o ac as obo helpe s a home o imp o e he quali y o li e o a ious use popula ions, such as elde and handicapped people, o o ac as obo co-wo ke s a ac o y loo s, helping o ins ance in assembly applica ions whe e collabo a ing wi h o he ope a o s may be equi ed. In hese semi/uns uc u ed en i onmen s, he obo ask may no be p ope ly o ganized and he model o he en i onmen comple ely known, and he e o e abundan p oblems ha e o be aken in o conside a ion, o example, he need o emo ing obs acles in o de o ha e p ecisely access o a pa icula objec , which equi es he in eg a ion o symbolic and geome ic planning le els, wi h skills such as pick-up, pu -down o na iga e. Mo eo e , pe cep ion sys ems using ision o dep h senso s may be equi ed o model he geome y and he pose o objec s in he en i onmen and egula ly upda e hei s a us. Di e en ypes o senso s ha e hei own limi a ions, so pe cep ion sys ems based on mul i-senso y da a in eg a ion combining in o ma ion om di e en sou ces a e e y use ul o ob ain in o ma ion, which in some sense is be e han he one ob ained when 1 CHAPTER 1. In oduc ion he sou ces a e used sepa a ely. To plan he asks o be done by he obo in he abo e-men ioned scena ios, sophis ica ed planning mechanisms a e equi ed o adap he ac ual s a e o he en i onmen and comply wi h cons ain s bo h a ask and a geome ic le els. The e a e wo dominan app oaches in he manipula ion planning domain, one based on classical ask planning and he o he based on knowledge and easoning. The o me mainly uses he Planning Domain De ini ion Language (PDDL) o desc ibe he wo ld. The main ad an age o his way o desc ip ion is ha i can easily handle asks wi h many ac ions, and in eg a e he geome ic (mo ion) cons ain s. Howe e , i makes he closed wo ld assump ion, i.e., i some ac s abou he wo ld a e no known o change, a planne may no be able o ind a solu ion. This limi a ion means ha obo s a e no able o begin a ask un il all objec s in he en i onmen a e known and he ac ions he obo can do on hem a e comple ely de ined. The la e has eme ged as a new domain o planning, ocused on making he obo able o lexibly pe o m manipula ion asks. The main ad an age o his app oach is ha i can easily in eg a e he knowledge om he en i onmen and adap he ac ion o be done acco dingly. Howe e , in complex manipula ion p oblems ha may ha e many ask (geome ic) cons ain s, such as he Towe s o Hanoi p oblem, whe e ask and mo ion le els a e coupled, hey may ail o compu e a long sequence o ac ions wi h easible mo ion solu ions (Lag i oul e al.,2018), (Beßle e al.,2018a). 1.2 P oposed solu ions To ackle he a o emen ioned limi a ions o bo h app oaches, some componen s a e equi ed ha may acili a e he p ocess o manipula ion planning. Fi s , a pe cep ion sys em o pe cei e he objec s and hei ea u es, and a mechanism o cap u e he seman ics o he ac ual scene and p epa e he planning acco dingly. Second, a mechanism o inco po a e geome ic knowledge and mo ion planning. Finally, a knowledge-based in e ence mechanism o eason on ela i e posi ioning, p econdi ions sa is ac ion and ac ion easibili y. Tha is, he in eg a ion o pe cep ion and knowledge wi h manipula ion planning app oaches, may help o co e some missing componen s such as easoning mechanisms able o analyze he easibili y o ac ions, he a ailabili y o placemen egions, he eachabili y o g asping mo ions, and he sa is ac ion o manipula ion cons ain s. These componen s play a signi ican ole in obo ic manipula ion, specially o bi-manual obo asks o o mul i- obo coope a ion, ha equi e geome ic easoning, including physics-based mo ion planning wi h que ies abou how in e ac ion wi h he objec s is o be done. Also, knowledge-based sensing modules can be in eg a ed wi h planne s coping wi h unce ain scena ios, ha equi e easoning abou he sensing sys em, he ea u es o he en i onmen en i ies, and senso limi a ions. This way, he obo can be awa e abou he ype o senso s ha i has and how o use hem. Hence, knowledge-based easoning is p oposed o acili a e he p ocess o manipula ion. This solu ion has he capabili y o make he wo ld open o cope wi h en i onmen al dynamic en i ies. I can be used o complex manipula ion asks ha equi e he combina ion o bo h Page 2 o 151 1.3. Con ibu ions symbolic and geome ic le els o planning and he in eg a ion wi h a pe cep ion module can p o ide a ich seman ic desc ip ion o he obo whene e needed. In his sense, a well-s uc u ed knowledge ep esen a ion plays a signi ican ole. Many ways a e used o knowledge ep esen a ion, such as on ologies, ha a e conce ned wi h s uc u ing concep s and ela ions such ha hey a e usable o easoning asks done by a i icial sys ems (e.g., obo s). Fo mally, an on ology is de ined as “an explici , o mal speci ica ion o a sha ed concep ualiza ion" (G ube ,1995). The concep ualiza ion e e s o he abs ac models o en i ies in a ce ain domain. These models a e achie ed by de ining hei ele an concep s along wi h hei ela ions. In his line, his hesis de elops a se ies o modeling and easoning ools o knowledge- o ien ed manipula ion planning in semi/uns uc u ed en i onmen s. The main idea is o use high-le el knowledge-based easoning o cap u e a ich seman ic desc ip ion o he scene, knowledge abou he physical beha io o he objec s, and in e ence mechanism o eason abou he po en ial manipula ion ac ions. Mo eo e , a mul i-senso y module is p oposed o pe cei e he objec s in he en i onmen and build he on ological knowledge. 1.3 Con ibu ions Mobile manipula o s ac ing as obo co-wo ke s a e equi ed o wo k au onomously in human en i onmen s, and in he p esence o human ope a o s. Au onomy can be achie ed wi h in eg a ed ask and mo ion planning capabili ies, ha a e able o ind easible plans o he obo o execu e complex asks. Pe cep ion capabili ies a e, howe e , a key issue o he success ul execu ion o asks, because human en i onmen s a e semi-s uc u ed and a ec ed by unce ain y. The e o e, a pe cep ion module wi h di e en senso s, including hose like RFID ha can cope wi h non-line-o -sigh (NLOS) si ua ions, as well as senso y usion mechanisms, is equi ed. The a ailabili y o such a pe cep ion module may allow o conside sensing ac ions in he ask planning p ocedu e, o educe he e ec s o unce ian y in he ini ial s a e and in he ac ions e ec s. Mo eo e , o be able o ace di icul manipula ion asks in hese semi-s ucu ed en i onmen s, some seman ic knowledge on he objec s o he en i onmen and on he possible manipula ion ac ions, is equi ed. This knowledge may guide bo h he planning a mo ion le el and a ask le el. This hesis con ibu es in his line by implemen ing and in eg a ing he necessa y module o inc ease he obo au onomy. The con ibu ion a e lis ed below. •Knowledge Guidance o Task and Mo ion Planning: A manipula ion knowledge amewo k, called Knowledge-based Task and Mo ion Planning (KTAMP) is p esen ed. The amewo k con ains he ool called, Pe cep ion and Manipula ion Knowledge (PMK) which is p esen ed in e ms o an on ology-based modeling and easoning p ocess. PMK aims a being sha ed and eused, and o his, PMK on ology elies on o he uppe and e e ence/domain on ologies: he Sugges ed Uppe Me ged On ology (SUMO) (Niles and Page 3 o 151 CHAPTER 1. In oduc ion Pease,2001a) and he Co e On ology o Robo ics and Au oma ion (CORA) (P es es e al., 2013). The easoning mechanism includes some easoning p ocesses o au onomous obo s o enhance Task and Mo ion Planning (TAMP) capabili ies in he manipula ion domain. A pe cep ion module can be in eg a ed wi h he amewo k o cap u e a ich seman ic desc ip ion o he scene, knowledge abou he physical beha io o he objec s, and easoning abou he po en ial manipula ion ac ions. The easoning scope o PMK is di ided in o ou pa s: easoning o pe cep ion (e.g. which pe cep ual ea u es can be ob ained and wi h which senso s?), he easoning o objec manipula ion ea u es (e.g. how can a gi en objec be manipula ed acco ding o i s cha ac e is ics and he cu en pose), he easoning o a si ua ion, (e.g. which a e he spa ial ela ions o he objec s in he scene?), and easoning o planning (e.g. can a gi en p imi i e be applied a he cu en scene?). The PMK ool is also in eg a ed wi h physics-based mo ion planning ha aims mainly o p o ide he way o in e ac ions be ween a obo and objec s holding speci ic manipula ion cons ain s. Specially, PMK p o ides: 1. S anda d ep esen a ion: The knowledge modeling is p oposed by adap ing he a ailable concep s p o ided by IEEE-1872 s anda ds o knowledge ep esen a ion o he obo ic domain. Mo eo e , some unco e ed concep s ela ed o manipula ion domains ha e been p oposed, such as knowledge ela ed o senso s. 2. Knowledge ep esen a ion o pe cep ion: The pe cep ion on ology is p oposed o include he pe cei ed in o ma ion om di e en senso s, e.g., he ep esen a ion is wo kable o came as o Radio F equency Iden i ica ion (RFID), and may include any implemen ed sensing lib a y. 3. Si ua ion analysis: In e ence p ocess p edica es a e de eloped based on Desc ip ion Logic (DL) o e alua e he objec s’ si ua ion in he en i onmen based on spa ial easoning, and o ela e he classes en i ies and eason o e hem. Mo eo e , po en ial placemen egion and spa ial eachabili y o he obo a e in oduced. 4. Planning enhancemen : The use o PMK as a black-box allows any planne o eason abou TAMP equi emen s, such as obo capabili ies, ac ion cons ain s, ac ion easibili y, and manipula ion beha io s. Tha includes a seman ic ex ension o au oma ically cons uc and ca ego ize he objec s in o di e en ypes acco ding o he objec s and ask cons ain s. Also, he in e ac ion dynamics ex ension o de ine a knowledge ha allows he planne o deal wi h in e ac ion dynamics. •He e ogeneous easoning planning app oach: An on ology-based amewo k o ailu e in e p e a ion and eco e y in planning and execu ion called FailRecOn is p oposed owa ds a mo e au oma ed, easoning and knowledge-d i en app oach o ailu e handling. Such an app oach would need a concep o wha " ailu e" means, wha kinds o ailu es migh happen and why, and concep s o de ine wha an app op ia e esponse migh be. The easoning mus also be in eg a ed in o he geome ic on ology ha gi es access o geome ic easoning such as collision check o check he ac ions’ easibili y as well as he pe cep ion ac ion loop o he obo , and able o guide a plan epai p ocess o esume o epea a ask a e a ailu e. In mo e de ail, FailRecOn p o ides: 1. On ology o mula ion: Fo mal de ini ion o concep s o desc ibe ailu es acco ding o aspec s such as causal mechanism, loca ion, ime ela i e o ask pe o mance, and Page 4 o 151 1.3. Con ibu ions unc ional conside a ions e.g. esou ces, and concep s o desc ibe eco e y s a egies acco ding o he plan epai ope a ions hey equi e. 2. Modeling in di e en ounda ions: –Fo obo ics domain: Modeling he absolu e abs ac concep s unde obo ics uppe -le el on ologies such as CORA, which uses he SUMO on ology as an uppe le el. –Fo enginee ing domain: Modeling he absolu e abs ac concep s unde e y gene ic ounda ional on ologies such as DUL. 3. In eg a ion o geome ic easoning module wi hin on ology: In eg a ion o how o call he low-le el geome ic easoning such as collision check, mo ion planning, in e se kinema ic (IK) and objec placemen om on ology, which is a s ep owa d mo e obo au onomy. 4. Use o he ailu e on ology: Desc ip ion o how o use he ailu e on ology in a ask and mo ion planning (TAMP) p ocess using a knowledge-d i en app oach, esul ing in an he e ogeneous easoning planning app oach. •Skill-based ask and mo ion planning: A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning amewo k, called SkillMaN is p oposed, which is equipped wi h a module wi h expe ien ial knowledge (lea ned om i s expe ience o gi en by he use ) on how o execu e a se o skills, like pick-up, pu -down o open a d awe , using wo k lows as well as obo ajec o ies. The amewo k also con ains an execu ion assis an wi h geome ic ools and easoning capabili ies o manage how o ac ually execu e he sequence o mo ions o pe o m a manipula ion ask (which a e o wa ded o he execu o module), as well as he capaci y o s o e he ele an in o ma ion o he expe ien ial knowledge o u he usage, and he capaci y o in e p e he ac ual pe cei ed si ua ion (in case he p econdi ions o an ac ion do no hold) and o eedback he upda ed s a e o he planne o esume om he e, allowing he obo o adap o non-expec ed si ua ions. Aiming a gi ing he obo mo e au onomy, speci ically, SkillMaN amewo k p o ides he ollowing se ices and modules: 1. Pe cep ion: An in eg a ed mul i-senso y module based on RFID senso s, wi h s o age da a capabili y, and RGB-D came as, 2. Si ua ion simila i y check: Se ice o seman ically check he si ua ion simila i y based on obo goals and pe cep ion ou comes. I helps he obo o decide whe he o use i s expe ience-based knowledge, 3. Planning: Se ice o in e lea e symbolic and geome ic easoning le els wi h he pe cep ion module o make he obo capable o pa ially igu e ou he en i onmen and plan unde pa ial in o ma ion, and 4. Expe ien ial knowledge: Module o s o e expe ien ial knowledge on how o execu e a se o skills, like pick-up, pu -down o open a d awe , o be used o ins ance o adap ing he mo ion o he obo in simila si ua ions. All hese se ices and modules a e used h ough a ask manage in a way ha can be smoo hly used in di e en asks. Page 5 o 151 Chap e 2 Rela ed Wo k This chap e ocuses on s a e-o - he-a esea ches in manipula ion planning, s anda dized knowledge ep esen a ion, and pe cep ion-based seman ic knowledge in he obo ics domain, as well as he in eg a ion be ween hem o inc ease obo au onomy. Mo eo e , a logic-based planning app oach is desc ibed in mo e de ail han o he app oaches because some o he on ologies p oposed in his hesis a e na u ally in eg a ed he e. 2.1 Manipula ion planning Manipula ion p oblems a e e e ed o as p oblems in which obo s manipula e objec s using a se o p imi i es, e.g., pushing, picking, o placing. Due o ask cons ain s, he limi a ion o gene ic mo ion planning eme ges, and he obo is equi ed o displace objec s when he e is no easible solu ions be ween wo obo con igu a ions. A mo e gene al manipula ion planning app oach has been de eloped by (Siméon e al., 2004) ha conside s mul iple possible g asps ( ha can be used o e-g asping he objec s) and s able placemen s o he mo able objec s o sol e he p oblem. In he ela ed ield o g asp planning, (Azizi e al.,2017) p opose a geome ic app oach based on de ec ing a comple e se o objec subsu aces in a clu e ed scene, which allows he end-e ec o o sa ely app oach and g asp he objec . In a simila way, (He le and Nebel,2017) p esen echniques o sampling app op ia e geome ic con igu a ions o objec placemen s, g asping poses, o obo posi ions, in o de o pe o m a speci ic ac ion. And ecen ly, a me hod o g asp planning in clu e ed en i onmen s (Muhayyudin e al.,2018) has been p oposed, which uses andomized physics- based mo ion planning o accoun o obo -objec and objec -objec in e ac ions. This allows a obo o push obs uc ing objec s away while eaching a a ge g asp pose. 13 CHAPTER 2. Rela ed Wo k A manipula ion planning amewo k wi h pe cep ion capabili y has been p oposed (Migima su and Bohg,2020) ha op imizes o e Ca esian ames de ined ela i e o a ge objec s. The esul ing plan emains alid e en i he objec s a e mo ing and can be execu ed by eac i e con olle s ha adap o hese changes in eal ime. The amewo k is applied o a o que-con olled obo in a pick and place se ing and demons a e i s abili y o adap o changing en i onmen s, inaccu a e pe cep ion, and imp ecise con ol, bo h in simula ion and he eal wo ld. As a complemen a y o his wo k, a lea ning echnique has been p oposed in (Shao e al.,2020) o endow a obo wi h he abili y o lea n manipula ion concep s ha link na u al language ins uc ions o mo o skills. In (Engle and Toussain ,2018) lea ning manipula ion skills om a single demons a ion is p oposed whe e a obo is demons a ed a manipula ion skill once and should hen use only a ew ials on i s own o lea n o ep oduce, op imize, and gene alize ha same skill. The manipula ion p oblem o Na iga ion in indoo clu e ed en i onmen , a which a obo needs o manipula e some objec s in he wo kspace in o de o each i s goal egion, has been add essed by in (S ilman e al.,2007;S ilman and Ku ne ,2008) and (Hause and La ombe, 2010;Hause e al.,2010;Hause ,2014) conside ing he exis ence o he objec s occluding he way be ween wo obo con igu a ions. The e iewed wo ks concen a e on geome y challenges in manipula ion wi h no high- le el easoning, while he e a e o he manipula ion p oblems which equi e o sea ch in a symbolic sea ch space o ind a sequence o ac ions. The app oaches, u he mo e, lack om unde lying disc e e symbolic ela ions (e.g. among obo s and objec s) which can b eak down he complexi y o he p oblem, unde s ood as he need o check he easibili y o he ac ions used in a ce ain ask. The e o e, combining ask and mo ion planning makes a signi ican ole o deal wi h di e en challenges o manipula ion p oblems o obo s. 2.1.1 Task planning The e has been a signi ican amoun o s udies in ask planning, comp ising a a ie y o di e en au oma ed ask planning app oaches such as hie a chical, plan g aph, and heu is ic planning. Hie a chical app oach (Wol e e al.,2010), (Kaelbling and Lozano-Pé ez,2011) es ablishes a ne wo k o assign possible p econdi ions o ac ions (p imi i e o compound). To ind a plan, asks a e decomposed and g ow in a sea ch space un il sa is ying p imi i es ac ions. The plan g aph (B yce e al.,2006), (Blum and Fu s ,1997) cons uc s a sea ch space o plans wi h he o m o a g aph and es ablishes a sequence o le els g adually expanded h ough he sea ch space. The main me i s o g aph-based planne s a e he abili y o analyze combina ions o ac ion o sa is y he ask, and he abili y o e alua e mul iple ways o each he goal. Rega ding heu is ic planning, one o he success ul app oach is FF ( as o wa d). I has wo main componen s: RPG ( elaxed planning g aph), and s a e space sea ch. RPG is he simpli ied e sion o he plan g aph, i.e., dele e lis ( he ac s which a e dele ed by ac ion) o ac ions a e igno ed. The la e is de o ed o he sea ch o he highes chance o success using he heu is ic alues, i.e. help ul ac ions ( hose ac ions ha execu ed om a s a e ha e a high chance o being Page 14 o 151 2.1. Manipula ion planning in he inal plan) ha i uses o guide he sea ch o he s a e space (Dea den and Bu b idge, 2014). To handle unce ain y in ask planning (such as he unce ain y in he ac ion e ec ), a ian s o FF such as con ingen , and con o man FF ha e been p oposed. Con ingen FF is he ask o gene a ing a condi ional plan gi en unce ain y abou ini ial condi ion and ac ion e ec s, bu wi h he abili y o obse e some aspec s o he cu en wo ld s a e. Con ingen planning can be ans o med in o an AND/OR sea ch p oblem in belie space ( he space whose elemen s a e se s o possible wo lds). The plan, which is a ee a he han a sequence o ac ions, can ea e e y possible ou come. I means ha he sea ch space is an AND/OR ee, and he plan is sub- ee whe e all lea es a e goal (belie ) s a es. The main ad an age o con ingen planne is minimizing ime o ind he a ge , and execu e manipula ion asks e icien ly (because he e is a plan wi h obse a ion) (Ho mann and B a man,2005). On he o he hand, con o man planning is he special case o con ingen planning whe e no obse a ions a e possible (B a man and Ho mann,2004), (Koenig e al.,2004). Acco ding o di e en applica ions, app oaches and s a egies may a y. Some esea ches ha e used FF planne s o manipula ion asks, like (S i as a a e al.,2014), (Akba i e al.,2018a), (Lag i oul e al.,2013). Besides he a o emen ioned app oaches, some knowledge-d i en app oaches use on ologies in planning. Fo ins ance, a knowledge-based ask and mo ion planning amewo k based on a e sion o he FF ask planne is p esen ed in (Akba i e al.,2016c). A easoning p ocess on symbolic li e als in e ms o knowledge and geome ic in o ma ion abou he wo kspace, oge he wi h he use o a physics-based mo ion planne a e used o e alua e he applicabili y and easibili y o manipula ion ac ions and o compu e he heu is ic alues ha guide he sea ch. The manipula ion p oblem, in ol ing knowledge abou he wo ld and he planning phase, is coded in he o m o an on ology, and add esses a high-le el and a low-le el easoning p ocesses o app aise manipula ion ac ions and p une he ask planning phase om dispensable ac ions (Akba i e al.,2018b). Mo eo e , in (Beßle e al.,2018a) an assembly-based planne p esen ed in On ology Web Language (OWL) o ma using Desc ip ion Logic (DL) is buil upon an exis ing planne o assemble a kid oy plane called "BATTAT". This app oach ex ends he p oposed planne wi h a no ion o ac ion, and geome ic easoning capabili ies. Ac ions a e ep esen ed in e ms o he ac ion on ology which also de ines ac ion p e-condi ions. P e-condi ions a e ensu ed by unning he planne o he ac ion en i y. This is used o ensu e ha he obo can each an objec , o else ies o pu away occluding objec s. To his end a geome ic easone is in eg a ed wi h he knowledge base. The in e aces o he geome ic easone a e hooked in o he logic-based easoning h ough p ocedu al a achmen s in he knowledge base. Since his planne has been used in his hesis, we in oduced in de ail in Sec. 2.7. 2.1.2 Mo ion planning Mo ion planning deals wi h de ec ing collision- ee pa hs o con ey a obo o a con igu a ion s a e. I is mos ly done in he con igu a ion space (C-space) (Lozano-Pe ez,1983). The C-space has as many dimensions as deg ees o eedom he obo has, and he e o e each poin ep esen s a con igu a ion o he obo . The subspace co esponding o collision- ee con igu a ions is called C ee and he subspace co esponding o collision con igu a ions is called Cobs. Mo ion Page 15 o 151 CHAPTER 2. Rela ed Wo k planning in C-space consis s in inding a pa h in C ee be ween wo con igu a ions. Wi h ega d o classical mo ion planning, esea ches ha e in es iga ed how o deal wi h geome ic cons ain s in planning ( hey do no impose di e en ial dynamic cons ain s). T adi ional me hods comp ise di e en echniques such as cell decomposi ion, oadmap-based me hods, and po en ial ields. In mo ion planning based on cell decomposi ion echniques, he i s s ep is o decompose ee space, ei he in an exac o an app oxima ed way, and ep esen he se o cells as a g aph. Then, some sea ch algo i hms such as A* (Ha e al.,1968) o Dijks a (Dijks a,1959) me hods can be employed o ind a pa h o minimal cos . The oadmap- based app oaches build a g aph o connec he ini ial and goal s a es in ee space and hey usually ind he solu ion pa hs based on he sho es dis ance. Rega ding his case, when he obs acles in he C-space a e ep esen ed as polygons, he isibili y g aph me hods (De Be g e al., 2000) gene a es a g aph whose nodes a e he e ices o he polygons and whose edges a e line segmen s wi h no in e e ence o obs acles. The las app oach uses po en ial ields compu ed in he C-space o p o ide a ac ion o he goal con igu a ion and epulsion om obs acles (Kha ib, 1986). The d awback o all hese app oaches is ha o p oblems wi h mo e han wo o h ee deg ees o eedom, hey a e ha d o impossible o implemen and compu a ionally p ohibi i e because hey equi e he cons uc ion o he Cobs acle in he con igu a ion space. Recen ly, much s udy is cen e ed in sampling-based mo ion planning o p o ide e icien solu ions o pa h planning by a oiding he need o compu e he whole C-space. I esul s in p obabilis ic comple e planning. The co e o sampling-based mo ion planning is o use andom- based echniques o sample he C-space and o in e connec hose collision- ee con igu a ions as oadmaps o ees o cap u e he connec i i y o C ee. To cope wi h high dimensional deg ees o eedom, i is ound as an app op ia e s a egy o educe he complexi y o p oblems. Some sampling-based mo ion planne s a e in es iga ed by (Elbanhawi and Simic,2014). The mos popula app oaches a e ee-based echniques, which p o ide a single que y, and oadmap-based echniques, which p o ide mul iple que ies as desc ibed below. I planning is ca ied ou in s a ic en i onmen s ha emain cons an a each manipula ion s ep, oadmap-based app oaches p o ide be e solu ions (e en hough hey a e cos ly o be implemen ed due o he need o explo ing he whole C-space) because hey make i possible o se mul iple que ies, like Pa h planning based on he P obabilis ic Roadmap Me hod (PRM) (Ka aki e al.,1996) ha wo ks in wo phases. The i s phase is he cons uc ion phase, ha spends a speci ic amoun o ime sampling C ee and in e connec ing samples wi h simple collision- ee pa hs o ming a oadmap. The second phase is he que y p ocess, which connec s a s a con igu a ion o a goal con igu a ion by using g aph sea ch echniques. In manipula ion planning, howe e , since he s a us o he en i onmen is al e ed a e execu ing each ac ion, ee-based algo i hms ha e be e solu ions because hey p o ide speci ic explo a ions o he C-space, e.g., ocused on a gi en que y. In his case, he en i onmen is econ igu ed and he C-space mus be again explo ed. He e, some o he well-known single que y sampling-based mo ion planne s a e e iewed as hey i be e in manipula ion p oblems and some o hem will be used la e in his hesis. Page 16 o 151 2.1. Manipula ion planning Pa h planning based on he Rapidly Explo ing Random T ee (RRT) (LaValle and Ku ne ,2001) explo es he con igu a ion space by expanding se e al b anches o a ee. In he gene ic RRT algo i hm, a ee is ini ialized a he oo whe e he ini ial s a e is placed and i inc emen ally g ows owa ds he goal con igu a ion along andom di ec ions biased by he less explo ed a eas. The RRT-Connec planne (Ku ne and LaValle,2000) is a a ian o RRT. The e a e wo ees oo ed a he s a and goal con igu a ions ha g ow o mee each o he . The me hod p o ides subs an ial imp o emen s in he sea ch e iciency. The RRT* planne (Ka aman and F azzoli, 2011) has been de eloped ha minimizes he cos o he e u ned solu ion. A e g owing he ee as done in he basic RRT algo i hm, he RRT* cap u es he se neighbo nodes No each new added nodes o e i y i i can be eached h ough hem wi h a less cos and, in his case, edges a e ewi ed. Then, his p ocedu e is epea ed o he nodes in N, which may o be ewi ed acco dingly. Kinodynamic Mo ion Planning by In e io -Ex e io Cell Explo a ion (KPIECE) planne is pa icula ly designed o complex dynamical sys ems (¸Sucan and Ka aki,2009). KPIECE g ows a ee o mo ions by applying andomly sampled con ols o a andomly sampled ime du a ion om a ee node selec ed as ollows. The s a e space is p ojec ed on o a lowe dimensional space ha is pa i ioned in o cells in o de o es ima e he co e age. As a esul o his p ojec ion, each mo ion will be pa o a cell, being each cell classi ied as an in e io o ex e io cell depending on whe he he neighbo ing cells a e occupied o no . Then, he selec ion o he cell is pe o mance based on he impo ance pa ame e ha is compu ed based on: 1) he co e age ( he cells ha a e less co e ed a e p e e ed o e he o he s); 2) he selec ion ( he cells ha ha e been selec ed less numbe o ime a e p e e ed); 3) he neighbo s ( he cells ha ha e less neighbo s a e p e e ed); 4) he selec ion ime ( ecen ly selec ed cells a e p e e ed); 5) he expansion (easily expanded cells a e p e e ed o e he cells ha expand slowly). The cell ha has maximum impo ance will be chosen, and a node o one o he mo ions o he cell will be andomly selec ed. The p ocess con inues un il he ee o mo ions eaches he goal egion. This app oach is used while using physics-based mo ion planning, which is also used o g asp-in- he-clu e asks, whe e he obo can in e ac wi h he obs acles obs uc ing he pa h owa ds he goal, mo ing hem away. 2.1.3 Combina ion o ask and mo ion planning The inc easing emphasis on eal wo ld applica ions has led AI planning esea che s o de elop algo i hms and sys em ha mo e closely ma ch ealis ic planning p oblems, in which he equi ed planning ac i i y is o en dis ibu ed o con inual. Dis ibu ed planning e e s o si ua ions whe e he planning ac i i y is dis ibu ed ac oss mul iple obo s, p ocesses o si es. Con inual planning e e s o an ongoing, dynamic p ocess in which planning and execu ion a e in e lea ed. A se o ac ions should be conside ed in each ask desc ip ion. Fo ins ance, o a manipula ion ask, he se o ac ions may include ansi (a obo mo es wi hou ha ing an objec ), ansmi (a obo mo es an objec o he goal), push, pull, pick and place. These ac ions should be o ganized in e ms o he cons ain s ha may exis be ween hem, and an e icien s a egy is equi ed o selec he sequence o excu able ac ions. Fo applying each ac ion, a ask Page 17 o 151 CHAPTER 2. Rela ed Wo k planne has o e alua e whe he he p econdi ions a e sa is ied o no in o de o pe o m he equi ed ac ion. Complemen a y o mo ion and manipula ion ac ions, sensing and easoning ac ions can also be de ined and used o educe unce ain y and ind a obus plan. In gene al, combina ion o ask and mo ion planning needs o sea ch in symbolic and geome ic space. A ask plan, including a sequence o symbolic ac ions, does no p omise ha ac ions become easible in e ms o geome y condi ions. This equi es he combined sea ch and mo i a es he esea che s o in es iga e e icien ways o he combina ion a bo h le els in o de o acqui e a easible plan. Combining ask and mo ion planning is an ac i e esea ch a ea in Robo ics and AI. This sec ion explains di e en s a egies employed o combine ask and mo ion planning wi h espec o di e en ypes o applica ions whe e he ini ial s a e and ac ion e ec s a e ully obse able. The e a e wo main ways o combining ask and mo ion planning in o ma ion: simul aneously o in e lea ed. The simul aneously app oach like he wo k p esen ed in (Akba i, 2018), accoun s o geome ic in o ma ion by calling a mo ion planne while ask planning is being pu sued. The manipula ion plan is hen a ailable a e he ask planning p ocess is e mina ed. The in e lea ed app oach like he wo k p esen ed in S i as a a e al. (2014), decouples mo ion planning om he ask planning pa . Task planning i s gene a es he sequence o ac ions, and hen call a mo ion planne o e alua e he easibili y o he plan. Upon ailu e, geome ic cons ain s a e ed back o he ask planne and he p ocess esumes. This can be epea ed se e al imes un il a easible manipula ion plan is achie ed. Nex , he TAMP app oaches a e p esen ed in which ask planning and geome ic easoning a e mo e igh ly in e wined based on he p oposed ways o combina ion. FF-based TAMP: In his line, hese s udies a e based on he simul aneously combina ion me hod. The s udies in (Cambon e al.,2009a) p esen an algo i hm which sea ches a symbolic and geome ic le els, whe e a mo ion planne calls he ask planne o guide oadmap sampling. Guidance is p o ided by a heu is ic alue based on he symbolic dis ance o he goal. In (Ga e e al.,2015) p oposed an app oach, called FFRob, which compu es he heu is ic alue by analyzing he easibili y o ac ions wi h a Condi ional Reachabili y G aph (CRG) based on a modi ica ion o PRM planne . I equi es a p e-p ocessing s ep o ini ialize he CRG by sampling objec s poses and obo con igu a ions, and de e mining condi ions unde which hese samples a e eachable o no . In (Akba i,2018), a simul aneous TAMP app oach is p oposed o e icien ly deal wi h bi- manual obo manipula ion p oblems in cons ained en i onmen s. The p oposed app oach is a heu is ic-based planne , which sea ches o a plan in s a e space, and akes in o conside a ion geome ic cons ain s while compu ing heu is ic alues. Speci ically, wo ypes o geome ic easoning p ocesses a e used: a) geome ic easoning abou placemen s o objec s, g asping poses, and in e se kinema ic solu ions; b) geome ic easoning abou mo ion. The o me in ol es Spa ial, Reachabili y, and Manipula ion easoning, which a e used o accoun o geome ic cons ain s in heu is ic alues. The p oposed heu is ic is able o co e a la ge ange o able op manipula ion p oblems. I igu es ou and excludes un easible mo ion planning que ies Page 18 o 151 2.2. Knowledge ep esen a ion using on ologies which ha e kinema ic p oblems o collisions in hei s a and goal con igu a ions, and guides s a e space sea ch. The la e calls a mo ion planne o alida ing s a e ansi ions and ei he e u ns a pa h o p o ides eedback in case o ailu e. The s a e o planne is upda ed by he cons ain s which a e de ec ed using bo h geome ic easoning p ocesses. Hie a chical-based TAMP: The wo ks in his di ec ion ollow he simul aneously s a egy. The wo k in (de Sil a e al.,2013) ocuses on a combina ion based on he HTN planne . I acili a es back acking a di e en le els, also including an in e lea ed back acking p ocedu e. The applica ion o combining hie a chical ask and mo ion planning has been used by (Alami e al., 2014) and (Alili e al.,2010) o a eamma e obo and obo ic assis an . They use di e en geome ic easoning p ocesses o ind ou a easible plan. LTL-based TAMP: LTL-based mo ion planning is a hyb id app oach ha p o ides a amewo k o desc ibe complex mo ion planning asks in e ms o empo al goals, and ha plans in disc e e and con inuous spaces. The wo k in (He e al.,2015) applied TAMP using he LTL ask planne . This app oach is done ollowing he in e lea ed app oach. Mo ion planning e alua ion launches a e a ask plan is p o ided. In he case o ailu e, ask planning inpu can be upda ed by a se o cons ain s in o de o ind ano he plan. The au ho s claim he planne is capable enough in mo ing away objec s ha block desi ed execu ions wi hou equi ing back acking. Cons ain -based TAMP: In his di ec ion, all he app oaches a e based on he in e lea ed combina ion me hod. The wo k in (Lag i oul e al.,2012) and (Lag i oul e al.,2014) in oduce he concep o geome ic back acking, which deno es he sys ema ic sea ch p ocess in he space o g asps and placemen s when ins an ia ing a symbolic plan. They use linea cons ain s gene a ed om symbolic ac ions and he kinema ic model o he obo in o de o p une he space o g asps and placemen s. The wo k in (Dan am e al.,2016a) p oposed he I e a i ely Deepened Task and Mo ion Planning me hod using he Sa is iabili y Modulo Theo ies (SMT). I inc emen ally de ec s cons ain s and keeps dynamically adding o elimina ing a numbe o ask cons ains based on he eedback ob ained om he RRT-Connec mo ion planne . The app oach is able o ind an al e na i e plan when an un easible one is iden i ied. I i s inds he ask plan, and hen mo ion planning is employed o e alua e i s easibili y. The wo k p esen ed by (Lag i oul and And es,2016) add esses TAMP by sol ing a culp i de ec ion p oblem. In he case o ailu e a he geome ic le el, a logical explana ion is compu ed. This explana ion is ed back o he Answe Se P og amming (ASP) ask planne , which p unes en i e amilies o plans leading o simila ailu es. The cycle epea s un il a easible plan is ound. 2.2 Knowledge ep esen a ion using on ologies Knowledge ep esen a ion echniques a e conce ned wi h s uc u ing concep ual knowledge such ha i is usable o easoning asks done by a i icial sys ems (e.g. obo s). A b ie in oduc ion o on ologies and i s languages is he ollowing: Page 19 o 151 CHAPTER 2. Rela ed Wo k On ology: An on ology is a s uc u ed way o ep esen ing knowledge. Fo mally, i is de ined as "an explici , o mal speci ica ion o a sha ed concep ualiza ion" (G ube ,1995). In his de ini ion, he e m o concep ualiza ion e e s o he abs ac models o an en i y in a domain. These models a e achie ed by de ining i s ele an concep s along wi h hei ela ions. The simples example o an on ology is a axonomy. I is a hie a chical s uc u e exp essing he uni e se o discou se based on ela ions, such as is-a and has-a be ween concep s and ins ances o a class. These concep s, ins ances and ela ions a e exp essed in o mal languages. Fo mal languages o ep esen ing knowledge: Fo mal languages a e used o exp ess on ologies (in e ms o concep s, ela ions and ins ances) and di ec ly a ec s he eusabili y o he on ologies. The e a e h ee main ca ego ies o o mal languages o ep esen ing knowledge: F ame based, Fi s -O de Logic based and Desc ip ion Logic (DL) based languages (Baade e al.,2017). 1. F ame based: I is a p ima y da a s uc u e used in a i icial in elligence. F ame based language is o iginally de eloped o deduc i e da abases. I ocuses on explici and in ui i e ep esen a ion o knowledge. The languages ha a e de i ed om namely ame based a e: F-logic, Open Knowledge Base Connec i i y (OKBC), and Knowledge Machine (KM). 2. Fi s -O de Logic: I is a s anda d o he o maliza ion o ma hema ics in o axioms ( ac s). I allows he use o sen ences ha con ain a iables. The languages ha a e de i ed om Fi s -O de Logic a e: Common Logic, CycL, and Knowledge In e change Fo ma (KIF). 3. Desc ip ion logic (DL): I p o ides an ex ension o namely ame based, wi hou going so a as o i s -o de logic. The languages ha a e de i ed om DL a e: KL-ONE, RACER, and OWL. A mo e de ailed desc ip ion o on ologies and DL can be ound in Appendix B Abundan s udies in es iga ed he use o obo knowledge in o de o connec low-le el da a (senso s) wi h high-le el in o ma ion (on ology). These s udies co e se e al a eas such as objec ecogni ion, ask planning, o na iga ion. Recen ly, some on ologies s anda ds in obo ics ha e been es ablished by in eg a ing some o he wo ks done in hese a eas, wi h he aim o making knowledge mo e gene al, and sha eable ac oss many domains. On ologies s anda ds will be explained in subsec ion 2.4. 2.3 The use o knowledge in di e en domains Many s udies ha e in es iga ed he use o knowledge in planning, like (Teno h and Bee z, 2009) and (Ruiz-Sa mien o e al.,2017), ha ca ego ize knowledge abou he wo ld in o e minological knowledge (TBOX), and asse ional knowledge (ABOX). The o me con ains a hie a chy o concep s, such as in e ac ion and ac ion, and hei ela ions, whe eas he la e con ains indi iduals ha a e ins an ia ions o hese concep s. In he na iga ion a ea, some Page 20 o 151 2.4. Knowledge uppe -le el ounda ions e o s wo ks such as (Lim e al.,2011), (Gemignani e al.,2016) and (K ieg-B ückne e al.,2005) use a me ic map and a opological map o de ine he obo en i onmen . The me ic map is used o he geome ical ep esen a ion o he obo wo kspace in e ms o ee and occupied a eas, while he opological map is used o cap u e he opology o he wo kspace. In he manipula ion planning domain, wo ks such as (Akba i e al.,2018c) p opose an on ological amewo k o o ganize he knowledge needed o physics-based manipula ion planning, allowing o de i e manipula ion egions and beha io s. O he s udies ha e in es iga ed he use o he obo knowledge in speci ic ields in o de o connec he exis ing low-le el da a wi h he high-le el in o ma ion (Rai and Hong,2013) and (Chang e al.,2009). This connec ion can be used in se e al di ec ions: objec ecogni ion and ca ego iza ion (Johns on e al., 2008a), con ex modeling (Young Cheol Go and Joo-Chan Sohn,2005) and con ex easoning (Anagnos opoulos and Hadjie hymiades,2009), (Da gie,2009), ask planning (Cambon e al., 2009b), locomo ion (Cha e jee e al.,2005). Fo objec ecogni ion and ca ego iza ion, a isual concep on ology ha is composed o spa ial concep s, spa ial ela ions, colo concep s, and ex u e concep s can all be used as an in e media e laye be ween domain knowledge and image p ocessing p ocedu es du ing he knowledge acquisi ion phase. Algo i hms ha e also been de eloped o isual concep lea ning, ea u e selec ion, and aining (Maillo e al.,2004). Mo eo e , symbol g ounding, which occu s be ween senso y ea u es and hei symbolic ep esen a ions (Johns on e al.,2008b), equi es obus objec ecogni ion me hods, combining many local ea u es wi h se e al o he isual ea u es such as shape, colo , and ex u e in a p obabilis ic and/o on ological app oach. 2.4 Knowledge uppe -le el ounda ions e o s Nowadays, he use o human- obo o obo - obo collabo a ion is playing a signi ican ole in obo ics. One o he basic equi emen s o any ype o collabo a ion is he need o a common ocabula y along wi h clea and concise de ini ions. The e o e, he need o a s anda d and well- de ined knowledge ep esen a ion is becoming mo e e iden . The s anda ds o on ologies a e discussed in (IEEE-SA,2015). On ologies o Robo ics and Au oma ion Wo king G oup (ORA WG) is one o he ecommended s anda d in obo ics (Schleno e al.,2012), as explained below. ORA WG: The goal o his wo king g oup is o de elop a s anda d on ology and associa ed me hodology o knowledge ep esen a ion and easoning in obo ics and au oma ion, oge he wi h he ep esen a ion o concep s in an ini ial se o applica ion domains. The s anda d p o ides a uni ied way o ep esen ing knowledge. I allows o ans e knowledge among any g oup o humans, obo s and o he a i icial sys ems. This g oup is comp ised o ou sub-g oups en i led: Uppe On ology/Me hodology(UpOM), Au onomous Robo s (AuR), Se ice Robo s (SeR), and Indus ial Robo s (InR). The e a e se e al gene al obo ics s anda ds ha ha e been conside ed in he Page 21 o 151 CHAPTER 2. Rela ed Wo k Planning On ology The planne is d i en by compa ing goals, ep esen ed in he TBox, wi h belie es, ep esen ed in he ABox, and con olled by me a knowledge called planning s a egy. The planning s a egy de e mines which pa s o he on ology a e o in e es in he cu en phase, how s eps a e o de ed, and how hey a e pe o med in e ms o how he knowledge base is o be manipula ed. Possible planning decisions a e ep esen ed in a da a s uc u e called planning agenda. Planning agendas a e o de ed sequences o s eps ha each, when pe o med, modi y he belie s a e o he obo in some way. The planne succeeds i he belie s a e is a p ope ins an ia ion o he goal desc ip ion. Di e en asks equi e di e en s a egies ha ocus on di e en pa s o he on ology, and ha ha e specialized ules o p ocessing he agenda. The s a egy o planning an assemblage, o example, ocuses on ela ions de ined in he assembly on ology. Planning o pu away pa s, on he o he hand, is mainly conce ned wi h spa ial ela ions. S a egies a e associa ed o en i ies ha should be planned wi h hem. To his end, he ela ion needsEn i y is de ined ha deno es en i ies ha a e planned by some s a egy. S a egies asse a uni e sal es ic ion on his ela ion in o de o de ine wha ype o en i ies can be planned wi h hem. Fo he assemblage planning s a egy, o example, he axiom is asse ed: ∀needsEn i y.(Assemblage ∨AssemblyConnec ion)(2.5) Planning decisions may no co espond o ac ions ha he obo needs o pe o m o es ablish he decisions in i s wo ld. Some decisions a e pu ely i ual, o only one missing piece in a se o missing in o ma ion equi ed o pe o m an ac ion. The mapping o planning decisions o ac ion en i ies is pe o med in a ule-base ashion. These ules a e desc ibed using he AgendaAc ionMappe concep , and a e linked o he s a egy ia he ela ion usesAc ionMappe . Each AgendaAc ionMappe u he desc ibes wha ypes o planning decisions should ac i a e i . This is done wi h agenda i em pa e ns ha ac i a e a mappe in case a pa e n ma ches he selec ed agenda i em. These a e linked o he AgendaAc ionMappe ia he ela ion mapsI em. Finally, he AgendaAc ionPe o me concep is de ined which is linked o he s a egy ia he ela ion usesAc ionPe o me .AgendaAc ionPe o me p o ide acili ies o pe o m ac ions by mapping hem o da a s uc u es o he plan execu i e, and in oking an in e ace o ac ion execu ion. They a e ac i a ed based on whe he hey ma ch a pa e n p o ided o he las agenda i em. Al hough hese on ologies a e use ul o such manipula ion applica ions which ha e a well-s uc u ed en i onmen , hey lack o show how i can be used in semi/uns uc u ed en i onmen s which equi es he in eg a ion o a geome ic easoning module o check he easibili y o he ac ions in such cases o combining wo pa s o e en i he pa h owa d a ce ain objec is a collision- ee. Pa o he cu en hesis will y o ill his gap. Page 28 o 151 Chap e 3 Knowledge Guidance o Task and Mo ion Planning 3.1 In oduc ion This chap e in oduces seman ic manipula ion knowledge o manipula ion planning. The manipula ion planning sys em includes he combina ion o ask and mo ion planning (TAMP) le els. The use o abs ac knowledge can enhance and acili a e he planning capabili ies in bo h le els and gi e mo e au onomy o he obo s o pe o m asks. Many app oaches exis o he ep esen a ion o knowledge and one o hem a e on ologies, which a e used o s uc u e he knowledge in e ms o concep s and ela ions. Reasoning p ocesses o e seman ic knowledge can be hen applied in o de ha obo s in e he pa icula si ua ion in hei wo kspace. 3.2 P oblem s a emen and p oposed solu ion 3.2.1 P oblem o maliza ion In manipula ion planning, dynamic in e ac ions be ween he objec s and he obo s play a signi ican ole. In his scope, dynamic engines, such as Open Dynamic Engine (ODE-h p: //www.ode.o g/) allow o conside hem wi hin mo ion planne s, gi ing ise o physics-based mo ion planne s ha conside he pu pose ul manipula ion o objec s. In his con ex , on he one hand, a he geome ic le el, he ep esen a ion o knowledge ega ding how he objec s ha e o be manipula ed eases a seman ic-based easoning ha educes he compu a ional cos o physics-based planne s. In his wo k, an on ology amewo k is p oposed o o ganize he 29 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning knowledge needed o physics-based manipula ion planning, allowing o de i e manipula ion egions and beha io s. A seman ic map is cons uc ed o ca ego ize and assign he manipula ion cons ain s based on he obo , he objec s and he ype o ac ions. The on ology amewo k can be que ied using Desc ip ion Language o ob ain he necessa y knowledge o he obo o manipula e he objec s in i s en i onmen . On he o he hand, a he symbolic le el, au onomous indoo se ice obo s a e supposed o accomplish asks, like se e a cup, which in ol e sequences o manipula ion ac ions. Pa icula ly, o complex manipula ion asks ha a e subjec o geome ic cons ain s, spa ial in o ma ion is equi ed, oge he wi h he way in which hese objec s can be manipula ed. In his line, in his chap e an on ological-based easoning amewo k called Pe cep ion and Manipula ion Knowledge (PMK) is p oposed as a guidance module o ask and mo ion planning. The PMK includes: (1) he modeling o he en i onmen in a s anda dized way o p o ide common ocabula ies o in o ma ion exchange in human- obo o obo - obo collabo a ion, (2) a senso y module o pe cei e he objec s in he en i onmen and asse he on ological knowledge, (3) an e alua ion-based analysis o he si ua ion o he objec s in he en i onmen , in o de o enhance he planning o manipula ion asks. The wo k desc ibes he concep s and he implemen a ion o PMK, and p esen s an example demons a ing he ange o in o ma ion he amewo k can p o ide o au onomous obo s. 3.2.2 P oposed F amewo k o e iew Figu e 3.1: Knowledge-based easoning amewo k o ask and mo ion planning (KTAMP). Wi h he aim o execu ing asks au oma ically, he in eg a ion o se e al laye s and modules is equi ed, co e ing pe cep ion, knowledge ep esen a ion and easoning, and planning a symbolic and geome ic le els. Page 30 o 151 3.3. A Knowledge P ocessing F amewo k o Physics-based Manipula ion Planning The p oposed amewo k Knowledge-based easoning o Task and Mo ion Planning (KTAMP) is composed o h ee main laye s, as shown in Fig. 3.1 planning and execu ion, knowledge, and assis an (low-le el) laye . The planning and execu ion laye con ains wo modules, he ask planning and he ask manage modules. The o me includes a ask planne o compu e a sequence o ac ions o be done, which equi es a p oblem and domain desc ip ion o se he ini ial scene, including he s a e o he wo ld en i ies, and he goal s a e. The la e p o ides in e aces o communica e wi h he agen s/ope a o s (e.g., obo s, humans o senso s). I also keeps moni o ing he execu ed ac ions o a omic ac ions and i e u ns a ailu e signal o he eco e y module i an e o occu s. Mo eo e , i has a p ocedu al s uc u e o each s ep o a ac ion. This s uc u e is o mally de ined as a wo k low ha can be au oma ically execu ed h ough in e acing exis ing so wa e componen s o he obo con ol sys em, e.g., execu ing a pickUp equi es a call o he In e se Kinema ics (IK) module o check eachabili y o g asping he objec s hen inding a collision- ee pa h owa ds a g asping con igu a ion. A pu Down ac ion mus sea ch o a ailable placemen oom. The knowledge laye con ains he Awa eness module o guide he planning and execu ion laye . I is composed o he PMK knowledge ha consis s o manipula ion and pe cep ual knowledge. The o me assis s he obo o igu e ou he en i onmen al en i ies, wo kspace and planning. The la e assis s he obo o igu e ou which a e he p ope algo i hms and pa ame e s o be used o he a ailable senso s in o de o ex ac da a. The PMK knowledge is desc ibed in de ail below in Sec ions 3.3 and 3.4. Finally, he assis an laye p o ides he low-le el modules ha allow o deal wi h: pe cep ion issues, like inding ou which senso s can be used o a sensing ac ion in a gi en si ua ion, which a e deal by he sensing module, and geome ic issues, like de e mining i a con igu a ion is collision- ee o i an in e se kinema ic solu ion exis s o a g ippe pose, which a e deal by he geome ic module. 3.3 A Knowledge P ocessing F amewo k o Physics-based Manipu- la ion Planning Desc ip ion o OUR-K Knowledge Classes The on ology app oach o physics-based manipula ion planning p oposed he e is he i s e sion o PMK. I is inspi ed by he On ology-Based Uni ied Robo Knowledge o Se ice Robo s in Indoo En i onmen s OUR-K (Lim e al.,2011). The main con ibu ion o he OUR-K amewo k is he es ablishmen o an on ology-based uni ied knowledge o se ice obo s in human en i onmen s by in eg a ing low-le el da a Page 31 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.2: The s uc u e o OUR-K on ology (Lim e al.,2011) (senso y da a) wi h high-le el knowledge (con ex in o ma ion). This amewo k is desc ibed wi hin a concep hie a chy h ough he use o an on ology. As shown in Fig. 3.2, i is composed o i e main classes: ea u e, objec , space, con ex , and ac ion. Each class has h ee le els (sub-classes) (excep ea u e class ha has wo), and each le el has h ee on ological laye s: me aon ology, on ology, and on ology ins ance. Me aon ology laye is used o ep esen gene ic in o ma ion, such as he concep o physical objec in he se ice obo ics ield. On ology schema laye is a laye used o domain speci ic knowledge, o ins ance, in se ice obo ics, on ology laye con ains he knowledge o a pa icula domain such as ki chen. On ology ins ance laye is used o s o e he in o ma ion o he objec s (such as hei ea u es). Because on ologies in gene al a e an objec -o ien ed and ame-based language he me aon ology laye can p o ide a empla e o on ology laye o build e minology, while he on ology ins ance laye can be de ined as an indi idual ame. The in o ma ion o on ological classes, p ope ies, and ins ances is ans e ed wi hin unidi ec ional easoning in he same knowledge le el. Whe eas, bidi ec ional easoning ela es se e al knowledge classes o knowledge le els. OUR-K classes a e desc ibed below: Fea u e class is used o de ine how objec s a e pe cei ed. I has wo le els: pe cep ual ea u e and pe cep ual concep . In pe cep ual ea u e le el, he eal en i onmen is desc ibed om he pe cep ual pe spec i e, i.e. senso s, ha pe cei e ea u es o ins ances such as colo , ex u e, and scale in a ian ea u e ans o m (SIFT). In pe cep ual concep le el, concep s a e g ounded o pe cep ual ea u es. Page 32 o 151 3.3. A Knowledge P ocessing F amewo k o Physics-based Manipula ion Planning Objec class is used o de ine objec s, hei unc ionali y and hei pa s. I is composed o h ee le els: pa -objec le el, objec le el and compound le el. Pa -objec includes pa s o objec s acco ding o hei unc ionali y ( o example, body and handle each has i s own unc ionali y, i.e. con aining and g asping espec i ely). Objec le el includes objec name and unc ionali y, o ins ance, he composi ion o a cup is body and handle. In compound le el, closely ela ed objec s ha can be u ilized oge he a e linked (e.g. a cup and a sauce ). Wo kspace class builds a seman ic map and uses i o enhance he ep esen a ion o he en i onmen , which equi es geome ical, posi ional in o ma ion and quali a i e ea u es, o exp ess ela ions. The cons uc ion o a seman ic map depends on wo ypes o maps: me ic and opological. The o me de ines a eas in he en i onmen , which may be emp y o occupied. The la e con ains he in o ma ion o he opology o he ee egion (e.g. wi h a g aph ex ac ed om a o onoi diag am). The ela ion be ween hese maps and objec s is s o ed in he seman ic map. Con ex class is used o unde s and he si ua ion o he objec s in he en i onmen . This si ua ion is de i ed om wo ypes o ela ions: spa ial and empo al, which a e de ined in spa ial and empo al le els, espec i ely, such as c owd, which means ha he obo will encoun e some obs acles. The spa ial le el con ains he spa ial ela ions (such as on, in, le , and igh unc ions) and space classes (seman ic map). Tempo al le el con ains he empo al ela ions such as be o e, a e , o e lap, mee . Ac ion class is used o de ine a ask and he way o execu ion i . Ac ion class consis s o h ee le els: p imi i e beha io , sub- ask and ask. In p imi i e beha io , pe cep ual, mo ion and manipula ion beha io s a e de ined (e.g. u n, go o, ex ac colo , ex ac SIFT). In a sub- ask le el, a sho - e m sequence o beha io s a e de ined, such as go oSpace, localiza ion. A ask such as na iga e, deli e y is de ined in ask le el. I can be decomposed in o sub- asks, which can be decomposed in o p imi i e beha io s. Fo example, ecognizeObjec is a ask o inding an objec wi hin an image (e.g. ex ac colo , ex ac SIFT). Fi s P oposal o a Physics-based Manipula ion On ology In p e ious wo ks (Akba i e al.,2016a), he au ho s p oposed a manipula ion on ology o ep esen knowledge o ace he manipula ion p oblems a mo ion planne has o deal wi h when he obo mo es and encoun e s obs acles in he en i onmen . Two classi ica ion o objec s a e p oposed in his on ology, ixed bodies and manipula able bodies (i.e. obs acles ha can be pushed away). The o me emains s a ic du ing he whole planning p ocess, e en i collision happens wi h o he manipula able objec s. The la e can be manipula ed du ing he planning. The manipula able bodies a e classi ied as ee manipula able bodies and cons ained-o ien ed manipula able bodies. The ee manipula able bodies can mo e in any di ec ion (acco ding o he dynamics o igid bodies) when collision occu . The cons ain -o ien ed manipula able bodies ha e some allowable mo ion di ec ions and o he s a e es ic ed. The manipula ion cons ain s a e modeled by de ining he manipula able egion om whe e he objec can be Page 33 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning pushed om (i.e. whe e he obo should be loca ed o apply a pushing o ce). The knowledge is s uc u ed by de ining six classes de i ed om a gene al manipula ion class. These classes desc ibe he ini ial s a e, goal s a e, ac ion, egion, objec ype, and objec elemen s. Ini ial s a e class desc ibes an ini ial loca ion o he obo , while goal s a e class shows he inal loca ion o he objec s. Ac ion class con ains di e en manipula ion p imi i e ac ions, like pick, place and push. Region class de ines h ee sub-classes: manipula ion, objec , and goal egion. Manipula ionRegion sub-class de ines he egion o manipula ion, i.e. om whe e he mobile obo can in e ac wi h he objec . Objec Region sub-class is de ined as a bounding box o an objec . GoalRegion sub-class is a ci cula egion de ined a ound he goal s a e. Objec Type class de ines he manipula able objec s and ee objec s and hei cons ain s. Objec Elemen s desc ibe he ea u e o he objec s. Al hough his p oposal co e s he manipula ion cons ain s, i lacks o co e knowledge ela ed o en i onmen al en i ies, pe cep ion de ices, agen s, and planning. 3.3.1 The P oposed F amewo k o Physics-based Manipula ion The ep esen a ion o obo knowledge wi h an on ology has in e es ing p ope ies like sha eabili y and exchangeabili y (Ali ezaie,2015). Knowledge is ep esen ed by de ining he p ope ies and ela ions be ween concep s ha p o ide a ich seman ic desc ip ion o aid he sol ing o pa icula p oblems. The aim o his wo k is o o malize a amewo k ollowing he s uc u e o he OUR-K on ology o cope wi h manipula ion planning p oblems by p o iding he necessa y knowledge (e.g, pe cep ion, and planning) o comple e scena ios. As shown in Fig. 3.3, he p oposed on ology amewo k has six classes. The objec class and he con ex class a e he same as in he OUR-K amewo k. The o he ou a e desc ibed in he ollowing subsec ions. Fea u e class In he p oposed amewo k, he ea u e class has wo le els: physical concep and physical ea u e. The concep s o igidbody and in e ac ion a e de ined in he concep le el, and hei ea u es in e ms o ma e ial and in e ac ion pa ame e s in ea u e le el. Fig. 3.4 desc ibes he on ology schema o he ea u e class. Fo example, in e ac ion is a concep de ined as “Con ac be ween wo su aces”, while i s ea u es depends on he ype o ma e ial, i.e. each ma e ial has i s p ope ies like ic ion coe icien s, densi y, e c. They a e linked oge he (ma e ial and i s p ope ies) ia axioms ( ac s). Fo example, o handle a cup (which is made om A), he in e ac ion occu s be ween wo igid bodies, cup and obo (i s end-e ec o is made om B), and each has i s physical p ope ies, as desc ibed below using DL: In e ac ion Page 34 o 151 3.3. A Knowledge P ocessing F amewo k o Physics-based Manipula ion Planning Figu e 3.3: Classes o he p oposed physics-based on ology o manipula ion, based on he OUR-K amewo k. Figu e 3.4: Schema o ea u e class ha explains he hie a chy o he concep s, ea u es, and ela ions ia axioms ∧∃hasSupe class(Rigidbody, P hysicalObjec ) ∧∃hasIn e P a ame e (in e ac ionP a ame e , Rigidbody) ∧∃hasma e ial(ma e ial, Rigidbody) ∧∃isma e ial(A, ma e ial) ∧∃isma e ial(B, ma e ial) ∧∃has ea u e( ic ion, in e ac ionP a ame e ) Axioms o he ea u e class de ine ha he igid bodies ha e a cons i uen ma e ial and ha his ma e ial has p ope ies, such as ic ion coe icien , densi y, slip, CFM (Cons ain Fo ce Mixing), and bounce. The physical ea u es a e used by he ODE simula o o accu a ely model he physical en i onmen . Page 35 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.5: Schema o wo kspace class ha desc ibes he ou come o seman ic map. The legend o di e en colo shows ha he classes in o ma ion ha is equi ed o gene a e he seman ic map. Ac o class Ac o class is a new-de ined class ha desc ibes he p ope ies o he obo in he en i onmen . I consis s o h ee le els: obo kinema ics, dynamics, and cons ain s. Kinema ic-le el de ines he kinema ic s uc u e o he obo and i s loca ion in he wo kspace, whe eas dynamic-le el con ains he dynamic pa ame e s o he join s o he obo , such as dumping, join s i ness, and maximum e o s. In he cons ain s-le el, based on kinema ics and dynamics ea u es, he obo wo king cons ain s a e ex ac ed. Wo kspace class In ou amewo k, he Wo kspace class con ains h ee knowledge le els: me ic map, opological map, and seman ic map. A me ic map con ains emp y and occupied spaces (by ei he objec s o he obo s). Topological map de ines he opology o he wo kspace, including whe e he objec s a e loca ed in he wo kspace. Seman ic map ca ego izes physical objec s in he en i onmen along he manipula ion cons ain s. These cons ain s a e de ined as a unc ion o he obo . I means ha seman ic map Sdepends on ou pa ame e s S= (O, R, A, T ), whe e: Ois he se o objec s ha is p o ided by objec class, Ris he se o obo wo king cons ain s ha is p o ided by he ac o class, Ais an ac ion ha is p o ided by he ac ion class, and Tis a opological Page 36 o 151 3.3. A Knowledge P ocessing F amewo k o Physics-based Manipula ion Planning Figu e 3.6: Schema o ac ion class ha is di ided in o h ee sub-classes: p imi i e beha io , sub- ask and ask. map ha is p o ided by he Wo kspace class. The ou come o he seman ic map is o assign he manipula ion cons ain by using DL easoning, i.e. as shown in Fig. 3.5, he ou come o he seman ic map due o easoning is he ype o objec and i s cons ain s. Ac ion class The ac ion knowledge class is composed o h ee le els: p imi i e beha io , sub- ask, and ask le els. A ask is decomposed in o sub- asks, while sub- asks in ol e a combina ion o p imi i e beha io s. The on ology schema o he laye s o he h ee le els in ac ion class is ins an ia ed by a planne . When eques ed o plan, he planne checks on he opological map o know which objec (o obo ) is occupied wi h which space. Then, he seman ic map is used o ex ac he cons ain s o he objec (single o composi e), and he obo w. . he ac ion. Con ex class, pa icula ly empo al le el, can be used by he planne o se he sequence o ac ion o a ask. 3.3.2 Usage in Manipula ion Table- op P oblem To illus a e he p oposal some simula ion examples a e pe o med using The Kau ham P ojec (Rosell e al.,2014), which is a C++ based open-sou ce ool o mo ion planning, ha includes geome ic, kinodynamic, and physics-based mo ion planne s. I uses OMPL (Sucan e al.,2012) o he co e se o planning algo i hms. OMPL is a C++ based open-sou ce lib a y o sampling Page 37 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.11: Main pa s o he sys em: The pe cep ion module, he Pe cep ion and Manipula ion Knowledge (PMK) amewo k, and he TAMP planning module. PMK asse s he pe cep ual da a, builds he IOC-Lab knowledge, and p o ides he easoning p edica es o he planning module. be e ages on a able. The se -up is p epa ed by YuMi, while TIAGo is used o ac ually se e he be e ages o he cus ome s. The PMK amewo k wi h easoning mechanisms is used o p o ide he easoning p edica es ela ed o pe cep ion, objec ea u es, si ua ion analysis, and geome ic easoning. These easoning componen s, discussed in de ail in his sec ion, acili a e he planning p ocess. The TAMP module is a combina ion o he FF ask planne and physics-based mo ion planning (Akba i e al.,2016b). I is used o ac ually plan he ask and p o ide a easible sequence o ac ions o he obo o be execu ed. The execu ion module uses YuMi and TIAGo obo s o execu e he se ing ask. PMK Knowledge S uc u e A p elimina y e sion o PMK s uc u e has been p esen ed in (Diab e al.,2017). I was inspi ed om an on ological schema p oposed in (Lim e al.,2011) o na iga ion asks in indoo en i onmen s, ha desc ibes concep s h ough he use a hie a chy o on ologies composed o h ee laye s: me aon ology laye , on ology schema laye , and on ology ins ance laye as shown in Figu e 3.13 and desc ibed in Sec. 3.3. Mo eo e , examples o how in o ma ion in e ed om Page 44 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion Figu e 3.12: A mo i a ion example o a wo- obo able- op manipula ion ask a he IOC-Lab. hese laye s a e desc ibed in Sec. 5.2. Following his hie a chical schema, we p oposed PMK amewo k 2 o au oma ed manipula ion asks whe e hese laye s a e composed o se en classes: Fea u e, WSobjec , Ac o , Senso , Wo kspace, Con ex Reasoning, and Ac ion (Some concep s’ names ha e been modi ied he e wi h espec o how hey we e p esen ed in (Diab e al.,2017), in o de o i wi h he cu en s anda dized p oposal). A p elimina y e sion o his amewo k basically p o ided he concep s, ela ions, and in e ence mechanism o physics-based mo ion planning o each he obo how o in e ac wi h he igid bodies. Howe e , he in o ma ion ega ding sensing was no conside ed, concep s ha we e p oposed we e no s anda dized concep s, and no knowledge ega ding spa ial and geome ic easoning o combined ask and mo ion planning equi emen s was included. The PMK amewo k p oposed he e co e s all he abo e missing issues. PMK Reasoning Mechanism The equi ed in e ence o TAMP consis s o : geome ic easoning o de e mine obo eachabili y and placemen egion, manipula ion cons ain s analysis o de e mine how o 2h ps://gi hub.com/MohammedDiab1/PMK Page 45 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.13: PMK on ology. in e ac wi h he objec s, and he mo ion analysis o de e mine ac ion easibili y. The easoning mechanism o PMK is di ided in o ou pa s, as shown in Figu e 3.11: easoning o pe cep ion, easoning o objec ea u es, easoning o si ua ion, and easoning o planning. Reasoning o pe cep ion is ela ed o senso s and algo i hms o simply answe ques ions like which a e he senso s he obo has? wha is he co esponding algo i hm o ex ac he pe cep ual da a om he senso ?. Reasoning o objec ea u es copes wi h he ea u es o he objec s such as colo and dimensions. Reasoning o si ua ion analysis is used o spa ially e alua e he objec s ela ions be ween each o he (i.e., cup inside box, and cup is eachable by le a m). Reasoning o planning is used o eason abou he p econdi ions o ac ions, ac ion cons ain s, and geome ic easoning (a m eachabili y, g asping pose eachabili y, and placemen egion). 3.4.3 Why PMK? This sec ion co e s he di e ences be ween PMK and o he knowledge-based p ocessing amewo ks, such as KnowRob (Teno h and Bee z,2017) and OUR-K (Lim e al.,2011), by highligh ing he impo ance o PMK ha is no co e ed by hem. We di ide he di e ences in o wo le els: modeling and easoning p ocess. A he modeling le el, al hough hese app oaches p o ide amewo ks ha include a comp ehensi e way o ep esen ing he on ologies ela ed o how o execu e manipula ion asks, hey lack he ep esen a ion o he me a-le el concep s using he common ocabula ies p o ided by s anda diza ion such as (Niles and Pease,2001b) (i.e., hey do no ollow any s anda diza ion). This may lead o di icul ies in inco po a ing/impo ing o he on ologies unde hei e minologies because o he con lic in he meaning o he concep s o he obo . This may be a p oblem o collabo a i e asks be ween obo s ha equi e some common ocabula ies. A he easoning p ocess le el, hey do no ully co e he a ea o TAMP o acili a e he planning p ocess. Fo example, in mo ion planning, o deal wi h igid bodies in clu e ed en i onmen s, he e is he need o de ine he way o apply ac ions such as push/pull, equi ing a ich seman ic desc ip ion o be ed o he planne , like he physics-based mo ion planne in (Muhayyudin e al.,2018). In ask planning, he obo needs o eason on: (a) he easibili y Page 46 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion o an ac ion a some ins an o he manipula ion planning p ocess (acco ding o objec ea u es, he he s a e o he objec could change and be ou o he obo capabili ies, e.g., i he cup is emp y he s a e is g aspable and i ull he s a e is pushable), (b) he selec ion o he placemen whe e he obo mus place he objec , and (c) he cu en cons ain s. Mo eo e , easoning abou pe cep ion knowledge is equi ed. PMK co e s hese gaps in he modeling and easoning p ocess le els. In he o me , by ollowing he s anda dized concep s p esen ed by SUMO and CORA. In he la e , by p o iding easoning p edica es o co e TAMP needs. 3.4.4 Knowledge Fo mula ion The classes men ioned abo e in Sec ion 3.4.2 we e shown in he me aon ology laye in Figu e 3.13. The concep s in hese classes a e o mula ed he e acco ding o he s anda dized concep s p esen ed in he IEEE 1872 s anda d. The on ology schema and on ology ins ance laye s a e no discussed since hey depend on speci ic domains (in he mo i a ion example, hese ha e been buil o he domain o ou obo ic lab). The o mula ion acco ding o he s anda d uses mainly SUMO, CORA, CORAX, and ROA (see Sec. 2.4). SUMO di ides he en i ies in o wo g oups: physical and abs ac . The physical g oup desc ibes he en i ies ha exis in space- ime, and is subdi ided in o objec and p ocess o ep esen , espec i ely, bodies and p ocedu es. The abs ac g oup desc ibes he en i ies ha do no exis in ime and include ma hema ical cons uc s (IEEE-SA,2015). Following he same modeling s a egy, as shown in Figu e 3.14, PMK di ides he knowledge in o knowledge ela ed o objec s (manipula ion wo ld), knowledge ela ed o p ocesses (manipula ion planning), and abs ac knowledge (manipula ion da a). The manipula ion wo ld knowledge includes he desc ip ion o objec s, obo s and senso s in he wo kspace. The manipula ion planning includes he easoning p ocesses o e PMK. These a e de ailed in he nex subsec ions. The manipula ion da a knowledge ep esen s he ea u es o objec s, such as colo , mass, obo cons ain s (e.g., join limi s) and senso cons ain s (e.g., maximum and minimum measu emen anges). Manipula ion Da a Knowledge I includes he Fea u e class, which is subdi ided in o h ee g adual le els o ep esen da a om low-le el o high-le el (abs ac ). The s anda dized concep o SUMO: Quan i y is used in le el one o desc ibe he quan i ies o he en i onmen , such as objec dimensions, colo , posi ion, o o ien a ion. A new concep Quan i y Agg ega ion has been in oduced in le el wo o include agg ega ions o quan i ies, like w ench ( o ce and o que) o pose (posi ion and o ien a ion). The s anda dized concep o SUMO: A ibu e is used in le el h ee o ep esen abs ac da a like ma ices. These concep s a e modeled and ela ed o he on ology schema laye (which in ou case desc ibes he IOC-Lab domain) o e ie e he equi ed in o ma ion. Fo ins ance, as shown in Figu e 3.15, he Quan i y concep is ela ed o he concep Pose o e ie e he in o ma ion o he g asping pose o YuMi g ippe . Page 47 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.14: S uc u e o he me aon ology laye o PMK i o ollow he s anda d IEEE-1872. Concep s in oduced in his wo k a e shown in black, while hose ha inhe i om he s anda d a e shown in colo : SUMO (blue), CORA (whi e), CORAX ( ed), and ROA (g een). All he e minologies a e de ined in he Appendix. Figu e 3.15: Desc ip ion o some abs ac manipula ion knowledge concep s. Manipula ion Wo ld Knowledge I includes ou classes WSobjec , Ac o , Senso , and Wo kspace. The main le el o he i s h ee classes is le el wo. In hese le els, he s anda dized concep s a e used o de ine he physical objec s and hei unc ionali ies, while le el one (componen -le el) ep esen s he desc ip ion o he componen s o he objec s and le el- h ee (g ouping-le el) ep esen s he desc ip ion ela ed o he g ouping o he objec s in he wo ld. Fo example, a cup has a body and a handle as componen s, and i can be g ouped wi h a sauce . As shown in Figu e 3.14, in he main le el o WSobjec , Ac o , and Senso classes, he s anda dized concep s o SUMO: A i ac , CORA: Robo and SUMO: Measu ing De ice ha e been used, espec i ely. Componen -le els use he new concep s A i ac Componen , Robo Componen and Measu ing De ice Componen . G ouping-le el uses he s anda dized concep s o Page 48 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion SUMO: Collec ion and CORA: Robo G oup o he WSobjec and Ac o classes, espec i ely, and he new concep o Measu ing De ice G oup o he Senso class. These concep s a e modeled and ela ed o he IOC-Lab on ology schema laye o e ie e he equi ed in o ma ion. Figu e 3.15 shows an example o he ela ion be ween he componen -le el o he obo and he physical obo , whe e he YuMi obo and i s g ippe a e de ined unde he Robo and Robo componen me a-concep s, espec i ely. The main le el o Wo kspace class uses he concep om CORAX: Physical En i onmen ha desc ibes he opology o he objec s in he en i onmen (i.e., which a ea is occupied by which physical objec ), he s anda dized concep SUMO: Region is used o desc ibe he geome ical ep esen a ion o he wo kspace in le el-one, and he new concep o Seman ic En i onmen is in oduced in le el- h ee o comple e le el- wo wi h he da a ( ea u es) o he physical objec s. Manipula ion Planning Knowledge The manipula ion planning knowledge ep esen s he pa ha is esponsible o easoning abou he si ua ion o he objec s and he obo , and o planning he asks. This easoning p ocess is done o e he manipula ion en i onmen knowledge o acili a e he planning p ocess o he asks. As shown in Figu e 3.14, his knowledge includes wo PMK classes, Con ex easoning and Ac ion. Each class has h ee le els, om high-le el o low-le el concep s. Le el- h ee ep esen s he symbolic in o ma ion ha depends on he i s wo le els. Fo example, in he Con ex Reasoning class, he s anda dized concep SUMO: Si ua ion ep esen s he s a us o he objec o obo in he wo ld wi h espec o space and ime. To co e bo h, Spa ial Con ex (such as le , igh , on, in) and Tempo al Con ex (such as be o e, a e , mee , o e lap) a e in oduced o desc ibe hem, espec i ely. These concep s a e modeled and ela ed o he IOC- Lab on ology schema laye o e ie e he equi ed spa ial in o ma ion abou he en i onmen en i ies. Fo ins ance, Figu e 3.15 shows he can objec , which is sub-class o A i ac , ha has he p ope y Spa ially Loca ed o epo he on ology wi h i s spa ial loca ion wi h espec o o he objec s, e.g., he can is on he small able. Fo Ac ion class, symbolic asks such as se e a e de ined in le el- h ee (Task), which a e composed o sho - e m sequences o simple ac ions such as mo e, pick up, mo eholding, place, ha a e de ined in le el- wo (Sub-Task). The s anda dized concep s o ROA: ask and ROA:sub asks a e used o desc ibe hese wo le els. Le el-one includes he new concep o a omic unc ion o ep esen p ocesses o mo ion, manipula ion, and pe cep ion, such as ask planne s, mo ion planne s, o pe cep ual algo i hms. Mo eo e , i includes p imi i e ac ions, p econdi ions, and pos condi ions ela ed o manipula ion beha io s. Al hough ROA p o ides he concep o obo beha io (Olszewska e al.,2017), i does no ully co e he manipula ion planning pe spec i e ha we need. On he one hand, o senso s, he co esponding sui able algo i hms (depending he ype o senso ) should be a ailable o ex ac he ea u es o he en i onmen (see Sec ion 3.4.4). On he o he hand, o planning (e.g., ask and mo ion), he co esponding algo i hms should be a ailable o plan acco ding o he p oblem. So, he new concep o A omic Func ion is in oduced o de ine he algo i hms ela ed o planning in e ms Page 49 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning o mo ion and ask, and he pe cep ual algo i hms. Knowledge Rep esen a ion o Pe cep ion To pe cei e a obo en i onmen , di e en senso s a e usually used. Senso s p o ide da a abou he en i onmen in he o m o signals (one dimension) o images (mul i-dimension), and o ob ain he use ul ea u es om he pe cei ed da a he sui able algo i hms ha e o be applied, o ins ance o de ec an objec pose some pose es ima ion algo i hms based on image ea u es can be applied, o al e na i ely algo i hms based on ags iden i ica ion can be used. In Figu e 3.14, senso s a e de ined in senso class as measu ing de ices (le el- wo). The senso s, which can be a ached wi h he obo o ixed in he en i onmen , may con ain di e en pa s, which a e ep esen ed as senso componen s (le el-one), like he ags, an enna, and eade ha con ain a RFID senso . Se e al senso s may also be g ouped as a de ice g oup (le el- h ee) o a gi en applica ion o domain, like he g ouping o a RFID senso and a 2D came a o objec localiza ion. This g ouping, o ins ance, allows he obo o unde s and ha i has wo ypes o senso s o loca e objec s, as well as hei di e ences, e.g., ype o da a ex ac ed om each senso , algo i hms o be used on he da a p o ided, o he bes en i onmen al condi ions o hei use. The senso g ouping concep makes he obo awa e o i s a ailable equi alen sensing s a egies and allow he selec ion o he p ope senso o use in each case acco ding o he si ua ion. PMK p o ides he ela ion be ween he Fea u e, Senso , and Ac ion classes ha allows o ex ac he in o ma ion om senso s. Fo ins ance, as shown in Figu e 3.15, he concep s o Came a and Algo i hm, which a e inhe i ed o m Measu ing De ice and A omic Func ion me a-concep s, espec i ely, a e modeled and ela ed o he IOC-Lab on ology schema laye o e ie e he equi ed algo i hm o ex ac he image ea u es. The in e ence mechanism equi ed o he sensing p ocedu e is de ailed in Sec ion 3.4.5. The main ad an age o his way o ep esen a ion is ha i is wo kable o any ype o senso and any ype o da a p ocessing algo i hm (like pose es ima ion om ag de ec ion in 2D images implemen ed in he a – ack–al a lib a y (h p://wiki. os.o g/a _ ack_al a ) and used in he case s udy shown below). 3.4.5 Case S udy In his sec ion, se ing ask is p oposed o se e a be e age o a cus ome . The YuMi obo (bi- manual obo ) is used o p epa e he be e ages. I o e comes some manipula ion cons ain s. To ackle he ask, a se o di e en easoning p edica es ega ding ask and mo ion planning, pe cep ion, a e p oposed. The desc ip ion o he ask, i s cons ain s, and he implemen a ion ools a e desc ibed below. Mo eo e , he easoning mechanism o acili a e he planning p ocess is in oduced. The use o hose p edica es in he p oposed ask is explained in de ail. The e alua ion o he case s udy and he sys em lexibili y a e inally discussed. Page 50 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion Task Desc ip ion Conside a manipula ion p oblem including he bi-manual YuMi obo and a se o objec s as depic ed in Figu e 3.12. The ask is o se e he wineglass on he se ingTable. Ini ially he wineglass is loca ed inside he box. Due o eachabili y limi a ions, bo h a ms ha e o collabo a e wi h each o he o sol e he ask. The esponsibili y o YuMi igh a m is o pick up he wineglass and place i on he smallTable and hen, using he le a m, pick up he wineglass up and place i on he se ingTable. The challenge is ha he placemen egion o he wineglass on he smallTable is al eady occupied by he can. Any planne o be used o sol e his ask needs a ich seman ic desc ip ion o he scene, able o answe ques ions such as wha a e he senso s he obo has?, wha a e hese senso s de ec ing?,wha is he associa ed algo i hm o ex ac he objec ea u es?, wha is he spa ial si ua ion o he objec s?,wha is he a ailable egions o place he objec ?, o how can he obs acle be emo ed? and some o he ques ions highligh ed in Figu e 3.16. PMK can p o ide answe s o hese kind o ques ions. The pe cep ion module consis s o wo 2D came as and ags o all he objec s (see Figu e 3.12). One came a is ixed on he op o he main able o sense he main able en i ies, and he o he is a ached o he YuMi le a m o pe cei e he se ing able. The ags a e used o iden i y he wo ld en i ies and seman ically link hem o he he p ope ies o each objec . Speci ically, he pu pose o he pe cep ion module is o de ec he posi ion o he objec s and hei IDs and asse hem on he on ology o build he on ology schema and he on ology ins ance laye s o he IOC-Lab en i onmen (as shown in Figu e 3.11). An exp essi e in e ence p ocess helps o iden i y he hidden knowledge and inc ease he obo s capabili ies. The PMK amewo k uses an in e ence mechanism ha consis s o P olog p edica es. These p edica es a e used o que y o e he on ology o ob ain he knowledge ha he obo equi es o manipula e he objec s in he en i onmen . The in e ence mechanism o manipula ion planning domain includes he easoning p ocess ela ed o sensing, ask planning, and mo ion planning. The ela ed gene ic p edica es a e explained in he ollowing subsec ions. Implemen a ion PMK can be in eg a ed wi h any ask and mo ion planne such as (Akba i e al.,2016a) o compu e he sequence o ac ions o sol e a manipula ion ask. The planne may ask he on ology ques ions abou how o pe o m he ac ions, wha a e he objec s’ poses, o which a e he in e ac ion pa ame e s o he objec s? The PMK handles he eques s o he planne and answe s by e ie ing in o ma ion, upda ing/dele ing o easoning o e i . The eques -answe ela ion is done using he se ice-clien communica ion o ROS (Robo Ope a ing Sys em, www. os.o g). The PMK is designed using on ology web language (OWL) wi h P o égé on ology edi o (h p: //p o ege.s an o d.edu/). On ology ins ances can be asse ed using in o ma ion p ocessed om low-le el senso y da a. The C++ lib a y a _ ack_al a (h p://wiki. os.o g/a _ ack_al a ) has been used o de ec he objec pose and ID. These da a a e asse ed in he PMK o ex ac a seman ic desc ip ion o he objec . The 2D came as ha e been used as a measu ing de ice Page 51 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.16: The desc ip ion o he cons ains o he mo i a ion example ela ed o senso s, objec geome y, in e ac ion lea ning, and ac ion easibili y. wi h wo ROS nodes called FixedCam and A achedCam, o he ixed and a ached came as, espec i ely. All he ans o ma ions o he objec s and came a a e calcula ed wi h espec o he wo ld ame loca ed a he YuMi base. Que ies o e he PMK a e based on SWI-P olog and i s Seman ic Web lib a y which se es o loading and accessing on ologies ep esen ed in he OWL using P olog p edica es. PMK On ology Rep esen a ion In he me aon ology laye , egion is a concep linked o a i ac and quan i y, as shown in Figu e 3.17 whe e he ela ion be ween he on ological laye s ha desc ibe he IOC-Lab unde he me aon ology concep s is illus a ed. In he on ology schema laye , he IOC-Lab egion has a i ac s such as box, able, wineglass, can, cup and quan i ies such as colo and dimension. In he on ology ins ance laye , he ins ance box01 o he subclass box ep esen s he s o age a ea ha con ains he ins ance wineglass01 and cup01 o he subclasses wineglass and cup, espec i ely. These ins ances ha e pe cep ual p ope ies such as pose and agID, ha a e asse ed wi h he ollowing P olog p edica e: d _asse (ins ance, egis e Name:objec p ope ies, asse ed alue), Page 52 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion Figu e 3.17: Taxonomy ep esen a ion o he IOC-Lab domain in he PMK amewo k. The concep s o egion, a i ac , and quan i y a e inhe i ed om Wo kspace, Wsobjec , and Fea u e classes, espec i ely. The Quan i y desc ibes he ea u es o each ins ance as shown in he p ope ies pa in he ins an ia ed a i ac s ( he o ien a ion in o ma ion o he objec s poses is no included in o de o simpli y he igu e). and linked o he ixed p ope ies s o ed on he on ology, so as o ha e a comple e knowledge abou he objec s. Then, once he he obo igu es ou he objec ID, all he ea u es can be ex ac ed. In o de o know whe e is he easoning done, who is doing he asse ion, and how he asse ion is done, see Fig. 3.19. Reasoning P ocess on Pe cep ion The in e ence p ocess ela ed o pe cep ion knowledge is basically he easoning abou ea u e ex ac ion algo i hms o pe cep ion, such as FixedCam o A achedCam nodes used o ex ac he poses and IDs om images. The in e ence p ocess ela ed o he pe cep ion knowledge o PMK basically depends on he ela ion be ween h ee classes (Fea u e, Senso , and Ac ion). As shown in Figu e 3.18, he de ice is a concep ha has quan i y and a omic unc ion. The quan i y con ains he cons ain s and pe cep ual da a. The cons ain s desc ibe he senso limi a ions, such as he ixed came a measu ing only he mainTable wi h ce ain minimum and maximum anges. The pe cep ual da a desc ibes he ype o da a and i s p ope ies, e.g., pose and IDs a e he p ope ies ha a e ex ac ed om an image. These da a a e di ided in o wo main pa s Fea u e o In e es and Obse able P ope y (names a e inspi ed om he Seman ic Senso Ne wo k On olgy, h ps://www.w3.o g/TR/ ocab-ssn/). The o me desc ibes he ype o da a ha a senso senses, such as came a senses images. The la e is an obse able cha ac e is ic (p ope y) o a Fea u e o In e es , such as colo is one o he image p ope ies. As an example, he RunFixedNode p edica e, shown below using DL, is applied o eason abou he loca ion and ID o he objec s by being associa ed wi h he FixedCam Node a omic unc ion (algo i hm). Page 53 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Figu e 3.22: The sequence o snapsho s o he execu ion o he se e a cup o co ee command. Video URL-link "h ps://si .upc.edu/p ojec s/kau ham/ ideos/PMK- inal.mp4". 3.4.7 Discussion Discussion abou he Resul s The mo i a ion example highligh s he signi icance o using PMK o make he obo unde s and he cu en scena io and ac acco ding o he ac ual manipula ion cons ain s. The manipula ion cons ain s ela ed o ask execu ion can be lis ed as: 1. Manipula ion cons ain s such as: F om whe e he objec can be in e ac ed?, Wha a e he in e ac ion pa ame e s? 2. Geome ic cons ain s such as: Wha is he spa ial obo eachabili y? Whe e can he objec s be placed? 3. Ac ion cons ain s such as: F om whe e can he ac ions be applied? 4. Pe cep ion easoning such as: Wha is he senso a ached o he obo ? How does i wo k? Wha a e i s cons ain s, such as he senso ange measu emen ?. PMK p o ides he answe s needed by combined ask-mo ion planne s and by physics-based mo ion planne s. Some o he amewo ks p oposed a knowledge-based p ocessing and easoning such as (Teno h and Bee z,2009;Lim e al.,2011). Al hough he e a e some simila i ies wi h hem in e ms o spa ial easoning o easoning abou he objec ea u es which could be common in he wo king domains, he easoning p ocess ela ed o planning (including geome ic easoning) and pe cep ion easoning a e newly p oposed in PMK. Fo ins ance, hese amewo ks p oposed Page 60 o 151 3.4. A Knowledge P ocessing F amewo k o Au . Rob. Pe cep ion and Manipula ion some spa ial ela ions such as on, inside, igh , le o he obo o spa ially ela ed be ween he en i onmen en i ies. The ex ended spa ial easoning ha includes some ex a p edica e such as spa ial obo eachabili y and a m selec ion o bi-manual obo s ha e been in oduced. In he case s udy, he impo ance o hese p edica es appea , o ins ance, when he obo needs o execu e he pick up ac ion o wineglass. Fi s , by asking abou which a m is eachable, hen by asking abou he eachable placemen egion o place he wineglass on he se ingTable wi h he le a m. Mo eo e , when using a physics-based mo ion planning, o apply he push ac ion, a que y is posed o know how o in e ac wi h he obs acle can, which needs, o ins ance, he in e ac ion pa ame e s like he ic ion coe icien . Mo eo e , o pe cep ion, PMK p oposes he easoning ela ed o he pe cep ual ea u es o he objec s in he en i onmen , like o he s, bu also some easoning ela ed o he sui able algo i hms ha he senso can un o ex ac he ea u es, he senso s ha a e associa ed wi h he obo , and he senso ea u es and i s limi a ions. The pe cep ion easoning p ocess makes he obo sma e and mo e lexible. This lexibili y can be use ul o cope wi h he ailu e o a senso , by p o iding an al e na i e one, i.e., PMK has he lexibili y o deal wi h mul i-model senso y sys ems. Discussion abou he Sys em The execu ion o manipula ion asks wi h knowledge-based planning app oaches no explici ly p epa ed o TAMP, like (Teno h and Bee z,2009), can be a challenge because his would equi e, on he one hand, om he knowledge pe spec i e, o p o ide all he componen s in a way ha hey ma ch wi h hei planning sys em. On he o he hand, om he planning pe spec i e, hey would equi e he de ini ion o he ecipes (s a egies) o execu ing he asks (sequence o ac ions), including all possible s a egies o execu ion and he way o swi ch be ween hem when equi ed, which can be a e y expensi e p ocess, especially o asks ha need long sequences o ac ions, such as hose in ol ing manipula ion in clu e ed en i onmen s. Some o he planning app oaches ely on PDDL and on planning s a egies bes i o cope wi h di icul ask planning challenges, like hose ound in he manipula ion domains, al hough he use o PDDL implies a closed-wo ld assump ion, which p ecludes hei use in mo e dynamic en i onmen s ha could equi e pe cep ion and knowledge-based geome ic easoning. The main ole o PMK is o acili a e he planning p ocess o TAMP by p o iding he necessa y planning componen s, such as geome ic easoning, dynamic in e ac ions, manipula ion cons ain s, and ac ion cons ain s. Mo eo e , i includes he pe cep ion knowledge and easoning abou he senso s ha he obo has, he co esponding algo i hms o ex ac he ea u es, and he senso s limi a ions. All hese componen s a e equi ed o au oma ically execu e complex asks, like hose p esen ed in (Lag i oul e al.,2018;Quispe e al.,2018), and o make he sys em lexible enough o adap o di e en si ua ions equi ing di e en manipula ion ac ions, as demons a ed wi h he wo mo i a ion examples. Page 61 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning 3.4.8 Summa y This s udy p oposed he o maliza ion and implemen a ion o he s anda dized on ology amewo k PMK o ex end he capabili ies o au onomous obo s ela ed o manipula ion asks ha equi e ask and mo ion planning (TAMP). In e ms o modeling, he aim has been o con ibu e wi h a uni ied amewo k based on s anda diza ion ha p o ides common ocabula ies in o de o ha e he lexibili y o inco po a e PMK wi h o he on ologies, o a oid con lic in he meanings o concep s. Mo eo e , in e ms o easoning p ocess, some impo an componen s o ask, mo ion, o combined ask-mo ion planning a e p oposed, such as easoning o pe cep ion, easoning abou he objec ea u es, easoning abou he en i onmen , geome ic easoning, and easoning o planning. To illus a e he p oposal, wo examples had been in oduced o show he PMK amewo k abili ies o que y knowledge abou he geome ic easoning and obo capabili ies o sol e TAMP p oblems. Mos o he equi emen s o TAMP, discussed in his wo k, a e me by he p oposed amewo k. The knowledge is o ganized in a way o acili a e he planning p ocess, so ha he obo can easily access he concep s i needs o i s asks, i.e., PMK enables he obo o comple e a manipula ion ask au onomously in spi e o hidden o pa ial da a. To subs an ia e he pe cei ed in o ma ion, knowledge ela ed o pe cep ion has been conside ed, which allows analyzing he si ua ion o he en i onmen . Since TAMP que ies he equi emen s o PMK, i au oma ically adap s o di e en scena ios. 3.4.9 Enhancemen The PMK amewo k dose no cope wi h he na iga ion p oblems which equi es he ex ension wi h mo e pe cep ion measu ing de ices, such as RFID senso s and 3D came as, o be applied o mo e complex manipula ion asks in ol ing assemblies. Also, an ex ension is planned o include concep s and easoning ela ed o ailu es, in o de o make PMK use ul in si ua ions whe e he ask planne may no be able o ind a easible sequence o ac ions o pe o m a gi en ask, o may need o eco e om an e o . This enhancemen has been done in Chap e i e. 3.5 Compa ison o PMK agains o he on ology-based app oaches o obo au onomy A compa ison be ween PMK and some p ojec s ha use on ologies o suppo obo au onomy is p oposed. The sys ema ic sea ch o p ojec s ha ul ill a se o inclusion c i e ia has been done, and he compa ison hem wi h each o he wi h espec o he scope o hei on ology, wha ypes o cogni i e capabili ies a e suppo ed by he use o on ologies, and which is hei applica ion domain. Page 62 o 151 3.5. Compa ison o PMK agains o he on ology-based app oaches o obo au onomy 3.5.1 Inclusion c i e ia Some amewo ks o p ojec s ha e been selec ed which a e conside ed o be objec o he analysis pe o med in his wo k. Howe e , among hem, he ocus is only on he discussion and he compa ison on he mos in luen ial app oaches. Hence, his wo k p o ides a lis o inclusion c i e ia o e ine he lis o su eyed p ojec s. As p esen ed in he Sec ion 4.4 in (Oli a es-Ala cos e al.,2019), au ho s b ie ly in oduce he excluded app oaches and p o ide some jus i ica ion o ou decision. P ojec s o amewo ks a e only conside ed in he scope o his wo k i hey sa is y all o he ollowing c i e ia: 1. On ology scope: The p ojec uses an on ology ha de ines one o he e ms shown in Table 3.2, such as Capabili y, Skill, Plan and Func ion, e c. ha a e iden i ied as pa icula ly ele an o au onomous obo ics ( hose e ms a e p esen ed in Sec. 3.1 in (Oli a es- Ala cos e al.,2019)); 2. Reasoning scope: I uses on ologies o suppo obo s mani es ing a leas one o he cogni i e capabili ies (discussed in Sec ion 3.2 in (Oli a es-Ala cos e al.,2019)); 3. T anspa ency: I is anspa en . Meaning ha some ma e ial (e.g., websi es, publica ions) is openly a ailable ha desc ibes he o e all goal o he p ojec , wha cogni i e capabili ies a e conside ed, and how and wha on ologies a e used; 4. Cu a ion: I is main ained. Meaning ha ecen de elopmen s o u u e plans a e e iden o a leas possible; and 5. Accessibili y: The e exis s –a leas a p o o ypical –so wa e ha is accessible, and ha demons a es how on ologies a e used o suppo a cogni i e capabili y. 3.5.2 Compa ison o abs ac concep and domain ca ego y The e a e six amewo ks/p ojec s ha sa is y he c i e ia om di e en obo ics applica ions a eas such as se ice and indus e ial, e c., and The PMK (Diab e al.,2019) is one o hem. Beside PMK, o he s like KnowRob (Teno h and Bee z,2009,2017), ROSETTA (S enma k e al.,2018), IEEE S anda d On ologies o Robo ics and Au oma ion (IEEE-SA,2015), ORO (Lemaignan e al.,2010), CARESSES (B uno e al.,2017) a e discussed in his e iew. Fo each o hem, hei unde lying p inciples and ounda ions a e discussed, as well as wha applica ion domain he sys em was designed o . Also he desc ip ion o how he amewo ks e ol ed o e ime, and wha impac hey ha e had so a . As men ioned in Table 3.2, abs ac concep s such as objec s, en i onmen map, ac ion, capabili y, ha dwa e, and so wa e a e p oposed in PMK and some o he o he amewo ks. Uniquely, PMK p oposed some e ms like Func ion and In e ac ion which a e no p oposed by o he amewo ks. As a esul o i , PMK is conside ed as a qui e comp ehensi e and lexible enough o model se e al obo ics domains h ough i . Page 63 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning Te m KnowRob 1/2 ROSETTA ORO CARESSES OROSU PMK Objec Yes/Yes Yes Yes No Yes Yes En i onmen map Yes/Yes No No No Yes Yes A o dance No/Yes No Yes No Yes No Ac ion Yes/Yes No Yes Yes Yes Yes Task No/Yes Yes Yes No No Yes Ac i i y No/No No No Yes No No Beha io No/No No No No No No Func ion No/No No No No No Yes Plan No/Yes No Yes No No Yes Me hod No/Yes No No No No Yes Capabili y Yes/Yes Yes No No No Yes Skill No/No Yes No No No Yes Ha dwa e Yes/Yes Yes Yes No Yes Yes So wa e Yes/Yes Yes Yes No Yes Yes In e ac ion No/No No No No No Yes Communica ion Yes/No No No No No No Table 3.2: Lis o ele an e ms o he au onomous obo ics domain, and hei co e age in he di e en chosen wo ks. Yes and No s a e o when he e m is o no co e ed by he on ology o he speci ic amewo k. No e ha in he cases when he e m is needed and aken om he uppe on ology used wi hin he amewo k, and/o when he knowledge is cap u ed using a simila e m, i is conside ed ha he e m is co e ed. I he uppe on ology con ains he e m bu i is no used, we conside ha he e m is no included. As shown in Table 3.3, PMK co e s se e al ca ego ies in obo ics domain. Decision making and choice ca ego y is co e ed because o he desc ip ion o a obo sys em which enhances he execu ion o plans wi h he suppo o he PMK on ology. Based on he belie s abou he wo kspace ( eachabili y o objec s, easible ac ions o execu e, e c.), he sys em makes decisions abou he dis ibu ion o ac ions among di e en obo ic a ms, and also abou ac ion’s pa ame e s, sligh ly modi ying he o iginal plan. The Pe cep ion and si ua ion assessmen ca ego y is co e ed because o he in eg a ion o a agged-based ision module wi hin PMK. In his module, he ags a e used o de ec he poses and IDs o wo ld en i ies and asse ing hem o he PMK o build he domain knowledge. Then, a easoning mechanism is used o p o ide he easoning p edica es ela ed o pe cep ion, objec ea u es, geome ic easoning, and si ua ion assessmen . Pa icula ly, o si ua ion assessmen , an e alua ion-based analysis is p oposed which gene a es ela ions be ween he agen and he objec s in he en i onmen based on he pe cep ion ou comes, being, hese ela ions used la e o acili a e he planning p ocess. The P oblem sol ing and planning ca ego y is co e ed because o he abili y o PMK o se e as ool o any planne o eason abou ask and mo ion planning in e ence equi emen s, such as obo capabili ies, ac ion cons ain s, ac ion easibili y, and manipula ion beha io s. The Reasoning and belie main enance ca ego y is co e ed because o he abili y o PMK o gene a e seman ic maps o he obo ’s wo kspace enhancing i s belie main enance. By means o compu e ision me hods, he obo de ec s objec s and i s p ope ies (e.g. poses) and, using he on ology, Page 64 o 151 3.6. Summa y o he chap e Cogni i e Capabili y KnowRob ROSETTA ORO CARESSES OROSU PMK Recogni ion and ca ego iza ion (Beßle e al.,2019)–(Lemaignan e al.,2010) (Menica i e al.,2017)– – Decision making and choice – – – (B uno e al.,2019)–(Diab e al.,2019) Pe cep ion and si ua ion assessmen (Bee z e al.,2015a)–(Sisbo e al.,2011)– – (Diab e al.,2019) P edic ion and moni o ing (Bee z e al.,2012)– – – – – P oblem sol ing and planning (Beßle e al.,2018c)– – – – (Diab e al.,2019) Reasoning and belie main enance (Beßle e al.,2018c)–(Wa nie e al.,2012) (B uno e al.,2019)–(Diab e al.,2019) Execu ion and ac ion (Teno h e al.,2014) (S enma k e al.,2015)–(Sgo bissa e al.,2018) (Gonçal es and To es,2015) (Diab e al.,2019) In e ac ion and communica ion (Yazdani e al.,2018)–(Lemaignan e al.,2011) (B uno e al.,2018)–(Diab e al.,2017) Remembe ing, e lec ion and lea ning (Bee z e al.,2015b) (Topp and Malec,2018)– – – – Table 3.3: Lis o cogni i e capabili ies o he au onomous obo ics domain and hei co e age in he di e en chosen amewo ks/on ologies. I is possible o ind he e e ence o he a icles in which he di e en easoning capabili ies a e add essed using he on ologies. i s o es a symbolic ep esen a ion o he wo kspace. A easoning p ocess o e hose symbolic belie s allows, o ins ance, o make assump ions abou abs ac spa ial ela ions (e.g. cup on he able). The Execu ion and Ac ion ca ego y is co e ed because o he abili y o PMK o link he ac ion ep esen a ion in wo o ms. The o me is he ep esen a ion o he p econdi ions and e ec s o each ac ion in an on ological o m, hen he planne can que y o e he PMK o e ie e he equi ed in o ma ion. The la e is o link he PDDL ep esen a ion o m wi h on ology. Fo each ac ion, he planne can que y o e he PMK o e ie e he equi ed in o ma ion om a PDDL ile. The In e ac ion and Communica ion ca ego y is co e ed because o he abili y o physics-based mo ion planne o que y o e he PMK o eason abou he in e ac ion pa ame e s wi h he physical objec s in a ce ain en i onmen . 3.5.3 Enhancemen This s udy concludes ha PMK ge s an impac on modeling and easoning le els. Howe e , i needs o be es ed in mo e planning app oaches like knowledge-enabled app oaches such as (Beßle e al.,2018c). Mo eo e , al hough PMK is an open-sou ce lib a y, i needs o be well-documen ed. 3.6 Summa y o he chap e In his chap e , a on ological amewo k called PMK is p oposed o p o ide use ul knowledge o guide and acili a e he planning p ocess wi hin a classical-based manipula ion planning amewo k. This planning amewo k acili a es he combina ion o ask and mo ion planning (TAMP) app oaches which includes Fas Fo wa d (FF), a classical symbolic planning app oach o compu e he sequence o ac ions o be done in a ce ain ask, and physics-based mo ion planning which deals wi h mo ions and possible in e ac ions wi h he objec s. The ools p oposed o p o ide use ul knowledge o he planning p ocess called Pe cep ion and Manipula ion Knowledge (PMK) is p oposed. I p o ides, on he one hand, a s anda dized o maliza ion Page 65 o 151 CHAPTER 3. Knowledge Guidance o Task and Mo ion Planning unde se e al ounda ions such as he Sugges ed Uppe Me ged On ology (SUMO) ,and he Co e On ology o Robo ics and Au oma ion (CORA) in o de o acili a e he sha eabili y and eusabili y when he in e ac ion be ween humans and/o obo s is done. On he o he hand, he in e ence mechanism is p oposed o eason abou TAMP in e ence equi emen s, such as obo capabili ies, ac ion cons ain s, ac ion easibili y, and manipula ion beha io s. Mo eo e , PMK allows o b eak he closed wo ld assump ion o classical-based manipula ion planning app oaches. This p oposal has been es ed o se ing ask in a able- op manipula ion p oblem. Page 66 o 151 Chap e 4 An On ology-based App oach o Failu e In e p e a ion and Reco e y in Planning and Execu ion This chap e in oduces knowledge-based ailu e in e p e a ion and eco e y ools o able- op manipula ion p oblems, especially in he assembly domain. This ool p o ides a ep esen a ion way unde SUMO and DUL ounda ions. Mo eo e , a easoning mechanism o eason on geome ic componen s is p oposed such as he easibili y o ac ions and mo ion cons ain s in a logic-based planning sys em. This easoning mechanism equi es he in eg a ion o low-le el geome ic planning modules ( ha include mo ion planning, collision check, in e se kinema ics, and objec placemen ) o eed back he planne on whe he he p econdi ions o ac ions a e me . The e o e, in his chap e , a geome ic and ailu e in e p e a ion and eco e y on ologies a e p oposed, linked wi h a low-le el geome ic module, o p o ide a he e ogeneous way o easoning ha helps any planning sys em. Some mo i a ing examples ha e been in oduced o illus a e he p oposal, which has been es ed wi h a logic-based planne . 4.1 P oblem s a emen and p oposed solu ion Au onomous mobile obo manipula o s may no show a obus pe o mance when placed in en i onmen s ha a e no igh ly con olled. An impo an cause o his is ha ailu e handling o en consis s o sc ip ed esponses o o eseen complica ions, which lea es he obo ulne able o new si ua ions and ill-equipped o eason abou ailu e and eco e y s a egies. Ins ead o lib a ies o ha d-coded eac ions ha a e expensi e o de elop and main ain, mo e sophis ica ed easoning mechanisms a e needed o handle ailu e. This equi es an on ological cha ac e iza ion o wha ailu e is, wha concep s a e use ul o o mula e causal explana ions o 67 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion Figu e 4.1: FailRecOn - The combina ion o he planning sys em o assembly asks wi h he p oposed geome ic easoning and ailu e in e p e a ion/ eco e y capabili ies in o ange. ailu e, and in eg a ion wi h knowledge o a ailable esou ces including he capabili ies o he obo as well as hose o o he po en ial coope a i e agen s in he en i onmen , e.g. a human use . We p opose he FailRecOn amewo k as a s ep in his di ec ion. We ha e in eg a ed an on ology o ailu e in e p e a ion and eco e y wi h: 1) a logic-based planning amewo k in assembly domain, 2) a heu is ic-based ask and mo ion planning amewo k, such ha a obo can deal wi h unce ain y, eco e om ailu es, and deal wi h human- obo in e ac ions. 4.2 FailRecOn F amewo k In his sec ion, an o e iew o he amewo k, he desc ip ion o he p oposed modules in he low-le el and he knowledge-le el, and how he da a is managed a e explained in de ail. 4.2.1 O e iew The p oposed amewo k – FailRecOn – is he con inua ion o KTAMP amewo k p esen ed in chap e 3, and i is composed o he same h ee main laye s (planning and execu ion laye , knowledge laye and assis an laye ) al hough now he knowledge laye has been ex ended, as shown in Fig. 4.1. Page 68 o 151 4.2. FailRecOn F amewo k The knowledge laye con ains a se o knowledge o guide he planning and execu ion laye : 1. Awa eness module: ha con ains a) PMK knowledge as desc ibed in de ail in chap e 3; b) Geome ic knowledge o p o ide he geome ic easoning esponsible o checking he easibili y o he ac ions. 2. Reco e y module, ha p o ides knowledge o in e p e he ailu es and p oposes eco e y s a egies like: 1) asking a human o assis ance o unsol able asks by he obo , 2) guiding he obo o au onomously eco e i sel , o ins ance by calling a sensing module o igu e ou he cu en scene o he wo ld o 3) keep epea ing he same ac ion wi h ano he pa ame e (e.g., epea a g asping ac ion wi h a di e en angle). The nex subsec ions a e o ganized o acili a e cap u ing he con ibu ions o his chap e . Fi s , Sec. 4.2.2 desc ibes he in eg a ion o he p oposed geome ic on ology o gi e access o he geome ic easoning module. The main ad an age o his is ha , ins ead o calling he module manually om he ask planning module, as done in he p e ious chap e , he obo can que y o e he knowledge o e ie e he sequence o p ocesses equi ed o execu e such ac ions in an au oma ic way. Howe e , his equi es he implemen a ion o a mechanism ha has he abili y o igu e ou he ailu es ha may occu . Fo his pu pose, ailu e in e p e a ion and eco e y on ologies a e p oposed in Sec. 4.2.3 whe e he ailu e concep s, modeling and causes a e desc ibed. 4.2.2 Geome ic knowledge Concep s desc ibing geome ic que ies An on ological module is de ined o co e geome ic no ions used in obo ics such as collision checking, placemen easibili y, e c. This on ology also desc ibes concep s like geome ic que ying, que y s a us, and s a us diagnosis, and con ains e ms o pa icula que ies such as IK (in e se kinema ics) o a speci ic obo . Some impo an e ms in he on ology a e b ie ly de ailed 1: •Geome ic que ying is de ined as an E en in which some spa ial easone – e.g. a collision checke – pa icipa es, and which is classi ied by/execu es a Geome icReasoningTask. •Geome icReasoningTask is de ined as a classi ica ion o he Geome ic que ying. I is ep esen ed as: Geome icQue ying (∃isClassi iedBy.Geome icReasoningTask)u(∃hasPa icipan .Spa ialReasone ) u(=1hasS a us.Que yS a us). We also say ha Geome icReasoningTask ∀ isExecu edBy.Geome icQue ying. 1(Desc ip ion logic is used, see Appendix B o de ails.) Page 69 o 151 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion Reco e y s a egies A eco e y s a egy is a me hod a obo can apply o epai o econs uc a plan whose execu ion esul ed in a ailu e. Because eplanning is a ime-consuming ope a ion, ins ead, epai ing he plan could be an op ion. Some eco e y s a egies ha e been p oposed such as: 1. Repea las ac ion: A obo could ei he employ a “ epea las ac ion” s a egy o ins ead ha e o add ess he unde lying ailu e cause. We can axioma ically encode ha a ” epea las ac ion" is no app op ia e when he ailu e has a sus ained cause. 2. Remo e ailu e cause: Depending on he na u e o he unde lying cause o a sus ained ailu e, we can use he on ology o o mula e new subgoals as pa o a Remo eFailu eCause s a egy by checking which o he obo -known si ua ions ge classi ied as Un ealizedP econdi ion, and hence he un ealized p econdi ions become goals o new planning que ies. 3. Igno e ailu e: A ailu e is no necessa ily a p oblem o con inuing a plan. I he expec ed ou come is an objec loca ed a X, bu he objec is a Y, his migh no be a p oblem i he objec is no needed again and no in he way o o he ac ions. Geome ic easoning beyond he scope o he on ology decides whe he an objec is “no in he way”. We say an Igno eFailu e s a egy may be app op ia e only when he si ua ion ac ually mani es ing p e en s no o he expec ed si ua ions. In o de o check whe he he p e en s ela ions hold o no – so as o asce ain whe he impedes holds o no – we would ha e o de e o o he easoning modules. Compe ency ques ions The p oposal is o exploi he on ology o disco e in o ma ion help ul o he obo o e alua e and, when needed, o e come a ailu e. When a ailu e is de ec ed, he on ology should be able o p o ide answe s o he ollowing ailu e ques ions (FQ) we imagine he obo aises: 1. Why was an Ac ion classi ied as ailu e? 2. Is he ailu e causing a p oblem o subsequen ac i i ies? 3. I he ailu e is causing a p oblem (and so mus be add essed), does i ma e why i happened and i so wha is he causal explana ion o he ailu e? 4. I he ailu e is causing a p oblem, wha a e app op ia e eco e y s a egies? 5. Assuming a eco e y s a egy is pu sued, how can success ul eco e y be assessed? Page 76 o 151 4.3. The in eg a ion o he p oposed on ologies wi h a logic-based planning Rega ding FQ1, his on ology o malizes concep s o o ganize he obo ’s in o ma ion abou he ac ion execu ed and he expec ed s. ac ual si ua ion, and iden i y aspec s o he misma ch e.g. ailu e loca ion o unc ional aspec s such as in ol ed esou ces. The obo ’s a ious easoning modules, such as mo ion planning and na iga ion, coope a e in answe ing FQ2. This coope a ion is guided by he on ology in ha i o malizes que ies abou p e en ion ela ions be ween ongoing si ua ions esul ing om he obo ’s pas ac ions and u u e si ua ions co esponding o goals being achie ed, que ies o be handled by mechanisms app op ia e o he na u e o hose si ua ions; e.g. na iga ion can check whe he a pa icula objec placemen p e en s a base mo emen . FQ3 e eals how he on ology con e s in o ma ion abou p e en ion ela ions (in his case, be ween si ua ions ha ha e al eady happened and ongoing o si ua ions ha a e pos condi ions o he ailed ac ion) in o selec ion c i e ia o eco e y s a egies. In pa icula , i allows dis inguishing be ween causes ha mus be add essed, and hence equi e de ining new subgoals, om causes ha can be igno ed. FQ4 is answe ed by he on ology by he c i e ia i de ines o eco e y s a egy applicabili y. These c i e ia a e necessa ily incomple e, in ha we o en can say when a eco e y s a egy is no app op ia e, bu his in i sel is no p oo ha he eco e y s a egy ac ually is app op ia e. Ul ima ely, i is he obo ’s eco e y a emp ha is he inal judge. Howe e , easoning guided by he on ology can il e ou in easible candida e s a egies. Finally, o FQ5, he eco e y s a egies de ined in he on ology also desc ibe wha coun s as a success, because “success” is dependen on he s a egy and he o iginal ans o ma ion expec a ion. Fo example, a ailed goal is simply abandoned by an Igno eFailu e, whe eas new subgoals a e de ined o Remo eFailu eCause. 4.3 The in eg a ion o he p oposed on ologies wi h a logic-based planning In assembly applica ions in able- op manipula ion p oblems, assembly ecipes can be used and elegan ly be ep esen ed in desc ip ion logic heo ies (see Sec. 2.7 and Appendix B). Wi h such a ecipe, he obo can igu e ou he nex assembly s ep h ough logical in e ence. Howe e , be o e pe o ming an ac ion, he obo needs, on one hand, o ensu e a ious spa ial cons ain s a e me , such as ha he pa s o be pu oge he a e eachable, non occluded, e c. On he o he hand, i ailu es occu ed while planning o /and du ing he execu ion phase, he obo mus be able o ecognize hem, hei sou ces and how o eco e hem. Such in e ences a e e y complica ed o suppo in logic heo ies, bu specialized algo i hms exis ha e icien ly compu e quali a i e spa ial ela ions conside ing he possibili y o ail, such as whe he an objec is eachable. Page 77 o 151 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion Figu e 4.2: P oposed In eg a ion. The b own on ologies p o ide by KnowRob g oup (Beßle e al.,2018a), while he blue and yellow boxes a e p oposed o his in eg a ion. In his chap e , as shown in Fig. 4.1, he FailRecOn amewo k is p oposed, which combines a logic-based planne , desc ibed in Sec. 2.7, wi h geome ic easoning and ailu e in e p e a ion/ eco e y capabili ies, o enable obo s o pe o m hei asks unde spa ial cons ain s. The geome ic easone is in eg a ed in o he logic-based easoning h ough decision p ocedu es a ached o symbols in he on ology. Mo eo e , each assembly s ep is o mally de ined in a wo k low ha s uc u es he s ep, and ha can be au oma ically execu ed in a decla a i e knowledge base. As shown in Fig. 4.2, he in eg a ion o geome ic and eco e y module is p oposed o he geome ic ailu es and he beha io o he obo when encoun e ing he a o emen ioned ailu es. This in eg a ion is use ul o analyze he inconsis encies i hey exis , using he eedback o he geome ic compu able p edica es (a easoning p ocess ha epo s he ailu es) ha a e capable o e alua e hose inconsis encies. The Geome ic easone consis s o wo submodules; he ailu e sol e and success. The o me is used o eason o e he Geome ic analysis on ology abou he he ailu e ypes. Mo eo e , i can p o ide solu ion(s) by calling he co esponding module om he on ology, i i exis s, o p o ide di e en solu ions o sol e he ailu e. The la e is used o epo he easibili y o ac ions. Page 78 o 151 4.3. The in eg a ion o he p oposed on ologies wi h a logic-based planning Figu e 4.3: Mo i a ing example in assembly domain showing cases ha need he planne o use he geome ic, ailu e and eco e y on ologies o in e p e que y and ac ion esul s. Desc ip ion o eachabili y and collision p oblems o he ed objec s ( op and bo om wings). Mo i a ion example We a e mainly in e es ed in assembly manipula ion asks o bi-manual obo s, which o en encoun e complexi y o ailu es in he planning and execu ion phases. Planning phase ailu es ypically e e o ailu es o he planne i sel , bu we will use planning phase ailu es o also e e o si ua ions whe e he planne easons ha some ac ion would be in easible, e.g. because objec s block access o wha he obo should each. A co ec selec ion o g asps and placemen s mus be p oduced in such an e en uali y. Depending on he ype o p oblem, goal o de mus be ca e ully handled especially in he assembly domain; e y la ge sea ch spaces a e possible, equi ing objec s o be mo ed mo e han once o achie ing he goals. Execu ion phase ailu es e e o ha dwa e ailu es ela ed o he sys em de ices e.g. obo o came a needs o be e-calib a ed, o so wa e ailu es ela ed o he capabili ies o e ed by speci ic so wa e componen s, o ailu es in ac ion pe o mance such as an unexpec ed occluding objec , o slippage. We selec ed a oy plane assembly, as shown in Fig. 4.3 a ge ed a 4-yea -old child en o he Page 79 o 151 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion expe imen al e alua ion. The oy plane is made o 21 plas ic pa s ha a e mainly pu oge he using a loose slide in connec ions, and ixed wi h bol s a e wa d. The pa s a e compa ably huge such ha g asping hem is easie . We use a dual a med YuMi obo and u he es he assembly planning in The Kau ham P ojec (Rosell e al.,2014) which is an open-sou ce p ojec o mo ion planning. I p o ides he exibili y o plan mo ions unde geome ic, and physics-based cons ain s. Fo he p oposed expe imen s, we use he RRT-Connec mo ion planne (Ku ne and LaValle,2000), (Gillani e al.,2016), and he in e se kinema ics app oach de eloped by Zaplana (Zaplana e al.,2018). Kau ham u he p o ides a ROS-based in e ace, and he YuMi obo has also a se o exis ing so wa e se ices ha we use o con olling he obo . On ologies a e encoded using he Web On ology Language (OWL) (An oniou and an Ha melen,2004), and designed using he P o égé 2edi o . Assembly Ac i i ies in Clu e ed Wo kspaces Assembly asks o en ha e a ixed ecipe ha , i ollowed co ec ly, would con ol an agen such ha a ailable pa s a e ans o med in o an assembled p oduc . These ecipes can elegan ly be ep esen ed using desc ip ion logic. Bu in e ing he sequence o assembly ac ions is no su icien o obo s because ac ions may no be pe o mable in he cu en si ua ion. This is, o example, he case when he obo canno each an objec because i is occluded. A no ion o space, on he o he hand, is e y complica ed in a logic o malism, bu specialized me hods exis ha e icien ly compu e quali a i e spa ial ela ions such as whe he objec s a e occluding each o he . The p oposed solu ion is depic ed in Fig. 4.2. We build upon an exis ing planne and ex end i wi h a no ion o ac ion, and geome ic easoning capabili ies. Ac ions a e ep esen ed in e ms o he ac ion on ology which also de ines ac ion p e- condi ions. P e-condi ions a e ensu ed by unning he planne o he ac ion en i y. This is used o ensu e ha he obo can each an objec , o else ies o pu away occluding objec s. To his end we in eg a e a geome ic easone wi h he knowledge base. The in e aces o he geome ic easone a e hooked in o he logic-based easoning h ough p ocedu al a achmen s in he knowledge base. Mo e de ails will be desc ibed in Sec. 4.4.1 4.4 Modi ica ions on knowledge ep esen a ion o assembly Some modi ica ions in he on ological ela ion ha e been in oduced o enhance he logic-based planning p esen ed in sec ion 2.7 in o de o smoo hly eason o e he p oposed on ologies. 2(h p://p o ege.s an o d.edu/) Page 80 o 151 4.4. Modi ica ions on knowledge ep esen a ion o assembly To connec wo pa s, hey mus be in he co ec ix u e o he in ended connec ion. I could be he case ha a ix u e blocks a equi ed a o dance. In ha case, he pa should be mo ed in o ano he ix u e ha exposes he equi ed a o dance, such ha he equi ed a o dance is exposed. To ensu e his, we asse ha equi ed a o dances mus be unblocked: we add ano he axiom ha es ic s he assemblesConnec ion ela ion o he ac ion: ∀assemblesConnec ion.(∀usesA o dance.F eeA o dance)(4.1) I asse s ha any a o dance used by he connec ion mus no be blocked. This en o ces he obo o use a ix u e ha exposes he equi ed a o dance. Finally, we de ine pa OccludedBy ≡hasA o dance ◦occludesA o dance−which ela es pa s o pa s occluding hem, and asse ha pa s canno be occluded by o he pa s when he obo in ends o pu hem oge he : ∀assemblesPa .(≤0pa OccludedBy.MechanicalPa )(4.2) This is used o make he obo pu away pa s ha occlude o he pa s ha p o ide equi ed a o dances o his ac ion. 4.4.1 He e ogeneous easoning p ocess The p oposed easoning sys em is he e ogeneous, which means ha di e en easoning esou ces and ep esen a ions a e used in o a cohe en pic u e ha co e s di e en aspec s, as shown in Fig. 4.3 and desc ibed in de ail in Sec. 4.2. In his sec ion, we will desc ibe he wo di e en easoning me hods used: geome ic easoning and knowledge-based easoning. a) Geome ic Reasoning The main ole o geome ic easoning is o e alua e geome ic condi ions o symbolic ac ions. Two main geome ic easoning p ocesses a e p o ided: Reachabili y Reasoning: A obo can ansi o a pose i i has a alid goal con igu a ion. This is in e ed by calling an In e se Kinema ic (IK) module and e alua ing whe he he IK solu ion is collision- ee. The i s ound collision- ee IK solu ion is e u ned, and, i any, he associa ed pose. Failu e may occu i ei he no IK solu ion exis s o i no collision- ee IK solu ion exis s. Spa ial Reasoning: We use his module o ind a placemen o an objec wi hin a gi en egion. Fo he desi ed objec , a pose is sampled ha lies in he su ace egion, and is checked o Page 81 o 151 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion Figu e 4.4: The sequence o assemble he Ba a oy. The scene has been modeled using PMK on ology. collisions wi h o he objec s, and whe he he e is enough space o place he objec . I he sampled pose is easible, i is e u ned. O he wise, ano he sample will be ied. I all a emp ed samples a e in easible, he easone epo s ailu e, which can be due o a collision wi h he objec s, o because he e is no enough space o he objec . b ) Knowledge-based Reasoning In his p ojec , knowledge-based easoning e e s p ima ily o checking whe he an indi idual obeys he es ic ions imposed on he classes o which i is claimed o belong, iden i ying an indi idual based on i s ela ions o o he s, and iden i ying a se o indi iduals linked by ce ain p ope ies (as done when iden i ying which pa s ha e been linked, di ec ly o indi ec ly, ia connec ions). The geome ic easone is in eg a ed h ough compu able geome ic ela ions. The obo can hen eason abou hem by asking ques ions such as “wha a e he occluded pa s equi ed in a connec ion?”: ?−holds( needsA o dance ( Connec ion , A o dance ) ) , holds( hasA o dance (Occluded , A o dance ) ) , holds( pa OccludedBy (Occluded , OccludingPa ) ). Occluded=’ PlaneBo omWing1 ’ , OccludingPa =’ PlaneUppe Body1 ’ . Page 82 o 151 4.4. Modi ica ions on knowledge ep esen a ion o assembly Figu e 4.5: YuMi plane assembly execu ion sequence. The obo can also eason abou wha ac ion p e-condi ions a e no ul illed, and wha i can do o ix his. This is done by c ea ing a planning agenda o he ac ion en i y ha only conside s p e-condi ion axioms o he ac ion: ?−e n i y ( Ac , [an , ac ion , [ ype , ’ Connec ingPa s ’ ] , [ assemblesConnec ion , Connec ion ]]) , agenda_c ea e ( Ac , Agenda ) , agenda_nex _i em (Agenda , I em ) . I em = " de ach PlaneBo omWing1 pa OccludedBy PlaneUppe Body1 " The obo can eason abou wha ac ion i should pe o m ha es ablishes a planning decision in i s belie s a e. I can, o example, ask wha ac ion i should pe o m o dissol e he pa OccludedBy ela ion be ween pa s: ?−holds( usesAc ionMappe ( S a egy , Mappe ) ) , p ope y_ ange (Mappe , mapsI em , Pa e n ) , indi idual_o (I em , Pa e n ) , c a l l (Mappe , I em , Ac ion ) . Ac ion = [an , ac ion , [ ype , ’ Pu AwayPa ’ ] , [mo esPa , ’ PlaneUppe Body1 ’ ] , . . . ] . 4.4.2 Case s udy: Use o ailu e on ology in obo ic assembly manipula ion planning To illus a e he p oposal he assembly o he Ba a oy is p oposed, as shown in Fig. 4.3, Fig. 4.4 shows snapsho s o he esul ing plan in simula ion and eal en i onmen , espec i ely. The sys em is es ed wi h di e en spa ial cons ain s. Geome ic easoning abou occlusions allows he obo o know when i needs o mo e pa s Page 83 o 151 CHAPTER 4. An On ology o Failu e In e p e a ion and Reco e y in Planning and Execu ion ou o he way and change he ac ion sequence p o ided by he logic-based planne . Some geome ic si ua ions a e used o es ing. The sequence o plan he assembly ope a ions is: 1. call IK module o check eachabili y o g asping he objec s; 2. call a collision checke o alida e a pa h o g asp an objec . Some ailu es can happen and ge epo ed o he planne , whe e he ailu es a e in e p e ed and a decision on he nex ac ion is made. Some si ua ions in manipula ion domain may happen o en, such as he case whe e an objec is blocking he chosen con igu a ion o g ap/place an objec . This si ua ion equi es he selec ion o al e na i e easible (o eachable) g asping poses and/o placemen s. Fo example, as shown in Fig. 4.3, i he objec Bo omWing has ou g asping poses om each side g1-g4, he g1 and g4 a e occluded by he holde s o P opelle Holde and TopWingHolde espec i ely, ha means g1 and g4 a e no easible and he obo is no able o each objec Bo omWing h ough hem. Meanwhile, he obo may no be able o g asp he objec Bo omWing h ough g2 and g3 because o he in easibili y o IK con igu a ions. By que ying o e he p oposed on ology, he obo will be able o analyze he cause and epo o he planne . The ailu e symp om p oduced when planning o use g2 and g3 o g asping is a Reachabili yFailu e, which is a Capabili yFailu e, whe eas a ailu e p oduced when planning o g asp using g1 o g4 is an OcclusionFailu e, which is an A o danceFailu e. Bo h o hese a e Incep ionFailu es which p e en he planned ask om e en being unde aken. Capabili yFailu e can be add essed by gene a ing new capabili ies, e.g. selec ing new g asping poses ha a e eachable. A o dance ailu es, meanwhile, can be add essed by manipula ing he en i onmen o be e expose i s a o dances, so by gene a ing in e media y goals o mo ing he occlude s ou o he way, he obo can e en ually g ab he Bo omWing. To in e p e he causes o hose si ua ions p esen ed in Fig. 4.3, he geome ic on ology is in eg a ed wi h he ailu e on ology as desc ibed in Fig. 4.6. This on ology desc ibes he ailu e symp oms, as well as he Failu eNa a i es which make use o hese symp oms o classi y ailu es. The na a i es may include o he in o ma ion o enhance he diagnosis p ocess, such as he ini ial goal and pa icipa ing objec s in he agen ’s ask, and a diagnos ic o indica e which componen ailed. Failu e symp oms a e on ologically cha ac e ized also in e ms o wha ailu e diagnos ics hey a e compa ible wi h; o example, a Reachabili yFailu e can only be used by a ailu e na a i e whe e he explana ion ole is played by a ailu e diagnos ic ha names some IK componen as he ailu e cause. These modules (i.e, IK and collision check) a e low-le el modules ha he symbolic le el conside s o be spa ial easone s. Fo mo e explana ion on how o use he FailRecOn amewo k, ano he example is p oposed Page 84 o 151 4.5. Summa y o he chap e Figu e 4.6: An in e p e a ion o blocking objec using he p oposed ailu e on ology. The concep s in blue belong o ailu e on ology, meanwhile he ones in yellow a e om he geome ic on ology. (see Sec. 5.9.4 Using eco e y module wi hin SkillMaN) in he nex chap e using he esul s achie ed in his chap e . 4.5 Summa y o he chap e This chap e p oposes: 1. The o maliza ion and implemen a ion o he ailu e in e p e a ion and eco e y on ologies as well as geome ic on ology o ex end he capabili ies o au onomous obo s ela ed o manipula ion asks ha equi e ask and mo ion planning along wi h execu ion. 2. The in eg a ion o geome ic on ology wi h ailu e in e p e a ion and eco e y ones, which is equi ed o he combina ion o geome ic and symbolic le els o planning o in e p e such ailu es. 3. The he e ogeneous way o easoning which imp o es he obo capabili y o execu e complex manipula ion asks. In he modeling le el, he absolu e concep s o he a o emen ioned on ologies a e modeled unde DUL and SUMO ounda ions o acili a e he usabili y o obo icis s communi y. A case s udy is in oduced o illus a e he use o ailu e on ology in au oma ed planning and wo k low execu ion phases by p oposing he common si ua ions ha could be encoun e ed by such planne . The a o emen ioned on ologies ha e he in e ace o access he low-le el geome ic modules o eed hem back he equi ed quali a i e spa ial easoning. Page 85 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning 5.3.1 Sensing module We ex end ou wo k p esen ed in chap e h ee (PMK), by enhancing he sensing module. The sensing module in eg a es di e en ypes o senso s, RFID, wi h one-dimensional ou pu da a, and RGB-D came a, wi h mul i-dimensional ou pu da a. The pu pose o he mul i-senso y in eg a ion is o co e he non-line o sigh (NLOS) by using RFID echnology as exploi ed in (Deyle,2011), and line o sigh (LOS) by using a came a. This in eg a ion allows he obo (especially he ones ha ha e na iga ion capabili ies) o igu e ou whe e he objec s a e loca ed in an indoo en i onmen (e en i hese objec s a e hidden, like cans inside a d awe ), and he s a us o he objec s (e.g., a can is ull o emp y). a) RFID echnology An RFID echnology is composed o h ee main pa s: eade , ags and an enna. The ags, like he ones used in (Deyle e al.,2014), ha e a physical s o age medium ha allows a obo o s o e ele an da a ela ed o he s a us o he objec o i s ela i e loca ion o spa ial ela ionships, in o ma ion ha can be au oma ically upda ed om he esul o he obo ac ions. The use o RFID echnology has appea ed om he beginning o his cen u y and mos o he ela ed wo ks a e ocused on localiza ion, like he wo ks p esen ed in (Deyle e al.,2014) and (Li e al.,2010). Howe e , ew e o s ha e been done o u ilize he memo y inside he ags o au onomous manipula ion asks, which equi es implemen ing obus s a egies o s o e and upda e he da a in memo y. He e, in his wo k, we make use o associa ed memo y o s o e he dynamic da a and upda e hem acco dingly o suppo he planning sys em by ex ac ing he ele an in o ma ion. The pu pose o using RFID in SkillMaN is: •To pa ially localize he objec s in he indoo en i onmen . This allows he obo o s a planning unde pa ial in o ma ion o he en i onmen ins ead o disco e ing he en i e en i onmen which inc eases he compu a ional cos o he planning p ocess. This includes igu ing ou he hidden objec s ha he o he senso s like came a can no de ec . Then, he in eg a ion wi h o he senso s like a came a can p ecisely ecognize he objec s. •To s o e ele an da a ega ding he s a us o he objec s in he en i onmen , such as a can is ull o emp y, in o de o adap he manipula ion beha io o he obo . b) Came a Using he came a and isual ags a ached o he objec s, he objec poses a e ob ained (mo e complex un agged-based pose es ima ion algo i hms could be used). Then, spa ial ela ionships a e ex ac ed geome ically o unde s and he s a e o he physical wo ld. Rela ionships Page 92 o 151 5.3. Assis an modules cu en ly suppo ed by he amewo k a e in,on,inside, igh and le as p esen ed in (Diab e al.,2019). The ags a e used o iden i y he wo ld en i ies and seman ically link hem o he he p ope ies o each objec . Speci ically, he pu pose o in eg a ing he sensing module wi h a came a is o p ecisely de ec he posi ion o he objec s and hei IDs and asse hem on he ele an on ology. Then, e alua e he spa ial ela ions o he wo ld en i ies wi h espec o each o he and he obo (e.g., objec A is loca ed on he igh side o he obo and on he le side o objec B.) 5.3.2 Geome ic module The geome ic module p o ides se e al se ices ha help a planne o e alua e he easibili y o he skills, as desc ibed in chap e ou . I can be summa ized in o ou main se ices: 1. In e se Kinema ics (IK) used o compu e he obo con igu a ions o a gi en g ippe pose, 2. Collision Check (CC) used o check he easibili y o a single con igu a ion o a ajec o y, 3. Mo ion Planning (MP) used o gene a e a sampled-based ajec o y o be execu ed, and 4. Objec Placemen (OP) used o sample he placemen egion. In SkillMaN, hese se ices may be equi ed in many si ua ions in he manipula ion domain, such as he case whe e an objec is blocking he chosen con igu a ion o g asp/place an objec . This si ua ion equi es he selec ion o al e na i e easible (o eachable) g asping poses and/o placemen s. Speci ically, hese se ices a e used o: 1. Compu e an al e na i e g asp o an al e na i e placemen pose o he objec . 2. Compu e he IK o he new g asp o he IK o he new objec placemen acco ding o he cu en g asp. 3. Compu e a collision- ee pa h o he new goal con igu a ion. These se ices may be equi ed du ing planning o ind a easible solu ion, o o gene a e eco e y s a egies o eco e a plan whene e a ailu e occu s. 5.3.3 Adap a ion module This is a module ha adap s he obo mo ions o he ac ual scene based on he pe cei ed poses o he objec s in he en i onmen . B oadly, he e a e wo sou ces om whe e o adap he Page 93 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning mo ions, one is om human demons a ions, he o he is om collision- ee mo ions compu ed by a mo ion planne . This is simila o wha human do, i.e., we in ui i ely know how o pe o m he mo ion p imi i es, al hough ou exac mo emen s a e only p oduced when we see he objec s and adap hem o he scene con ex while we pe o m he skill. The echnique used o imi a ion mo ions has been he Dynamic Mo emen P imi i es (DMPs), ha implemen s a se o di e en ial equa ions ha allow desc ibing any mo ion. Complex mo ions ha e long been hough o be composed o se s o p imi i e ac ions ha a e execu ed oge he . DMPs a e a ma hema ical o maliza ion o he mo ions o he p imi i es using dynamical sys ems heo y. These dynamical sys ems ha e s able beha io s using he basic se o pa ame e s p o ided by he DMP ool, al hough ex a pa ame e iza ion acco ding o he ask a hand may imp o e he esul s. 5.4 Knowledge modules Fo mally, knowledge is di ided in o knowledge ep esen a ion and in e ence mechanism. The o me copes wi h how he knowledge is ep esen ed, he la e copes wi h how o in e he ele an knowledge. In SkillMaN, he main goal is o cap u e knowledge abou 1. how he planning sys em can be guided by he knowledge, 2. how he simila i y o he si ua ions can be checked, 3. how o manage he pe cep ion sys em, 4. whe he a pa h is easible o no , 5. how can obo s pe o m skills and wha skills a e needed o achie e ce ain goals, 6. how can a si ua ion be in e p e ed as a ailu e, and 7. which a e he a ailable eco e y s a egies o a gi en ailu e. The knowledge modules a e i s in oduced and hen he he e ogeneous in e ence mechanism will be de ailed a he end o he subsec ion. 5.4.1 Expe ien ial knowledge module The knowledge-based expe ience, o expe ien ial knowledge, is di ided in o wo main pa s: knowledge o planning and si ua ional knowledge. Page 94 o 151 5.4. Knowledge modules Figu e 5.4: The si ua ion modeling in SkillMaN. The blue and o ange a e abs ac concep s, while he ed e e ed o he ins ances o he classes. The mul i-senso y module is used o build he desc ip ion o he en i onmen . A) Knowledge o planning: One o he signi ican equi emen s o he planning sys em is o eason abou how o pe o m skills in semi/uns uc u ed en i onmen s. This equi es ha ing geome ic in o ma ion, based on he cu en si ua ion o he en i onmen al en i ies, o how o manipula e he objec s, e.g. which is he expe ience-based easible g asp. This is wha we call "geome ic skills expe ience". Cu en ly, in SkillMaN, he e a e wo sou ces om whe e o build he geome ic skills expe ience: humans and obo s. Fo humans, he use can build manually he geome ic skill expe ience including he desc ip ion o he ask cons ain s (ei he p og ammed o included in an on ology). Fo he obo s, i he obo s a s explo ing he way o execu ing an ac ion and inds a easible solu ion, i s o es his solu ion o be la e used i equi ed. Fo example, le ’s conside he side-g asp is used o picking an objec om a able and he e is an obs acle occluding he pa h om a ce ain angle, se e al angles could be applied o explo e he easibili y o he g asping con igu a ions. Once ound, he obo s o es hese con igu a ions o be used in simila si ua ions. This knowledge is equi ed o guide he planning sys em especially when some mo ion cons ain s exis . In he p oposed case s udy, se e al cons ain s ha e been in oduced o show he impo ance o using geome ic skills expe ience wi hin a planning sys em as shown in Sec. 5.8. B) Si ua ional knowledge: In SkillMaN, expe iences a e hough o be si ua ions ha p o ide a ela ional con ex on a se o e en s ha occu ed, and objec s ha we e in ol ed. This includes, e.g., wha oles an Page 95 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning objec plays du ing an skill, and wha he diagnosis is in case a ailu e was aised du ing skill execu ion. Si ua ions in ou knowledge base, as desc ibed in Fig. 5.4, can be w i en as a uple < S, O, E >, whe e Sis he skill ha was execu ed, O he se o objec s ha we e in ol ed, and E he se o e en s ha occu ed. The ep esen a ion s a egy o he Desc ip ions and Si ua ions on ology (Maass e al.,2007) has been ollowed whe e desc ip ions a e used o c ea e iews on he ela ional con ex o si ua ions. In pa icula , he p oposed amewo k associa es skills o he si ua ions whe e he skill was execu ed, and exploi s his in o ma ion o he ealiza ion o a se o in e ence mechanisms. The en i onmen ’s desc ip ion is used o seman ically link low-le el pe cep ion da a wi h high-le el knowledge, and o analyze he si ua ion o he en i onmen en i ies in o de o enhance he ask execu ion. The agged-based senso s, i.e., RFID and came a, a e used o iden i y he wo ld en i ies and seman ically link hem o he p ope ies o each objec . Speci ically, he pu pose o he sensing module is o de ec he posi ion o he objec s and hei IDs and asse hem on he on ology o build he desc ip ion o he en i onmen and i s ele an ins ances ollowing he Pe cep ion and Manipula ion Kowledge (PMK) p esen ed in chap e 3. C) Expe ien ial da a: Ou sys em eco ds senso y da a o e ime and associa es i o si ua ions du ing which he da a was acqui ed. This is mainly o cap u e he ajec o ies ha we e execu ed by he obo , and o associa e hem wi h ask, en i onmen , and execu ion. This is use ul o machine lea ning applica ions whe e exp essi e que ies can be answe ed a highe le els o he knowledge base, and esul s o such que ies may se e as il e o he lowe -le el da a o ga he only he da a ma ching a seman ic si ua ion. In SkillMaN, wo ypes o s o age mediums a e used, he memo y o RFID and knowledge da abase. The o me is used o s o e he dynamic da a such as objec s’ posi ions and hei s a us (e.g., a can is ull o emp y). The la e is used, beside guiding he planning sys em, o s o e he s a ic da a such as objec s’ ea u es. Da a managemen is de ailed in Sec. 5.4.5 5.4.2 Awa eness module This module con ains low-le el knowledge ela ed o pe cep ion, o geome ic issues equi ed o he e alua ion o he ac ions easibili y, and o he way o execu ing skills. A) Pe cep ual knowledge To pe cei e a obo en i onmen , di e en senso s a e usually used. Senso s p o ide da a Page 96 o 151 5.4. Knowledge modules Figu e 5.5: The ep esen a ion o pe cep ual knowledge in SkillMaN. Page 97 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning abou he en i onmen in he o m o signals (one dimension) o images (mul i-dimension), and o ob ain he use ul ea u es om he pe cei ed da a he sui able algo i hms ha e o be applied, o ins ance o de ec an objec pose some pose es ima ion algo i hms based on image ea u es can be applied, o al e na i ely algo i hms based on ags iden i ica ion can be used. The complexi y inc eases when he in eg a ion be ween he senso s exis s. Pe cep ual knowledge is he knowledge ela ed o he obo senso s o o senso s associa ed o he en i onmen . This knowledge is used o guide he p oposed mul i-senso y module. A i s e sion o he pe cep ual knowledge was p esen ed in chap e 3, whe e wo came as wo ked in pa allel o pe cei e he able- op en i onmen . He e, we enhance he ep esen a ion o he knowledge o be capable o wo king wi h di e en ypes o senso s including RFID and came a. The pe cep ual knowledge o mul i-senso y in eg a ion in SkillMaN, as shown in Fig. 5.5, is ep esen ed as a uple < D, C, A > whe e D is a measu ing de ice (senso ), C is he senso cons ain s o limi a ions, and A is he co esponding algo i hm o ex ac he ea u es om he senso signals. This knowledge is esponsible o answe h ee main ques ions which a e he senso s a ached o he obo ?, wha ype o da a he senso s pe cei e and wha a e hei limi a ions?, how o ex ac he ele an da a?. To answe he i s ques ion, a desc ip ion o he senso s is p oposed o make he obo unde s and which a e he g oup o senso s i has. Mo eo e a desc ip ion o he componen s o each senso , like he ags, an ennas and eade o he RFID is included. To answe he second ques ion, a desc ip ion o he pe cep ual ea u es is p oposed o cla i y o he obo which ype o da a (i.e, one o mul i dimensions) he senso s a e pe cei ing, and wha a e he cons ain s o limi a ions o each senso . To answe he hi d ques ion, a me hod o call he co esponding algo i hms is p oposed o ex ac he ele an da a. Using Desc ip ion Logic (DL, (Baade e al.,2017)), he knowledge o RFID senso s is ex- p essed as: RF IDKnowledge :− ∃hasSupe class(RF ID, Senso ) ∧∃Sense(RF ID, RSSI) ∧∃hasSensingComponen s(RF ID, T ag) ∧∃hasSensingComponen s(RF ID, Reade ) ∧∃hasSensingComponen s(RF ID, An enna) ∧∃hasID(RF ID, aggedID) ∧∃hasCons ain s(RF ID, minRange) ∧∃hasCons ain s(RF ID, maxRange) ∧∃hasAlgo i hm(RF ID, eadT ag) And he knowledge o came a is exp essed as: Page 98 o 151 5.4. Knowledge modules Came aKnowledge :− ∃hasSupe class(Came a, Senso ) ∧∃Sense(Came a, Image) ∧∃hasCons ain s(Came a, minRange) ∧∃hasCons ain s(Came a, maxRange) ∧∃hasAlgo i hm(Came a, iagoCam) B) Geome ic knowledge The geome ic knowledge has a s uc u e o sequen ial access o he geome ic se ices in he assis an laye . The main ad an age o his on ology is ha , ins ead o calling he module manually om he ask and mo ion planning clien , he obo can que y o e he knowledge o e ie e he sequence o p ocesses equi ed o execu e such ac ions in an au oma ic way. C) Skill knowledge A skill, in SkillMaN, is a desc ip ion o wha he obo can do. The SkillMaN p o ides some me hods o eaching new skills o he obo inspi ed by he wo k p esen ed in (Munawa e al., 2018): 1. P imi i e skills consis o a sequen ial lis o a omic ac ions, which e e s o a single ac ion o ges u e, including i s p econdi ions and e ec s. Fo example, an openD awe skill is composed o he sequence o ac ions: mo e o he handle posi ion, close he g ippe , and inally pull he d awe . 2. Rule-based skills consis o a se o “i A hen B ules" o issue app op ia e ges u es acco ding o senso s ou come. Bo h me hods, howe e , canno be execu ed on hei own. They equi e a s uc u e, such as a wo k low, ha con ains he abs ac s eps ha a e usually equi ed o ask execu ion. This s uc u e is desc ibed a a symbolic le el and g ounded o be a ached o each skill using he assis an laye . The main di e ence be ween he bo h a o emen ioned me hods is a pe cep ion- based condi ional node in he s uc u e. Tha means he s uc u e includes some b anches o a gi en alue ha should be sensed. Page 99 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning 5.4.3 Reco e y module Knowledge o eco e y is a module ha p o ides an in e p e a ion o he ailu es ha occu . In SkillMaN, an in e p e a ion ailu e on ology desc ibed in chap e 4co e ing se e al sou ces o ailu es, is used bo h du ing planning and execu ion. I o e s eco e y s a egies o : 1. Geome ic ailu es, ha may appea when e.g. he obo can no each o g asp/place an objec , he e is no collision- ee pa h o he e is no easible In e se Kinema ic (IK) solu ion; 2. Ha dwa e ela ed ailu es ha may appea when e.g. he obo in a eal en i onmen equi es o be e-calib a ed (g ippe o a m), o i is sen o a non- eachable con igu a ion; 3. So wa e agen ela ed ailu es, ha may appea when e.g. he obo has so wa e componen s ha ail like when an algo i hm is no able o ex ac he p ope ea u es. 5.4.4 He e ogeneous in e ence mechanism This sec ion p esen s a he e ogeneous way o easoning ha includes symbolic easoning o e he knowledge module and geome ic easoning. The o me includes il e ing he si ua ion om he da abase, si ua ion simila i y check, skill easoning, seman ic easoning ega ding he en i onmen and i s en i ies, manipula ion cons ain s and pe cep ion. The la e includes he geome ic easoning o check he easibili y o he gene a ed skills. They a e discussed below. Symbolic easoning A.1) Fil e ing si ua ion: Fil e ing si ua ion is a p ocess o inding hose si ua ions ha sa is ies a skill desc ip ion. I means he obo has o de ec he si ua ions ha use a speci ic skill in hei desc ip ion, e.g. using a P olog p edica e (WIELEMAKER e al.,2012) he easoning on “which a e he si ua ions ha con ain a ce ain skill?” is: ?− i l e S i u a i o n ( h as Skill ( Si ua ion , S k i l l ) ) , ?− i l e S i u a i o n ( hasPa icipan ( Si ua ion , Objec ) ) , ?− i l e S i u a i o n ( hasPa ( Si ua ion , E en ) ) . Si ua ion=[S kill , Objec , E en ] . A.2) Si ua ion simila i y check: The simila i y o si ua ion scenes is compu ed using axonomic in o ma ion om he si ua ional on ology oge he wi h in o ma ion abou wha makes up he compa ed en i ies. No e ha scenes a e compound en i ies – ha is, a scene has objec s and agen s as pa icipan s. Page 100 o 151 5.4. Knowledge modules Figu e 5.6: An example o simila i y check be ween wo scenes. This example is also a pa o he expe imen al scenes o scena io one in Sec. 5.8 (s o age ask). Objec s and agen s hemsel es a e compound en i ies; an objec o agen may ha e o he objec s as pa s. Also, he desc ip ion o an agen includes he skills o be execu ed, geome ic-skills expe ience, and agen goal. En i ies ha a e conside ed simple – he pa s o objec s o agen s – a e compa ed using Wu- Palme simila i y (Wu and Palme ,1994), al hough o nume ical s abili y easons, he loga i hm o his simila i y sco e is used he e, i.e., o wo indi iduals x and y: Sim(x, y) = log dep h (lca C(x), C(y)) 0.5(dep h C(x)) +dep h C(y)) (5.1) whe e C(x)is he class o which indi idual xbelongs, dep h(A)is he dep h o class A in a axonomy, and lca(A, B)is he lowes common ances o o classes A, B in ha axonomy. The in ui ion behind Wu-Palme simila i y is ha simila classes should be close o each o he in he axonomy. To compa e indi iduals x, y ha a e compound en i ies, hei pa s a e ma ched such ha o e e y pa xpo x, we ind he pa ypo y ha maximizes Sim(xp, yp). Then, he sum o he simila i y sco es ob ained om hese ma ching is added o Sim(x, y). The in ui ion he e is ha we wan o ha e he simila i y o complex objec s such as si ua ions o scena ios o depend on he na u e o hose scena ios as well as hei pa icipan s. We only compa e he “ ee” o pa -hood ela ions o e iciency easons. In p inciple, he e may be many s o ed scenes one could compa e he cu en si ua ion o, and il e ing ou mos o hem so ha only a ew ele an candida es emain. Once some candida e simila scenes a e selec ed, he mo e in ensi e p ocedu es o adap ing obo mo ion om he s o ed scene o he cu en one can be used o asce ain he use ulness o he s o ed expe ience o he cu en ask. Page 101 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning 5.7 Implemen a ion and se -up 5.7.1 Implemen a ion ools A) Pe cep ion The C++ lib a y a – ack–al a (h p://wiki. os.o g/a _ ack_al a ) has been used o de ec he objec pose and ID. Mo eo e , he C++ lib a y ThingMagic Me cu y API (h p://www. hingmagic.com/manuals- i mwa e) o RFID echnology has been used o de ec he objec s, including he hidden ones, and o s o e he ele an dynamic in o ma ion. Some se ices a e implemen ed o ead he agID, ead he da a om memo y and w i e/upda e he da a on he memo y. These IDs a e asse ed in he knowledge o ex ac a seman ic desc ip ion o he objec . All he ans o ma ions o he objec s and came a a e calcula ed wi h espec o he wo ld ame. B) Planning and adap a ion A planning sys em consis s o wo main phases: ask planning and mo ion planning. The i s is implemen ed using he Fas Fo wa d (FF) ask planne o gene a e a sequence o ac ions. The la e is implemen ed using The Kau ham P ojec (Rosell e al.,2014). The Kau ham P ojec is a C++ based open-sou ce ool o mo ion planning, ha enables o plan unde geome ic and kinodynamic cons ain s. I uses he Open Mo ion Planning Lib a y (OMPL) (Sucan e al., 2012) as a co e se o sampling-based planning algo i hms. In his wo k, he RRT-Connec mo ion planne is used o gene a e a pa h be ween wo con igu a ions. The main echnique on o which he imi a ion mo ions ha e been implemen ed is he DMP (Ijspee e al.,2002). The DMP expe imen s a e pe o med i s in simula ion and a e wa ds using he eal obo . The expe imen consis ed in lea ning by eco ding he execu ion o he planned-base mo ion and hen changing he ini ial and inal poin s o see how he planned ges u es a e imi a ed. An in e ace has been implemen ed o command he a m o pe o m such mo ions ha equi e imi a ion ges u es. This in e ace can be di ided in o h ee main pa s, he da a acquisi ion p ocess, he DMPs gene a ion, and he execu ion o he mo ion. The da a acquisi ion is pe o med by eco ding he mo ion. The DMPs gene a ion is done using he mo ion eco ded as inpu o lea n how o pe o m he DMP p imi i e in a new si ua ion. The execu ion o he mo ion uses he ini ial con igu a ion and goal s a e equi ed o adap he mo ion in simila si ua ions. The in eg a ion be ween he planning and adap a ion ools is done au oma ically oge he in he p epa a ion phase. Page 108 o 151 5.7. Implemen a ion and se -up In SkillMaN, he p oposed abs ac p imi i es a e desc ibed: 1. eleaseG ippe : an a omic ac ion used o open he g ippe . 2. closeG ippe : an a omic ac ion used o close he g ippe . 3. pick-place: an skill ha con ains he a omic ac ions mo e, hold and pu -down; i is used o ans e ing he objec s be ween wo loca ions. 4. openD awe : is an skill ha con ains he a omic ac ions mo e, hold and pull; i is used o opening/closing he d awe s. 5. se ing: is a skill ha con ains he ac ions mo e and pou ; i is used o se ing he be e ages o a cus ome . The mo ions ha a e adap ed a e ini ially ei he compu ed by he mo ion planne , e.g, in case 3 and 4, o copied om human demons a ions, e.g, in case o 5. In case o 1 and 2, he mo ion o closing and opening he g ippe is p ede ined. These abs ac p imi i es co espond o he basic unc ions o he obo manipula o , which can be implemen ed in many di e en ways. Ou way o implemen ing such p imi i es is a he lowes mo o con ol le el. The ocus o ou wo k is, howe e , no a speci ic implemen a ion, bu a he we would like o p opose a way o combine hem o seamlessly pe o m skills. C) Knowledge The knowledge is designed using on ology web language (OWL) using he P o égé on ology edi o (h p://p o ege.s an o d.edu/). On ology ins ances can be asse ed using in o ma ion p ocessed om low-le el senso y da a. Que ies o e he knowledge o eason o check he simila i y a e based on SWI-P olog and i s Seman ic Web lib a y which se es o loading and accessing on ologies ep esen ed in he OWL using P olog p edica es. A ROS (Robo ope a ing Sys em) in e ace has been implemen ed in o de o acili a e he que y-answe p ocess as a clien -se ice communica ion. The PMK app oach, as p esen ed in chap e 3is used in his wo k. I is explici ly implemen ed o enhance Task and Mo ion Planning (TAMP) capabili ies in he manipula ion domain. I is in eg a ed wi h he mul i-senso y module allowing he ins ances o be asse ed o he on ology using in o ma ion p ocessed om low-le el senso y da a. Page 109 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning Figu e 5.9: Na iga ion expe imen : (a) he plan iew o he indoo en i onmen , (b) he eal scene o how he obo de ec s he ables used in he indoo en i onmen , and (c) pa h planning, obs acle a oidance capabili ies, and na iga ion poses on he map. D) Na iga ion and mapping Fig. 5.9 desc ibes he na iga ion s a egy p oposed in SkillMaN. In (a), he plan iew o he indoo en i onmen has been shown. By using he mobile capaci y o TIAGo, i plans owa d he na iga ion posi ion o he objec s (e.g, TIAGo na iga e owa d he picking and se ing ables) un il i de ec s he labels a ached o hem, as shown in (b). Du ing he na iga ion, TIAGo has he capabili ies o pa h planning wi h obs acle a oidance and localiza ion o he objec s in he map, as shown in (c). All he objec s in he en i onmen a e localized wi h espec o he e e ence ame. D.1) Na iga ion: TIAGo has au onomous na iga ion unc ionali ies implemen ed using he ROS 2D na iga ion s ack (h p://wiki. os.o g/na iga ion). This package is one o he mos commonly used o implemen mapping and au onomous na iga ion solu ions in obo s unning on ROS. I akes in in o ma ion om odome y and senso s eams and ou pu s eloci y commands o send o he mobile base. This na iga ion so wa e is composed o se e al di e en ROS nodes, se ices and opics ha a e able o pe o m SLAM (Du an -Why e and Bailey, 2006). Using he in o ma ion s o ed on he map and he da a o i s su oundings p o ided by di e en senso s, his package is capable o compu ing a sui able pa h o lead he obo o a ce ain goal posi ion wi hou hi ing any obs acle. D.2) Mapping: The mapping and pose gene a ion p ocess s a s by c ea ing he occupancy Page 110 o 151 5.7. Implemen a ion and se -up Figu e 5.10: Task managemen and he communica ion wi h he ROS-based se ices om symbolic and low-le el modules. g id map o he en i onmen o he obo . To ob ain i , he gmapping (h p://wiki. os.o g/ gmapping) package ins alled in he obo has been exploi ed. This map is necessa y o he na iga ion o success ully mo e h ough he oom a oiding any collision. D.3) Localiza ion: Localiza ion is achie ed by wo king wi h he amcl package (h p://wiki. os.o g/amcl). This package is a p obabilis ic localiza ion sys em o a obo mo ing in 2D. I implemen s he adap i e Mon e Ca lo localiza ion app oach (MCL), which uses a pa icle il e o ack he pose o a obo agains a known map. MCL gene a es a cloud o pa icles which ep esen he possible s a es o he obo dis ibu ion. Each pa icle ep esen s a possible pose and o ien a ion o he obo on he map. 5.7.2 Task manage algo i hm The SkillMaN is no implemen ed o a speci ic ask, i is qui e gene al and i accep s se e al asks in indoo en i onmen s wi h he conside a ion o some changes ega ding he desc ip ion o he en i onmen , as discussed in Sec. 5.9.3. All he modules men ioned in Fig. 5.3, and he p o ided se ices o each module as desc ibed in Fig. 5.10, a e used by he ask manage . The ask manage is esponsible o call hese se ices in o de o au onomously execu e he asks, as desc ibed in Algo i hm 1. The sensing module is managed h ough he pe cep ual knowledge, he ollowing se ice is Page 111 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning Algo i hm 1: askManage 1ini ialS a e ← unPe cep ion (RFID) // un RFID senso o pe cei e he en i onmen using he pe cep ual knowledge 2while Task goal no deli e ed do 3D = loadP DDL // load he domain and p oblem iles om skill knowledge 4P = F F (D)// plan a symbolic le el o compu e he sequence o skills 5skillName ← i s Ac ion(P) 6while skillName do 7objec s, poses ← unPe cep ion(Came a) // pe cei e he ac ual s a e o he en i onmen acco ding o he skill 8Y = skillName, objec s // s o e he cu en si ua ion. 9F = il e Si ua ion(skillName) // e u n a se o si ua ions ha use he same skill 10 S = simila i yCheck(F, Y) // e u n he si ua ion which is mos simila in he simila i y check, i any 11 i S! = emp ySe hen 12 Mo ←loadMo ion (S) 13 expKnow ←expe ien ialKnow // e u n he geome ic-skills expe ience 14 Adap edMo ←adap Mo ion(Mo , expKnow) // adap mo ion in o he cu en si ua ion 15 C=checkF easibili y(Adap edMo )// e i y he easibili y o he adap ed mo ion 16 i Adap edMo = easible hen 17 execu e(Adap edMo ) 18 s o e[Adap edMo , skillName, Y ]// s o e he execu ed mo ion, skill and he cu en scene si ua ion 19 i S=emp ySe o C=in easible hen 20 Mo =gene a eMo ion(skillName)// gene a e a collision- ee mo ion o a new si ua ion 21 i Mo = easible hen 22 execu e(Mo ) 23 s o e[Mo , skillName, Y ] 24 skillName ←nex Ac ion(P) Page 112 o 151 5.7. Implemen a ion and se -up used o his pu pose: • unPe cep ion used o selec he co esponding algo i hm(s) associa ed o he a ailable senso s. The sensing se ices a e used o p o ide he ini ial s a e o he en i onmen o he planne o whene e equi ed, using RFID senso and came as. The RFID senso has ou main se ices: 1. Ini ializeRFID used o se up he RFID sys em (i.e., eade , an ennas and ags), 2. ReadRFIDTag used o ead he RFID ags ID associa ed o he en i ies, 3. ReadRFIDMem used o ead he dynamic da a s o ed in ags’ memo y, and 4. W i eRFIDMem used o upda e he ags’ memo y. The came a has wo main se ices: 1. Cam s a us used o ini ialize he came a, and o inpu ( om ei he a human guidance p ocess o a mo ion planne ) he mo ion o be adap ed, and 2. Loca eCam used o es ima e he objec s poses and hei IDs. In he planning phase, wo se ices ha e been used o call he he heu is ic-based ask planne FF (Fas Fo wa d) and o load he domain and p oblem iles: 1. loadPDDL used o au oma ically load he PDDL domain and p oblem iles, as desc ibed in Sec. 5.4.4 2. FF used o au oma ically compu e symbolically he sequence o skills o be execu ed. In he analysis o each ac ion he solu ion plan, wi h a guidance om knowledge modules, he ollowing se ices a e used: 1. il e Si ua ion used o il e he si ua ions ha include a speci ic skill in he da abase, 2. simila i yCheck used o compa e he cu en si ua ion wi h he o he s s o ed in he da abase, Page 113 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning 3. expe ien ialKnow used o p o ide geome ic skills, based on he obo expe ience, e.g. he ype o g asp acco ding o he cu en si ua ion. The geome ic se ices a e manged h ough he geome ic knowledge by calling ollowing se ice: •gene a eMo ion used o compu e he ini ial and goal con igu a ions (acco ding o he ac ion/skill o be pe o med and he eachabili y and spa ial easoning p edica es p oposed in Sec. 5.4.4), and he collision- ee pa h be ween hem using he mo ionPlanning se ice. The geome ic se ices a e used o combine he geome ic module wi h a symbolic planning le el o gua an ee he easibili y o he planned skills. This module has ou main se ices, p o ided by The Kau ham P ojec : 1. mo ionPlanning used o compu e a collision- ee pa h, 2. collisionCheck used o e i y whe he a obo con igu a ion is collision- ee o an objec a a gi en pose is no in e e ing wi h o he s, 3. in e seKinema ics used o compu e he obo con igu a ions o a gi en desi ed pose o he end-e ec o , 4. objec Placemen used o sample/check he a ailabili y o placemen loca ions o he objec s. The adap a ion se ices a e used o imi a e/adap he mo ion o each skill o be execu ed in such si ua ions. The e a e wo se ices managed h ough he geome ic knowledge: 1. ainMo ion used o inpu ( om ei he a human guidance p ocess o a mo ion planne ) he mo ion o be adap ed and e u ns he weigh s used o shape he his mo ion, and 2. adap Mo ion used o compu e he ini ial and goal con igu a ions (acco ding o he ac ion/skill o be pe o med and he eachabili y and spa ial easoning p edica es p oposed in Sec. 5.4.4), and o adap he ained mo ion is e i ied wi h he ainMo ion se ice. This in e ace is es ablished based on ROS (Robo Ope a ing Sys em) se ice-clien communica ion. 5.7.3 Expe imen al se -up The expe imen has been done a IOC lab and i is composed o : Page 114 o 151 5.7. Implemen a ion and se -up Figu e 5.11: a) he obo plans how o open he i s d awe ; b) he obo execu es he openD awe skill; c) he obo plans how o pick he black can (based on i s s a us, he e i is emp y) wi h he help o expe ien ial knowledge abou wha is he bes g asp o place i in he i s d awe ; d) he obo execu es he place skill; e) he obo execu es he close ac ion using he ule-based skill; ) he obo checks he simila i y o he cu en si ua ions, i inds he same skill has been used wi h he same objec (a d awe in he ile cabine ), hen execu es he skill wi h he same mo ion used o open he i s d awe ; g) he obo plans how o pick he ed can (based on i s s a us, he e i is ull) wi h he help o expe ien ial knowledge abou wha is he bes g asp o place i in he second d awe ; h) hen, he obo execu es he place skill. Video URL: h ps://www.you ube.com/wa ch? =bTmWAkjC93c 1. The TIAGo obo . 2. A ile cabine wi h ou d awe s. 3. A s o age able ha con ains he objec s (cans). 4. Se e al cans ha may be ull o emp y. 5. A se ing able ha con ains a cup on a ay whe e he obo mus pou he con en s o a can o a cus ome . 6. A pe cep ion sys em wi h came a and an RFID senso ha includes: •A eade ha has he capaci y o eading ou an ennas, dis ibu ed a ound he lab, and le he obo de e mine he egion whe e he objec s a e loca ed. •The ags which ha e a unique ID and a memo y wi h a space o 64 cha ac e s. Page 115 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning Figu e 5.12: a) he obo checks he simila i y o he cu en si ua ion, i inds he same skill has been used wi h he same objec (i.e., a d awe in he ile cabine ), hen adap s he skill wi h he same mo ion used in he da abase; b) he obo igu es ou he op-g asp is no easible o pou ing ac ion, he op o he ile cabine is used as a placemen oom o change he g asp ype; c) he obo changes he g asp ype om he op-g asp o he side-g asp; d) he obo se es he con en s o he can in he cup o a cus ome , he se e mo ion is adap ed om he expe ience, acco ding o he cu en pose o he obo and loca ion o he cup. Video URL: h ps://www.you ube.com/wa ch? =bTmWAkjC93c 5.8 Expe imen al scena ios Be o e desc ibing he p oposed scena ios, some assump ions should be aken in o accoun . 1. All he objec models a e de ined in he knowledge da abase. 2. All a omic ac ions/skills o be used a e desc ibed wi h hei p econdi ions and e ec s in knowledge da abase. 3. All he RFID’s an ennas a e dis ibu ed in he en i onmen in a way ha a oids he in e ac ion be ween he signals ecei ed om each one. Tha means ha each an enna only ecei es he in o ma ion o he ags loca ed in i s co e age egion. 4. All en i onmen al en i ies a e labeled wi h ei he RFID o ision-based ags. 5. All he ea u es ecei ed om he mul i-senso y pe cep ion sys em (RFID and came a) a e eliable enough. 5.8.1 Scena io one: S o age ask Fig. 5.11 shows a sequence o snapsho s o he s o age cans ask. The ask is o classi y he cans on he able o s o e hem in he d awe s based on hei s a us. Fi s ly, he obo checks he s a us o he selec ed can by eading his in o ma ion om RFID memo y. A e compu ing he symbolic plan, he obo can apply he skill openD awe ( he i s ac ion o he symbolic plan) Page 116 o 151 5.8. Expe imen al scena ios o he co esponding d awe o he ile cabine , as shown in Fig. 5.11 a-b. Then, o gene a e a collision- ee pa h, he mo ion planne has been called. Then, wi h guidance om seman ic knowledge and he expe ien ial knowledge (i geome ic expe ience exis ), he obo can eason abou how o apply he pickUp skill o he can om he able and how o pu Down i inside he d awe and close he d awe , as shown in Fig. 5.11 c-e. A e simila i y check p ocess o he cu en si ua ion i.e., compa ing i wi h he ones ha ha e a simila desc ip ion s o ed in he da abase, he imi a ion p ocess is used o imi a e hose skills o be applied o he o he cans, as shown in Fig. 5.11- . The second can is ull and he co esponding d awe whe e o be s o ed is he second one, shown in Fig. 5.11 g-h. The pe cep ion sys em is used o check i he p econdi ions o he ac ions a e sa is ied o no . Fo example, o apply he openD awe skill in he second d awe , he i s one should be closed o allow he obo o pu Down he selec ed can co ec ly in he d awe . 5.8.2 Scena io wo: Se ing ask Fig. 5.12 shows a sequence o snapsho s o he se ing can ask. The ask o se e he con en s o he can inside a cup on he ay. The obo can no apply he pou ing skill wi h he g asping pose used o pick i up om he d awe (which is op-g asp). The obo needs o empo ally place he can o change he g asp om he op o side g asping con igu a ion. The op su ace o he d awe is used as a ee placemen oom o his sub- ask. Then he obo is able o se e he can. Mo eo e , i he posi ion o he cup is changed, he obo will be able o adap he mo ion wi h he new posi ion. Table 5.1: Tes he skill openD awe , pickUp and se ing using adap a ion me hod s he planning sys em wi h and wi hou expe ien ial knowledge. Skill Pa ame e Adap a ion Planning Wi h Exp.know Wi hou Exp.know openD awe S – success a e 100 100 50 T – A g. ime (sec.) 4.5 11.5 39.5 pickUpF omD awe S – success a e 100 100 50 T – A g. ime (sec.) 12.5 33.5 58.5 se ing S – success a e 100 – – T – A g. ime (sec.) 12 – – Page 117 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning Figu e 5.16: The condi ional plan esul s om he planning p ocess. a) lowcha desc ibing he plan ob ained by he con ingen FF. b) lowcha added when execu ing he plan o moni o ing and epai i necessa y he ac ion ou comes shown in ed in he plan. Figu e 5.17: The execu able plan. •T ans e : a skill done by he obo o mo e an a ached objec be ween poses. •Push: a skill done by he obo o push an objec om one pose o ano he one. •Open: a skill done by he obo o open a box-like con aine (a icula ed cap wi h p isma ic join is assumed wi h wo posi ions co esponding o ully closed and ully opened, he s a e being s o ed in he con aine s objec s ea u es). •HumanT ans e : a skill done by a pe son o ans e /push an objec o he obo wo kspace. •HumanOpen: a skill done by a pe son o open a box-like con aine . Sensing ac ions do no in ol e mo ion, and a e de o ed o obse e objec s a us. The obse a ion is done a un- ime. The sensing ac ions conside ed in he example a e : •CheckColo : a sensing ac ion done by a obo o de e mine objec colo . •CheckPose: a sensing ac ion done by a obo o de e mine objec pose. Page 124 o 151 5.9. Discussion Figu e 5.18: The in eg a ion o con ingency plan wi h eco e y knowledge. •CheckCon aine : a sensing ac ion done by a pe son o e alua e whe he a con aine is open o no . •CheckCan: a sensing ac ion done by a pe son o e alua e whe he can-like objec s a e illed o no . The easoning ac ions a e de o ed o in e p e execu ion phase ailu es, p o iding a s a egy (o maybe mo e acco ding o he ailu e ype) o eco e y. The easoning p ocess is done a un- ime. In his example he ollowing easoning ac ion is conside ed: •T ans e - o-box-Failu eIn e p e a ion: a easoning ac ion o e ailu e on ology in e p e s ailu e cause while ans e ing an objec in a con aine , and p o ides a eco e y ac ion o he obo o human depending on he in e p e a ion p ocess. The con ingen Fas -Fo wa d planne inds a he condi ional ee o manipula ion skill plans o execu e he ask, sensing ac ions will be in ol ed in he b anching nodes, and easoning ac ions will be associa ed o he moni o ing o he execu ion o he mo e sensi i e manipula ion skills. C) The in eg a ion o eco e y module in SkillMaN wi h condi ional plan The comple e condi ional ee o plans is ep esen ed in Fig. 5.16. While he planning p ocess is aking place, he e a e se e al challenges in e ms o in e p e a ion o he ac ual s a e which Page 125 o 151 CHAPTER 5. A Skill-based Robo ic Manipula ion F amewo k based on Pe cep ion and Reasoning is cap u ed by a senso y ac ion and handled by he p oposed s a e in e p e a ion. The main challenge is o include in he plan he knowledge abou who is esponsible o he eco e y ac ion execu ion (i.e., human o obo ). Tha means, o ins ance, ha he obo can au onomously epai he ailu e o ans e ing he blue cylinde B inside he box i i in e p e s ha he box is closed. The plan au oma ically ob ains b anching nodes in which sensing and/o easoning ac ions a e assigned o moni o manipula ion skill ou comes in a semi-au oma ic way. Logically, he sensing/ easoning ac ions should be assigned a e each manipula ion skill, howe e his inc eases he compu a ional cos . Ins ead, we assign hem o some manipula ion skill ha a e expec ed o ha e a high p obabili y o ailu e. The easoning ac ion T ans e - o-box-Failu eIn e p e a ion is assigned o he ac ion T ans e - B-box1 based on moni o ing he esul o he skill execu ion. The esul has a Boolean ou come, he success sequence is au oma ically ob ained by he con ingen FF planne as shown in Fig. 5.16-a). I , while moni o ing, a ailu e occu s, he easoning ac ion in e p e s he cause o he ailu e which is a ailed ask execu ion ha has unmani es ed pos condi ions (i.e., Non ealizedSi ua ion). The eco e y s a egies ha e been p o ided based on he cu en s a us o he box (closed o lipped) as shown in Fig. 5.16-b). Reco e y s a egies could be ei he o ask help om humans o he obo eco e s he ailu e by i sel . The o he FALSE ou come b anch o he sensing ac ion CheckCon aine Box1 s a es he box is al eady closed and based on he planne , he ans e o he box cap is applied. The inal execu able plan gene a ed by he planning sys em has a se o easible manipula ion skills as shown in Fig. 5.17. The planne may ask he on ology ques ions abou why cylinde B has no been ans e ed o he box?, once i in e p e s he cu en si ua ion (i.e., he box is closed), hen he nex eques is which is he bes eco e y s a egy?. These eques s o he planne a e handled and answe s by e ie ing in o ma ion, upda ing/dele ing o easoning o e i . As shown in Fig. 5.18- a, he eques -answe ela ion is done using he se ice-clien communica ion o ROS (Robo Ope a ing Sys em, www. os.o g), and in Fig. 5.18-b, he eques -answe que ies o he planne a e desc ibed. 5.10 Summa y o he chap e This amewo k discusses he impo ance o in eg a e pe cep ion, planning, knowledge-based easoning (including expe ience), in a skill-based manipula ion amewo k o le he obo au oma ically pe o m he asks ha include e e y-day ac i i ies. Mo eo e , he amewo k also includes he p ocedu es o de e mine how o manage he da a equi ed o e icien ly pe o m he asks. Two examples ha e been in oduced. In bo h examples, a se o skills such as pickUp, pu Down, openD awe and se ing a e in oduced. The i s example wi h wo scena ios including manipula ion in indoo en i onmen has been in oduced o show he capabili ies o he obo o use he p oposed modules o execu e he e e y-day asks in semi-s uc u ed Page 126 o 151 5.11. Enhancemen en i onmen s. Fo e e y-day asks, he adap a ion me hod is powe ul in e ms o ime when he obo al eady has expe ience o how o execu e he ask. Fo planning, expe ien ial knowledge is used as a geome ic-skill expe ience o acili a e he planning p ocess and educe he cos ha inc eases due o he explo a ion p ocess. In he second example, he in eg a ion o he amewo k wi h a eco e y module has been done, besides he manipula ion skills, a sensing and easoning ac ions a e in oduced o in e ac wi h he physical en i onmen and moni o he esul s o he manipula ion skills. The sys em shows lexibili y o be adap ed in se e al en i onmen s and obo ic s uc u es. 5.11 Enhancemen Fu u e wo k will be how o inc ease he adap a ion capabili y o wo k wi h mo e complex si ua ions which include spa ial ela ions. Also, inding ou he way o au oma e he p ocess o es ablishing a new concep o an unknown en i onmen al en i y. Page 127 o 151 Chap e 6 Conclusions and Fu u e Wo k 6.1 Conclusions The p esen hesis has de eloped se e al amewo ks based on pe cep ion, easoning, lea ning, and planning o add ess he inc easing challenges o obo ic manipula ion p oblems. Di e en so s o modeling and easoning p ocesses ha e been also p oposed inside he amewo ks o come up wi h a easible manipula ion plan. To sum up, he challenges o obo ic manipula ion p oblems conside ed in he hesis a e summa ized as ollows: •Table- op manipula ion p oblems: The need o eason on he cu en s a e o he wo ld in o de o apply one ac ion o ano he (like pick o push an objec o ake i apa ) in a able- op manipula ion scena io. KTMP amewo k o e s he way o encode he knowledge and eason upon i . •Assembly manipula ion p oblems: The need o eason on he esul o ac ions o he need o e alua e ac ions p econdi ions (like he easibili y o a gi en g asp o execu e a pick ac ion) in an assembly ask. FailRecOn amewo k o e s a lexible easoning ool ha is pe ec ly adap ed o knowledge-d i en planning schemes. •Mobile-based e e y-day manipula ion p oblems: The need o make he obo awa e o si ua ion simila i ies o e icien ly e-use p e ious known ways o execu e ac ions by adap ing he obo mo ions o s o ed pa e ns (like opening one d awe once he obo knows how o open ano he ). The SkillMaN amewo k in eg a es he p e ious ools wi h a simila i y si ua ion e alua o and a mo ion adap a ion ool. Two app oaches o planning a e used in his hesis, heu is ic-based app oaches (i.e., Fas Fo wa d (FF) and con ingen -based FF), and knowledge-based planne p o ided by he KowRob 129 CHAPTER 6. Conclusions and Fu u e Wo k g oup. The execu ion o manipula ion asks wi h knowledge-based planning app oaches no explici ly p epa ed o TAMP, like (Teno h and Bee z,2009), can be a challenge because his would equi e, on he one hand, om he knowledge pe spec i e, o p o ide all he componen s in a way ha hey ma ch wi h hei planning sys em. On he o he hand, om he planning pe spec i e, hey would equi e he de ini ion o he ecipes (s a egies) o execu ing he asks (sequence o ac ions), including all possible s a egies o execu ion and he way o swi ch be ween hem when equi ed, which can be a e y expensi e p ocess, especially o asks ha need long sequences o ac ions, such as hose in ol ing manipula ion in clu e ed en i onmen s. Some o he planning app oaches ely on PDDL and on planning s a egies bes i o cope wi h di icul ask planning challenges, like hose ound in he manipula ion domains, al hough he use o PDDL implies a closed-wo ld assump ion, which p ecludes hei use in mo e dynamic en i onmen s ha could equi e pe cep ion and knowledge-based geome ic easoning. PMK allows o b eak he closed wo ld assump ion o classical-based manipula ion planning app oaches. Robo s, like any o he agen , some imes ail. Knowledge-based obo s can eco e om ailu e by easoning whe he o y once mo e, o y some hing else o o mo e o o he asks. We a gued ha he choice has o be based on he concep ual (on ology), he planning ( ask) and he execu ion ( easibili y) le els. This equi es o in eg a e adi ional obo ics domains ( he obo has o ac ) and AI ( he obo has o plan) wi h unplanned si ua ions ( he obo is in an unexpec ed s a e). The in eg a ion o all hese iews aises a a ie y o esea ch ques ions, and so does ou wo k which add esses only pa o his esea ch opic. Fo ins ance, ha dwa e ela ed ailu es ha may appea when e.g. he obo in a eal en i onmen equi es o be e- calib a ed (g ippe o a m). Also, so wa e agen ela ed ailu es, ha may appea when e.g. he obo has so wa e componen s ha ail like when an algo i hm is no able o ex ac he p ope ea u es. One o he ocus o his hesis has been he jus i ica ion and de elopmen o FailRecOn , a gene al and eliable amewo k o ailu e and eco e y managemen . Founda ionally, he p oposed on ologies a e modeled unde SUMO and DUL ounda ions. We ound ha SUMO concep s ha e some limi a ions in he e ms desc ip ions and some missing ocabula y ha we p oposed in PMK on ology. On he con a y, DUL has a well-s uc u e and wide ange o meaning ul concep s ha can be lexibly used in such domains. To inc ease he obo au onomy, he in eg a ion o se ices ha a e necessa y o e e y- day ac i i y asks, wi h a knowledge sou ce, easoning engine and skills desc ip ions a e e y impo an . Mo eo e , he use o expe ien ial knowledge is e y use ul when he obo encoun e s he same si ua ion. Mo eo e , he adap a ion me hodology sa es ime compa ing o mo ion planning. Howe e , huge e o s mus be done in his line o enhance he capabili ies o lea ning om demons a ions in such si ua ions. Finally, he amewo ks ha e been illus a ed o show he main ools and he low o in o ma ion among modules used o pe cep ion, easoning, lea ning and planning le els. Conce ning he obo ic sys ems, we ha e es ed ou esul s wi h he obo s Yumi ABB and TIAGo PAL. All he ela ed esul s a e shown in ideos in URL: h ps://www.you ube.com/ channel/UC6lZ7d7qm5wh5 3 sbEIFgg? iew_as=subsc ibe and al e na i ely in h ps://si .upc. Page 130 o 151 6.2. Fu u e Wo k edu/p ojec s/kau ham/Videos.h ml. 6.2 Fu u e Wo k Along wi h he conclusion poin s s a ed abo e, he cu en hesis, mo eo e , opens new esea ch p oblems ha equi e u he conside a ion such as: F om he many issues ha he KTMP amewo k aises, in he u u e we aim o •inc ease he easoning capabili ies o he amewo k; • educe he unce ain y o he low-le el in o ma ion by using deep lea ning echniques. •inc ease he abs ac concep s o include some no ions like beha io which is impo an o be used in cons ain -based manipula ion planning. •benchma k PMK wi h o he app oaches ha includes mo e conc e e me ics in ela ion o o e all sys em pe o mance, compa ibili y wi h o he amewo ks (e.g. planne s, pe cep ion sys ems), ex ensibili y, eusabili y, scalabili y, ypes o applica ions i can be applied o. F om he many issues ha he FailRecOn amewo k aises, in he u u e we aim o •en ich he causal explana o y module; •imp o e he sea ch o an op imal ma ch be ween wha is known abou a de ec ed ailu e and he eco e y s a egies; •include eco e y s a egies om a alse belie s a e (e.g, caused by alse de ec ion), and •op imize he in e connec ions among he FailRecOn submodules. F om he many issues ha he SkillMaN amewo k aises, in he u u e we aim o •inc ease he obo au onomy by implemen ing a lib a y o ac ions/skills and a sophis ica ed easoning mechanism o allow he obo o eason on he bes ac ion/skill ha can be used in he cu en si ua ion; •inc ease he adap a ion capabili ies o co e mo e complex manipula ion p oblems which equi es spa ial easoning; Page 131 o 151 CHAPTER 6. Conclusions and Fu u e Wo k •build a ailu e-based expe ien ial knowledge ha allows he obo o p e en he epe i ion o i s mis akes. Some o hese wo ks a e al eady in p epa a ion. Thei de elopmen shall make he obo s sma e and mo e adap i e. Page 132 o 151 Appendices 133