Overview of some Command Modes for Human-Robot Interaction Systems

Copy igh © 2022 by Au ho /s and Licensed by IADITI. This is an open access a icle dis ibu ed unde he C ea i e Commons A ibu ion License which pe mi s un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed Jou nal o In o ma ion Sys ems Enginee ing and Managemen 2022, 7(2), 14039 e-ISSN: 2468-4376 h ps://www.jisem-jou nal.com/ Li e a u e Re iew O e iew o some Command Modes o Human-Robo In e ac ion Sys ems Abdelouahab Zaa i1* 1 Uni e si y o Cons an ine-B o he s Men ou i- Cons an ine, Alge ia * Co esponding Au ho : azaa [email protected] Ci a ion: Abdelouahab Zaa i (2022). O e iew o some Command Modes o Human-Robo In e ac ion Sys ems. Jou nal o In o ma ion Sys ems Enginee ing and Managemen , 7 (2), 14039 . h ps://doi.o g/10.55267/iad .07.12011 ARTICLE INFO ABSTRACT Recei ed: 06 Dec. 2021 Accep ed: 18 Feb. 2022 In e ac ion and command modes as well as hei combina ion a e essen ial ea u es o mode n and u u is ic obo ic sys ems in e ac ing wi h human beings in a ious dynamical en i onmen s. This pape p esen s a syn he ic o e iew conce ning he mos command modes used in Human- Robo In e ac ion Sys ems (HRIS). I includes he i s his o ical command modes which a e namely ele-manipula ion, o - line obo p og amming, and adi ional elemen a y eaching by demons a ion. I hen in oduces he mo s ecen command modes which ha e been os e ed la e on by he use o a i icial in elligence echniques implemen ed on mo e powe ul compu e s. In his con ex , we will conside speci ically he ollowing modes: in e ac i e p og amming based on he g aphical-use -in e aces, oice-based, poin ing-on-image-based, ges u e-based, and inally b ain- based commands. Keywo ds: human- obo in e ac ion, eleope a ion, speech-based commands, image-based commands, ges u e- based command, b ain-compu e in e ace. INTRODUCTION In a mode n echnological sense, he i s obo s which appea ed du ing he second wo ld wa we e se ial a m manipula o s. They we e mainly dedica ed o elemanipula e objec s in haza dous en i onmen s such as nuclea plan s and o pe o m simple epe i i e indus ial asks. Du ing his his o ical pe iod, he adi ional modes o command and con ol we e elemanipula ion, o -line obo p og amming, lead- h ough and each pendan (Low, 2006; Wallén, 2008). La e on, since he se en ies, wi h he p og essi e imp o emen o compu e pe o mances, se ial obo ha e p o en o be e y e icien o a ious indus ial asks. Mo eo e , since he eigh ies, pa allel obo s joined he domain o indus y adding p incipally mo e p ecision and speed compa ed o se ial obo s (Cecca elli, 2001; Gaspa e o and Scale a, 2019). Nex , since he nine ies, new shapes o obo ic sys ems appea ed and con inuously e ol ed in s uc u es, unc ionali ies and pe o mances (Zamalloa e al., 2017). Di e en obo shapes a e expe ienced such as mobile obo s, cable-based obo s, lexible obo s like snakes, a ious o he animals-like obo s, d ones, and e en humanoid obo s (Ramos, 2018). I is a ac ha mode n obo s a e becoming popula and p o ed o be capable o achie ing ela i ely complex asks in a ious en i onmen s including unknown, haza dous and/o emo e a eas. Some examples o such sys ems a e space explo a ion obo s as o e s, assis i e obo s o elde ly and disabled people, su ge y obo s, e c. Humanoid obo s a e pa icula ly ema kable because o hei esemblance o human beings. They a e going o be used in social en i onmen s wi h he capabili y o in e ac di ec ly wi h human beings. They will be also used o escue ope a ions, du ing wa s as soldie s, e c (Fo d, 2015). In he cou se o his ex ensi e e olu ion o mode n obo ic sys ems, adap i e and sma Human Robo In e ac ion (HRI) ises una oidably o be essen ial o ensu ing he success o pe o ming complex asks and missions. As a consequence, HRI has eme ged and s ands mo e and mo e as a opic o a pa amoun impo ance. In he same con ex , command modes play a undamen al ole in any dedica ed amewo k o human- obo in e ac ion as hey a e subs an ial ea u es by which communica ion, in e ac ion, coope a ion and expec ed mu ual unde s anding be ween humans and obo s become possible. In ac , his opic is an essen ial key o success ul de elopmen o mode n and u u is ic HRIS (She idan, 2016) and jus i ies he in e es o p esen his o e iew abou he mos used command modes o HRIS. This pape b ie ly p esen s some o he mos ele an HRI echniques ha a e used o gene a e obo commands. I includes elemanipula ion ha e ol ed owa ds ele obo ics, o -line obo p og amming, in e ac i e-based command, oice-based 2 / 11 Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 command, ges u e-based command, poin ing on image-based command, and b ain-based in e ac ion and command. TRADITIONAL MODES OF HRI The e a e ac ually di e en echniques h ough which a human can in e ac wi h and command obo s. Howe e , a he beginning o he eme gence o obo ics, he e we e ew elemen a y in e ac ion and command modes. Teleope a ion Teleope a ion cons i u es he i s mode o in e ac ion and command ha has been employed in eleope a ing obo ic sys ems. I s a ed a ound he 1945 wi h he i s mas e -sla e manipula o designed o manipula e adioac i e ma e ial in ho cells (She idan, 1992). Teleope a ion is a manual con ol pe o med by an ope a o who is usually si ua ed in a local si e con olling emo ely a obo which is si ua ed in i s wo kspace en i onmen . The mos common con igu a ion o eleope a ion is ha o he mas e -sla e whe e he human ope a o (mas e ) di ec ly con ols he emo e obo (sla e) by means o a hand con olle . T adi ionally, he ask moni o ing was assu ed by di ec ision i possible and by came as. Figu e 1 illus a es he gene al o ganiza ion o a eleope a ion sys em p esen ed in (She idan, 1992). On he le side, one can dis inguish he human ope a o (mas e ) who is eleope a ing and supe ising he ask. On he igh side, one can dis inguish he subo dina e (sla e) which is execu ing he ecei ed commands. Howe e , his mode con inuously in ol ed he ope a o ’s a en ion in a di ec coupling. I , hus, gene a ed a igue and a bo ing eeling du ing epe i i e asks, leading o cogni i e a igue o he ope a o . Besides ha , o he d awbacks can a ec eleope a ion such as he weakness o senso y eedback, he limi ed communica ion bandwid h, he ope a o 's subjec i e expe ience and p esen wo k condi ions. The e is also one issue ha e y se iously a ec s and educes he pe o mance o eleope a ion and which is ha d o add ess. I is he con ol ins abili y ha is caused by ime delays o he eedback signal be ween he ope a o and he elemanipula ed obo (Lichia dopol, 2007; She idan, 1992; Zaa i, 2000). Thus, o o e come hese p oblems, i was necessa y o inco po a e a ce ain deg ee o au onomy in o he sys em in o de o elie he ope a o . So, wi h he con inuously inc easing powe p ocessing o compu e s and IA, he de elopmen o adi ional eleope a ion leaded o ele obo ics and o supe iso y con ol sys ems. These las echniques we e de eloped since he se en ies and we e en iched wi h mo e HRI echniques, sma , adap i e and mul imodal use in e aces (Oussalah and Zaa i, 2003; Zaa i and Van B ussel, 1997). The mos common echniques which ha e been used la e on o elie eleope a ion we e: elep esence, elep og amming, semi-au onomous con ol, in elligen assis ance o suppo ope a o s and augmen ed eali y (Lichia dopol, 2007; Makha ae a and Va ol, 2020; W. S. Kim e al., 1992). O -line P og amming Mode Con a ily o elemanipula ion echniques whe e he human ope a o is ully engaged in he in e ac ion wi h he obo ; o o -line p og amming echniques, he ask execu ion is ully managed by an au oma ic con olle so ha he ope a o is le almos in an obse e ole. The o -line obo p og amming echnique is simila o compu e p og amming languages which a e usually used o simula ion. I consis s o w i ing a ex ual p og am (code) ha con ains he speci ica ions o ask execu ion. The p og am is a so wa e which is w i en independen ly o he obo cell (o -line). To execu e he ask by he obo , he ope a o in e ac s wi h a compu e om whe e he uploads he p og am and launches i s execu ion. Once he p og am s a s, he compu e manages au oma ically he ask by means o he obo con olle acco ding o he p og am ins uc ions. In he beginning phase o obo ics, he ope a o is le ou o he ask ope a ions unable o in e ene. Ne e heless, while supe ising he ask execu ion, he/she can only in e ene i necessa y by means o an eme gency bu on o s op he ask execu ion by disabling he powe supply. Con a y o adi ional eleope a ion; ex ual p og amming, while i is ully au oma ic, lacks lexibili y o pe o ming complex asks which equi e some skills. I is no p ac ical in uns uc u ed en i onmen s. Mo eo e , because o he unce ain ies inhe en o modelling eal en i onmen s, obo s and senso s. This mode is no applicable a all in unknown and emo e en i onmen s. I is howe e well adap ed o p og amming simple asks (Mi si e al., 2005; Pan e al., 2012; Yong and Bonney, 1999). O he echniques: eaching by demons a ing Beside eleope a ion and o -line p og amming, he e a e wo o he adi ional me hods ha we e used o acili a e he p og amming o obo s o pe o ming some simple epe i i e bu ela i ely complex manipula ion asks. They a e based on eaching by demons a ion, and hey a e namely he lead h ough me hod and each pendan me hod. "Lead h ough me hod:" is a me hod ha is also e e ed o as hand guidance p og amming me hod. I is an in ui i e me hod ha in ol es he human ope a o o demons a e he ask by pe o ming i manually by guiding he obo end- e ec o . Du ing he demons a ion phase, he ope a o mo es by means o his own hands he obo 's he end-e ec o and guides he obo while pe o ming he ask. The obo con olle s o es he ajec o y p o ided du ing he demons a ion. This enables he obo o play back au oma ically he demons a ed ask du ing he p oduc ion cycle. The walk h ough me hod was usually app op ia e o some ype o asks such as sp ay pain ing and a c welding (A gall e al., 2009; Eakins e al., 2013; Qi and Zhang, 2009). "Teach pendan me hod:" like he lead- h ough me hod, each pendan me hod consis s o eaching he obo how o pe o m he gi en ask, bu by means o a each pendan which is used as an in e ace ool ha se es o guide he obo . The each pendan is a hand-held de ice ha is used o con ol he 3 / 11 Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 obo wi hou con ac and emo ely. I is p o ided in he o m o a po able de ice like a able wi h bu ons, swi ches and dials co esponding o speci ic unc ions designed o con olling he obo . Du ing he eaching o lea ning phase, he ope a o holds and uses he each pendan o d i e he obo s ep by s ep o any desi ed loca ions which a e equi ed o pe o ming he ask. The ea e , he ele an poses a e s o ed, and inally, du ing he p oduc ion cycle, he obo epea s au onomously he mo emen s and ac ions ha lead o pe o m he ask a speci ied speed. (Fukui e al., 2009; Joseph, 1998). The lead h ough and each pendan me hods in ol e he human ope a o in he ask, bu only du ing he eaching phase which is pe o med ou o he p oduc ion cycle. A e lea ning he p ocesses, he obo ac ions a e con e ed in o ex ual p og ams which a e used he ea e o au oma ically pe o m he ask (A gall e al., 2009). INTERACTIVE COMMANDS BASED ON GUI I we conside he p e iously p esen ed adi ional in e ac ion modes, we no ice ha he e was a need o ee he ope a o om his con inuous engagemen du ing ull eleope a ion and in e sely o gi e him he possibili y o in e ene when needed du ing he ull au oma ic ask execu ion o he p og amming mode. To his end, he in e ac i e con ol mode, which has eme ged a ound 1995, has p o ided an al e na i e o he use o in e ene du ing ask execu ion and e en o swi ch om one command mode o ano he . I was os e ed by he appa i ion o so wa e in e acing acili ies. I has been made possible conjoin ly wi h he de elopmen o he objec -o ien ed p og amming echniques and hei co esponding languages such as C++, Ja a, Py hon, e c. Basically, in e ac i e p og amming enables o buil up G aphical Use In e aces (GUI) on compu e sc eens ha con ain g aphical objec s like windows, panels, bu ons, slide s, pop-up menus, e c. Each g aphical objec can be linked o a unc ion ha can be execu ed each ime he use ac i a es he co esponding objec om he GUI. The gene a ion o unc ions can be done by mouse clicks on pa icula widge s, by yping speci ic cha ac e keys o he compu e keyboa d o by sc een ouch (Figu e 2). The GUI enables also he p esen a ion o da a in di e en o ma s and s uc u es ( ex s, images, ideos) (Appels al e al., 2018; Mye s, 1995). F om he HRI poin o iew, he in oduc ion o in e ac i e p og amming echniques was welcomed as i gi es he possibili y o gene a e in e ac i ely commands ia GUI while he obo is e en pe o ming ope a ions wi hou necessa ily s opping i . I enables also he e lec ion o eedback in o ma ion ia he GUI. As shown in Figu e 2, many needed unc ions can be p og ammed and launched ia he GUI. Posi ion commands, join commands, eloci y commands, o ce commands, ei he cons ained o no can be gene a ed in e ac i ely. Wi hin his in e ac i e mode, he ope a o can ca y ou pick-and-place asks wi h obo manipula o s o di ec a mobile obo o some des ina ion. Fo ins ance, o achie e asks wi h his mode, he ope a o di ec s he obo by a se ies o clicks on he app op ia e bu ons. In addi ion, his mode enabled o combine pu e eleope a ion wi h pu e ex ual p og amming, p o iding a way o o e coming hei inhe en limi a ions. These modes o eleope a ion, o -line p og amming and in e ac i e p og amming can be combined and sha ed o pe o m asks mo e e icien ly. This in e ac i e mode has opened many ways o HRI and coope a ion (Ma ion e al., 2017; Zaa i, 2000; Zendoui e al., 2018) Figu e 1. Teleope a ion sys em Figu e 2. In e ac i e commands Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 4 / 11 MODERN COMMAND MODES Mode n command modes make he e olu ion o he ask easie and mo e e icien . Ye , hey equi e mo e complex sys ems o be implemen ed o hey need senso s, ela i ely as compu e s and AI echniques. They need algo i hms ha p ocess da a, ex ac ele an ea u es and impo an in o ma ion as humans do. In he ollowing sec ions, we p esen speech-based command, poin ing on image-based command, ges u e-based command, and b ain-based command. Speech-Based Commands I is well known ha speech is he mos na u al way o communica e be ween humans o he wise alking o obo s a he ea lie age o obo ics was like a d eam. Howe e , la e on wi h he de elopmen o AI, human speech appea s o be an in e es ing mean o command, con ol and communica e wi h obo s. S ill, using human speech o command obo s equi es speech ecogni ion echniques, which s a ed in he ea ly 1950s. O cou se, ea ly sys ems had limi ed ocabula y, bu mode n speech ecogni ion sys ems ha e e ol ed and a e now widely a ailable in many domains. They ha e gained use wi h in elligen assis an s, such as Amazon's Alexa, Apple's Si i and Mic oso 's Co ana, Google’s Google Assis an and o he s (Te zopoulos and Sa a zemi, 2020). These sys ems a e enabling speech in e ac ion wi h compu e s and o he de ices. They ha e capabili ies o in e p e wha is said h ough speech ecogni ion sys ems and e en may espond o ques ions o commands ac oss ex - o-speech sys ems. In pa icula , au oma ic speech unde s anding o gene a ing obo commands is linked o he ield o Au oma ic Speech Recogni ion (ASR). ASR is in ended o con e human speech in o w i en ex s. Mos Speech-based commands and in e ac ion be ween humans and obo s ha e been de eloped wi h he ecogni ion o isola ed wo ds because i is easie o be used as obo commands. Examples o sen ences ha a e used by speech sys ems o command obo s a e: go o ini ial posi ion, ind able, mo e le , e c. Howe e , because o he complex na u e o oice signals, ASR s ill emains in some ci cums ances a ela i ely ha d issue o obo s o unde s and speech commands. The p inciple o speech commands The p inciple used o mos wo d ecogni ion sys ems can be illus a ed in Figu e 3 and Figu e 4. I comp ises wo phases: he lea ning phase and he ecogni ion phase. The lea ning phase consis s o c ea ing a lis o wo ds which a e s o ed in o a dic iona y as e e ence wo ds. The ecogni ion phase consis s o iden i ying any new spoken wo d o one o he e e ence wo ds s o ed in he dic iona y. An example o ASR we ha e implemen ed was based on he ollowing p ocedu e: any spoken wo d which is a con inuous acous ic signal is ansla ed by he mic ophone in o an elec ic con inuous signal. This con inuous elec ical signal is hen sampled by a sound ca d. Some digi al ope a ions a e hen applied such as p e-emphasis, Fas Fou ie T ans o m (FFT), powe spec um, il e bank in eg a ion (Mel's Fil e ), loga i hmic comp ession, Disc ee Fou ie T ans o m. The inal ou pu is a se o coe icien s which a e called Mel F equency Ceps al Coe icien s (MFCC) (Muda e al., 2000). MFCC a e he main ea u es ha cha ac e ize a speech signal. They se e o build he dic iona y o he obo commands ( e e ences) a e a aining phase o he use . They se e also in he ecogni ion phase o iden i y any new unknown obo command by compa ing i o hose s o ed in he dic iona y. In o de o ecognize any spoken wo ds conside ed as possible obo commands, ASR is using a ious me hods o classi ica ion such as Hidden Ma ko Model (Rabine , 1989), VQ ec o quan i ica ion (Linde e al., 1980), and lea ning echniques as Neu al Ne wo ks (Paul and Pa ekh, 2011). Howe e , he Mul iLaye Pe cep on (MLP) is o special impo ance o acous ic modeling in ASR (Pin o, 2010). As an example in one o ou implemen a ion, he ole o he classi ie is played by he MLP (Zaa i e al., 2015). I selec s he closes e e ence wo d wi h espec o he spoken one. The scheme o an ASR is ep esen ed in Figu e 4. Figu e 3. Lea ning phase Figu e 4. Recogni ion phase 5 / 11 Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 Howe e , speech-based command is in e es ing o handicapped people using o ins ance wheelchai s id he ope a o ’s hands canno be used o i hey a e busy wi h some o he asks. Wi h he ecen de elopmen o pe sonal and humanoid obo s, his mode is also gaining mo e in e es . Ne e heless, ARS applied o obo ic domain s ill su e s om many ba ie s. I is sensi i e o ha d noisy en i onmen s. The esul s depend on he pe sonal cha ac e is ic o he ope a o ha can only be imp o ed by a long aining. E icien success can be ob ained wi h limi ed ocabula y and canno ma ch all and new con ex s (Oussalah and Zaa i, 2003; Zaa i e al., 2015). Commands by Poin ing on Images (Look- hen-Mo e) The use o ision o command and con ol obo s is a e y impo an elemen o enhancing obo applica ions. Today, many comme cial obo ics in eg a ing ision sys ems a e a ailable. Bu because o he sensi i i y and complexi y o image p ocessing, he e iciency o such sys ems is mainly limi ed o s a ic indus ial applica ions. Fo ele obo ics, ision-based con ol echniques, as s essed in (She idan, 1992), make a powe ul se o ools o obo in e ac ion and con ol especially in uns uc u ed and unknown en i onmen s. These echniques do no equi e p e- knowledge o models abou he objec s o wo k wi h. Wi h hese echniques, he human ope a o designa es he objec s o loca ions o in e es om he ecei ed images; a e wa ds he sys em ex ac s hei co esponding ea u es and heads owa ds hem in eal space. In all hese echniques, he supe iso ’s ole, as obse ed in (She idan, 1992) “is limi ed o concep ion and poin ing, and he ele- obo do he es ”. In gene al, ision-based obo command sys ems can be di ided in wo main olds ega ding he applied con ol sys em a chi ec u e. The ea lie echnique is named look- and-mo e and he o he one is isual se oing. Image-Based as Look- hen-Mo e Commands Because eal- ime image p ocessing was ela i ely slow, he ea lie ision-based sys ems we e p incipally designed o applica ions o ype "look- hen-mo e" which a e usually dedica ed o s a ic en i onmen s. They we e also e e ed some imes as poin -and-click o image-based commands (Kim and S a k, 1989; Oussalah and Zaa i, 2003). F om he con ol sys em iewpoin , hei con ol a chi ec u e is an open loop one wi hou con inuous image p ocessing eedback. These sys ems include s e eo ision sys ems o cap u e images. They wo k as ollows (Figu e 5). A scene is selec ed by he ope a o which inside which he objec o in e es appea s in bo h s e eo ision came as. The ope a o selec s his objec in one image by poin ing on i wi h a poin ing de ice such as a mouse click. The s e eo ision sys em g abs au oma ically he le and igh images. Then a s e eo ision algo i hm p oceeds hese images in o de o ex ac he co esponding 3D coo dina es o he selec ed objec o in e es in he eal wo d. Once, hese 3D coo dina es a e es ima ed, hey a e sen as a posi ion command o he obo o heading owa ds his loca ion o his objec . In ac , he obo execu es his command by pe o ming “blind” mo emen s which assumes ha he en i onmen emains s a ic a e he obo has s a ed o mo e. I is an open-loop app oach o a ype i e and o ge (Wang, 2016) . By epea ing his p ocess, he ope a o can di ec easily he obo o pe o m manipula ion asks and/o na iga ion. Howe e , while his echnique is simple, i enables high le el commands. I ep esen s an e icien way o gene a e look- hen-mo e obo commands o s a ic en i onmen s. A success ul implemen a ion o an au oma ic image-based (look- hen-mo e) commands named click-and-mo e, has been de eloped o six DOF manipula o s. I has been desc ibed in (Oussalah and Zaa i, 2003; Zaa i and Van B ussel, 1997). I has also been ex ended o mobile manipula o s (Zaa i, 2000) and o pa allel cable-based obo s (Bouchemal and Zaa i, 2014). All ou expe imen s ha e p o en ha i is a e y powe ul echnique o a ious ypes o obo s in he ange whe e calib a ion o he came as has been pe o med. Visual Se oing Con ol Wi h he inc easing powe o image p ocessing, i becomes possible o apply machine ision o dynamic sys ems ac ing o mo ing in a non-s a ic en i onmen and capable o acking mo ing a ge s. This echnique, known as ision-based con ol o isual se oing appea ed in 1979 (Co ke, 1994; Hans, 2018). Visual se oing is dis inguished om look- hen-mo e since i enables o con ol he obo 's mo ion using eal- ime eedback based on ision senso s (Vah enkamp e al., 2008). Ges u al-based Commands Ges u e-based obo command is a echnique ha uses he mo emen o some body pa s in o de o in e ac and command obo s. Ges u es can be o any ype o body mo emen s: head mo emen , hand ges u e, anybody mo emen and e en acial exp essions. Also, ools mo ed by a human ope a o such as pencils, lags, s icks, e c. can be used o in e ac and command obo s. As he many mode n modes o obo command, ges u e- based command has been s udied, designed and implemen ed by many au ho s. A su ey conce ning ges u e-based in e ac ion is p esen ed in (Gal án-Ruiz e al., 2020; S. Mi a and T. Acha ya, 2007). Technically, in mos applica ions, ges u e-based in e ac ion equi es de ec ion and iden i ica ion o ges u es as in ended obo commands. Ac ually, de ec ion and iden i ica ion o ges u es a e mainly based on objec ecogni ion and acking app oaches in ol ing CCD came a senso s. As o isual se oing, se e al image p ocessing me hods a e used o image ecogni ion such as ea u es-based, appea ance-based, g adien -based, lea ning me hods, e c. (Bake and Ma hews, 2004; Bonci e al., 2021; Lucas and Kanade, 1981; Nea chou, 2011). P inciple o Ges u e-Based commands The p inciple o ges u e-based in e ac ion sys ems is simple. I equi es senso s ha cap u e sequences o images du ing he mo emen o a human body pa o o any ool mo ed by he ope a o . These sequences o images a e hen analyzed by an image p ocessing so wa e in o de o ack he mo ion o some de ec ed elemen s o in e es . Once a signi ican Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 6 / 11 mo emen is de ec ed and acked; he inal con igu a ion o he ges u e is iden i ied. I is in e p e ed wi h espec o a code ha links de ec ed mo emen s wi h co esponding obo commands. This iden i ica ion o he ges u e is inally sen as a co esponding obo command. Las ly, he obo pe o ms he in ended ac ion o ask. Figu e 6 summa izes he p inciple o ges u e-based obo command showing he sequence o he designed and implemen ed main ope a ions (Zaa i, 2021). Applica ions o ges u e-based obo command a e nume ous and can be adap ed and ex ended acco ding o many needs and con ex s. Ges u e exp essions can be employed in egions whe e he speech is useless. This can happen, o ins ance, in noisy places, in unde wa e a eas, and in emp y space whe e he medium canno con oy he oice wa es as wi h as onau s (Liu e al., 2016). I can be also used o lea ning by demons a ion and o imi a ion as ep oducing ope a ions in medical ca e o ele-su ge y (S aub e al., 2011). This command mode is also in e es ing in some mili a y ac i i y o communica e and di ec emo e eams, au onomous and unmanned sys ems (Ellio e al., 2016). Mo eo e , i can be used in assis i e obo ics o supe ising dea people, o su eillance o disabled people, e c. (Bouchemal and Zaa i, 2013). In addi ion, he ges u e-based in e ac ion can be designed wi h con ac o wi hou con ac . Examples o ges u e in e ac ion wi h con ac a e used o eaching on blackboa ds, ables and o he suppo s. In hese si ua ions, ma ke s and colou s can be employed o acili a e he acking o elemen s o in e es (Bouchemal and Zaa i, 2013; Nea chou, 2011; Sigalas e al., 2010). Mo eo e , wi h he e en o Co id-19 pandemic, ges u e-based con ol wi hou con ac is gaining a special impo ance by a oiding ouch and con ac wi h con amina ed people and objec s like doo handles, machines, e c. This command mode o e s he ad an age o eeing he ope a o om he con ac wi h he compu e . Ne e heless, some di icul ies which a e ela ed o image p ocessing and en i onmen al issues can limi he capabili ies and pe o mances o his command mode. BRAIN-BASED INTERACTION AND COMMANDS Beside he a ie y o exis ing human-machine in e ac ion and command echniques, hese las decades, biological ones ha e opened new unp eceden ed pe spec i es owa ds e y in e es ing and p omising inno a i e esea ches. Indeed, bio- mechanical ac ions, myo-elec ic and bio-cybe ne ic echniques can be used o in e ac ion, command and con ol o dynamical sys ems such as obo s, wheelchai s, and e en pa alyzed body pa s o human (Be na-Ma inez, 2011; Ma inek e al., 2021; Rechy-Rami ez and Hu, 2015). In his con ex , he mas e ing o he b ain ac i i y o human beings o in e ac ing and commanding sys ems e eals o be a e y a ac i e opic. Indeed, b ain ac i i y is he p ime sou ce o gene a ing command signals. In he pas , unde s anding mind hough s and de ec ing in en ions we e conside ed as a kind o elepa hic capabili y which only exis s in science ic ion. Howe e , wi h ecen ad ances in neu o-sensing echnologies, his belie is inally u ning in o eali y (Nam e al., 2018). As a esul , he ield o human- obo in e ac ion has been signi ican ly en iched by he b ain-based in e ac ion and command echniques. I is ac ually possible o a subjec , by means o B ain Compu e In e aces (BCI), o de ec o gene a e commands ha can be used o manipula e i ual o eal sys ems acco ding o in en ions h ough he b ain ac i i y (Nicolas-Alonso and Gomez-Gil, 2012). Figu e 5. Look hen mo e commands Figu e 6. Look hen mo e commands 7 / 11 Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 P inciple o B ain-Compu e In e aces C ow I is well es ablished ha human b ain ac i i y consis s o emi ing wa es wi h ce ain pa e ns as a esul o ex e nal s imuli o men al s a es (Buzsaki, 2006; Doelling and Assaneo, 2021). BCIs a e speci ic de ices which a e capable o cap u ing and measu ing b ain ac i i y, hen ex ac ing in e es ing ea u es, and inally con e ing hese ea u es in o ou pu s o in e ac wi h and command dynamic sys ems. Thus, BCIs enable use s o in e ac wi h compu e s and o he de ices by means o b ain ac i i y only (A onso e al., 2014; A onso, 2013; Yonck, 2002). They p o ide a unique way o communica ion be ween a human and a machine (o de ice) wi hou any neu omuscula in e en ion (Abi i e al., 2017; Wolpaw e al., 2002). Mo e gene ally, B ain–Machine In e aces (BMIs) a e de ices ha ansla e neu onal in o ma ion in o commands capable o con olling ex e nal so wa e o ha dwa e such as compu e s, wheelchai s, p os hesis, and obo s. Figu e 7 shows a scheme o a BCI in e acing and con e ing b ain ac i i y o a use in o command signals wi h possible eedback (do ed lines) o he use . The design p inciple o BCIs is simila o mos o he in e aces such as speech-based, image-based, ges u e-based ones. I consis s o con e ing basic signals in o commands by means o a se o p ocessing ope a ions: de ec ing he p ima y use 's signals, p ocessing i in o de o ex ac ele an ea u es, classi ying he ob ained ea u es in o de o decide o which p ede ined command i is supposed o ha e been issued; hen his command is sen o execu ion (see Figu e 8). BCI echniques Di e en echniques a e used o measu e b ain ac i i y o BCIs. Mos BCIs use elec ical signals which a e de ec ed using senso s placed in asi ely o non-in asi ely. The e a e se e al echniques o nonin asi e BCI, such as EEG (elec oencephalog aphy), MEG (magne oencephalog aphy), o MRT ( unc ional magne ic esonance imaging: omog aphy) (Fe ei a e al., 2008). Elec oencephalog aphy (EEG) is a physiological me hod o eco d he elec ical ac i i y gene a ed by he b ain ia elec odes placed on he scalp su ace. The signal ampli ude is usually unde 100 μV and he equency band o no mal EEG signals is usually abo e DC up o 50 Hz (Ma inek e al., 2021). In in asi e echniques, special de ices a e inse ed di ec ly in o he human b ain by su ge y. In Semi-in asi e, de ices a e inse ed in o he skull on he op o he human b ain, o di ec ly on he co ex (called Elec oco icog aphy – ECoG); he su ace o he b ain (signal ha ing abou 1- 2 mV o ampli ude (Fe ei a e al., 2008). In gene al, non-in asi e echniques a e conside ed as sa es , o low-cos ype o de ices and hen easies o s udies. Howe e , he cap u ed a human b ain signals a e weake compa ed o in asi e echniques which a e in di ec con ac s wi h neu al cells (Fe ei a e al., 2008; S ey l e al., 2016). Applica ions and Pe spec i es o BCIs F om he beginning, mos applica ions o he biological and physiological con ol echniques we e o ien ed owa ds he assis ance o handicapped and disabled people. The p og ess in de eloping BCIs is p o iding a lo o hope o his communi y, especially o hose wi hou muscula capabili y. Wha is ema kable conce ning he BCIs is ha mos applica ions dedica ed o medical applica ions can in ol e he use a he same ime as a commande and as a subjec . A e a aining pe iod, he use can lea n how o gene a e commands by hough s, o con ol and ac i a e p os hesis membe s o his own body pa s such as pa alyzed membe s (Baniqued e al., 2021; Mane e al., 2020). Figu e 9 p esen an ope a o wea ing a BCI o con olling a mechanical sys em. Figu e 7. BCI in e acing b ain ac i i y o sys ems Figu e 8. Global o ganiza ion o BCIs Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 8 / 11 Figu e 9. Ope a ion wi h BCI commanding a mechanical sys em BRIEF SUMMARY ABOUT AI AND HRI Since he eme gence o he i s heo ies and app oaches o AI, he e was a hope o in use hese echniques in pa icula o obo s o imp o ing hei adap i i y and sma ness. The low o echniques p o ided by AI such as symbolic compu a ions, gene ic algo i hms, neu al ne s, uzzy logic, and machine lea ning, has se ed and s ill se es o a emp ing o sol e and o e come many issues ha limi obo ic applica ions. Ac ually, AI has ed obo s by nume ous capabili ies wi h mo e o less sa is ying success. Some ema kable ields a e speech unde s anding, objec ecogni ion based on ision, au onomous na iga ion, manipula ion asks, in e ac ions wi h humans and o he en i ies, coope a ion, and so on (Pe ez e al., 2018; Seme a o e al., 2021). Bu ollowing he inc easing complexi y o obo ic sys ems designed o au onomously pe o m asks and missions in a ious en i onmen s; speci ic con ol a chi ec u es o in elligen sys ems equi ing high le el lexibili y and adap i i y was p oposed and implemen ed. One popula example o such a chi ec u e is he h ee laye ed one which combines e lexi e and eac i e beha io s as well as delibe a i e capabili ies (A kin, 1998, p. 199; R. B ooks, 1986; S. Ga cıa e al., 2018). Howe e , conce ning he con ibu ion o AI o he pa icula domain o HRI, some au ho s ha e analyzed and discussed many ele an applica ions, challenging issues and po en ial p omising pe spec i es (Feil-Sei e and Ma a ic, 2009; Lemaignan e al., 2017; Seme a o e al., 2021; She idan, 2016). In ac , he mos challenging goal o AI ela ed o HRI is s ill how o design obo s ha can unde s and and in e p e human beha io s and in en ions in dynamic and possibly unp edic ed si ua ions in o de o ake app op ia e decisions? To his end, con inuous e o s a e spen on esea ching echniques and designing join cogni i e a chi ec u es o imp o e HRI by ying o allow obo s gaining mo e knowledge and au onomous de elopmen as human beings do. E o s a e o ien ed o de eloping obo mechanisms o enable de elopmen al lea ning and capabili y o acqui ing skills (Lemaignan e al., 2017; Nicolescu and Ma a ic, 2005; Seme a o e al., 2021). Beside supe ised and unsupe ised lea ning, ein o cemen lea ning, deep lea ning; de elopmen al au onomous lea ning is a e y impo an and challenging app oach. I is a kind o a sel -lea ning compe ence whe e he obo a emp o acqui e knowledge and skills by con inuous obse a ion and imi a ion o people and o he in elligen sys ems. As wi h humans, i equi es obse ing e en s and si ua ions, classi ying and o ganizing hem, ex ac ing ele an ea u es and in e ing ules and unde s anding si ua ions and pa ne 's in en ions (A en s and G ei ans, 2022; Nicolescu and Ma a ic, 2005). Among he issues and pe spec i es is he need o make mo e na u al he in e ac ion and he unde s anding be ween humans and obo s as be ween humans hemsel es. This eques cons i u es also one main objec i e o he 4 h indus ial gene a ion which in ends o es ablish na u al coope a ion and collabo a ion be ween humans and sma obo s named some imes "cobo s" in o de o achie e common asks and missions (A en s and G ei ans, 2022; Chak abo i e al., 2017; Ja aid and Khan, 2021; Zamalloa e al., 2017). Mo eo e , he ac ual and u u is de elopmen o di e en kind o in elligen obo s and hei a ious possible in e ac ions wi h humans is p essing owa ds a homogeneous ep esen a ion o Human-Robo con ol a chi ec u es. The e is a se ious need o de elop a uni ied con ol a chi ec u e o HRIS ha ele a es he cogni i e compe encies o he obo s o app oxima e he le el o humans and elimina es he e o e he dis inc ion be ween humans and a i icial in elligen sys ems when wo king oge he as eam membe s (Ha io and Adams, 2013; K äme e al., 2012; Zaa i, 2021). CONCLUSION This pape has b ie ly desc ibed he mos command modes in ol ed in HRIS. This includes he adi ional in e ac ion and command modes which a e namely ele-manipula ion, o -line obo compu e p og amming and lea ning by demons a ion (lead- h ough and each pendan ). I hen in oduces he modes which ha e been os e ed la e on by he conjunc ion o obo ics wi h he eme gence o a i icial in elligence echniques and he p o ision o powe ul compu ing machines. The ollowing modes we e conside ed: in e ac i e commands based on GUI, oice-based commands, poin ing on image-based commands, ges u e-based commands, and inally b ain-based commands. In addi ion, some ele an and challenging issues co esponding o some command modes ha e been b ie ly discussed. One can no ice ha he gene a ion o obo commands ollows almos he same p ocess: de ec ing he in ended command signal gene a ed by he ope a o by means o speci ic senso s; hen analyzing and iden i ying his command w. . a dic iona y; and inally o de ing he obo o execu ing his command. The cons i u ion o he dic iona y ha con ains he e e ences a e usually cons i u ed h ough a lea ning p ocess. I s pe spec i es on he sho and mean e ms, he e is a need o imp o e he command modes o make hem mo e lexible and use iendly. These modes can also be combined unde a mul imodal ope a o in e ace o p o ide mo e lexible in e ac ion and con ol o obo s. The e is also a need o use echniques o a i icial in elligence o enable a human- obo coope a ion and coo dina ion as pa ne s. On he o he hand, 9 / 11 Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039 BCI is aking mo e a en ion o he capabili y i o e s o con olling i ual as well as eal dynamic sys ems. P obably, he mos impo an domain in he nea u u e is he applica ions o BCI in o de o help handicapped and disabled people using hei pa alyzed limbs, wheelchai s, p os hesis, and se ice obo s. REFERENCES Abi i, R., Heise, G., Zhao, X., Jiang, Y., Abi i, F.A., 2017. B ain compu e in e ace o ges u e con ol o a social obo : An o line s udy. 2017 I anian Con e ence on Elec ical Enginee ing (ICEE) 113–117. A onso, A., Angélico Gonçal es, M., Lima, J., Co a, M., 2014. UsaWeb. A model o usabili y e alua ion web in e aces, in: Ibe ian Con e ence on In o ma ion Sys ems and Technologies, CISTI. P esen ed a he Ibe ian Con e ence on In o ma ion Sys ems and Technologies, CISTI, p. 6. h ps://doi.o g/10.1109/CISTI.2014.6876952 A onso, A.P., 2013. P oyec o de E aluación de In e aces — PAI,. Uni e sidad de Vigo, Depa amen o de In o má ica., Vigo. Appels al, M., Michalak, J., Os e be g, M., 2018. Easy o Use G aphical Use In e ace o Robo P og amming. A en s, J., G ei ans, M., 2022. Sma Indus ial Robo Con ol T ends, Challenges and Oppo uni ies wi hin Manu ac u ing. Applied Sciences 12. h ps://doi.o g/10.3390/app12020937 A gall, B.D., Che no a, S., Veloso, M., B owning, B., 2009. A su ey o obo lea ning om demons a ion. Robo ics and Au onomous Sys ems 57, 469–483. h ps://doi.o g/10.1016/j. obo .2008.10.024 A kin, R.C., 1998. Beha io -based obo ics. MIT p ess. Bake , S., Ma hews, I., 2004. Lucas-Kanade 20 Yea s On: A Uni ying F amewo k. In . J. Compu . Vision. Baniqued, P., S anye , E., Awais, M., Alazmani, A., Jackson, A., Mon-Williams, M., Mush aq, F., Hol , R., 2021. B ain- Compu e In e ace Robo ics o Hand Rehabili a ion A e S oke: A Sys ema ic Re iew. Jou nal o Neu oEnginee ing and Rehabili a ion 18. h ps://doi.o g/10.1186/s12984-021-00820-8 Be na-Ma inez, J., 2011. Robo ic Con ol Based On The Human Ne ous Sys em. In e na ional Jou nal o A i icial In elligence & Applica ions 2, 107. h ps://doi.o g/10.5121/ijaia.2011.2409 Bonci, A., Fio i, S., Higashi, H., Tanaka, T., Ve dini, F., 2021. An In oduc o y Tu o ial on B ain–Compu e In e aces and Thei Applica ions. Elec onics 10, 560. h ps://doi.o g/10.3390/elec onics10050560 Bouchemal, B., Zaa i, A., 2014. Image-based con ol o cable- based obo s. In e na ional Jou nal o Con ol, Au oma ion and Sys ems 12, 118–125. h ps://doi.o g/10.1007/s12555-012-0445-y Bouchemal, B., Zaa i, A., 2013. Ges u al and Image-Based Con ol Combina ion o ehabili a ion applica ions using cable-based obo s. P esen ed a he In e na ional Con e ence on Technology o Helping People wi h Special Needs, Riyadh, Kingdom o Saudi A abia. Buzsaki, G., 2006. Rhy hms o he B ain. Ox o d uni e si y p ess. Cecca elli, M., 2001. A His o ical Pe spec i e o Robo ics Towa d he Fu u e. Jou nal o Robo ics and Mecha onics 13, 299– 313. h ps://doi.o g/10.20965/j m.2001.p0299 Chak abo i, T., Kambhampa i, S., Scheu z, M., Zhang, Y., 2017. AI challenges in human- obo cogni i e eaming. a Xi p ep in a Xi :1707.04775. Co ke, P., 1994. High-Pe o mance Visual Closed-Loop Robo Con ol. Doelling, K., Assaneo, M., 2021. Neu al oscilla ions a e a s a owa d unde s anding b ain ac i i y a he han he end. PLoS biology 19, e3001234. h ps://doi.o g/10.1371/jou nal.pbio.3001234 Eakins, W., Rossano, G., Fuhlb igge, T., 2013. Lead h ough Robo Teaching. P esen ed a he IEEE Con e ence Technology P ac ice Robo Applica ion, pp. 1–4. Ellio , L.R., Hill, S.G., Ba nes, M.J., 2016. Ges u e-Based Con ols o Robo s: O e iew and Implica ions o Use by Soldie s. P esen ed a he Human Resea ch and Enginee ing Di ec o a e, ARL, US A my Resea ch Labo a o y, ARL-TR-7715. Feil-Sei e , D., Ma a ic, M., 2009. Human-Robo In e ac ion, in: Jou nal Abb e ia ion: Encyclopedia o Complexi y and Sys ems Science. Sp inge , pp. 4643–4659. Fe ei a, A., Celes e, W., aua cheein, F., Bas os, T., Sa cinelli- Filho, M., Ca elli, R., 2008. Human-Machine in e aces base on EMG and EEG applied o obo ic sys ems. Jou nal o neu oenginee ing and ehabili a ion 5, 10. h ps://doi.o g/10.1186/1743-0003-5-10 Fo d, M., 2015. Rise o he Robo s Technology and he Th ea o a Jobless Fu u e. Basic Books, New Yo k, NY, US. Fukui, H., Yonejima, S., Yamano, M., Dohi, M., Yamada, M., Nishiki, T., 2009. De elopmen o eaching pendan op imized o obo applica ion. 2009 IEEE Wo kshop on Ad anced Robo ics and i s Social Impac s 72–77. Gal án-Ruiz, J., T a ieso, C., Teje a-Fe milch, A., Pinan- Roesche , A., Es eban-He nández, L., Dominguez Quin ana, L., 2020. Pe spec i e and E olu ion o Ges u e Recogni ion o Sign Language: A Re iew. Senso s 20, 3571. h ps://doi.o g/10.3390/s20123571 Gaspa e o, A., Scale a, L., 2019. A B ie His o y o Indus ial Robo ics in he 20 h Cen u y. Ad ances in His o ical S udies 08, 24–35. h ps://doi.o g/10.4236/ahs.2019.81002 Hans, S., 2018. A B ie Compa a i e S udy o Visual Se oing Sys ems. IJSRD - In e na ional Jou nal o Scien i ic Resea ch & De elopmen 6, 2321–0613. Ha io , C., Adams, J., 2013. Modeling Human Pe o mance o