Copy igh © 2022 by Au ho /s and Licensed by IADITI. This is an open access a icle dis ibu ed unde he C ea i e Commons A ibu ion License which pe mi s un es ic ed use,
dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal wo k is p ope ly ci ed
Jou nal o In o ma ion Sys ems Enginee ing and Managemen
2022, 7(2), 14039
e-ISSN:
2468-4376
h ps://www.jisem-jou nal.com/
Li e a u e Re iew
O e iew o some Command Modes o
Human-Robo In e ac ion Sys ems
Abdelouahab Zaa i1*
1 Uni e si y o Cons an ine-B o he s Men ou i- Cons an ine, Alge ia
*
Co esponding Au ho :
azaa [email protected]
Ci a ion: Abdelouahab Zaa i (2022). O e iew o some Command Modes o Human-Robo In e ac ion Sys ems. Jou nal o In o ma ion Sys ems
Enginee ing
and Managemen , 7
(2),
14039
.
h ps://doi.o g/10.55267/iad .07.12011
ARTICLE INFO
ABSTRACT
Recei ed: 06 Dec. 2021
Accep ed: 18 Feb. 2022
In e ac ion and command modes as well as hei combina ion a e essen ial ea u es o mode n and u u is ic
obo ic sys ems in e ac ing wi h human beings in a ious dynamical en i onmen s. This pape p esen s a
syn he ic o e iew conce ning he mos command modes used in Human-
Robo In e ac ion Sys ems (HRIS). I
includes he i s his o ical command modes which a e namely ele-manipula ion, o -
line obo p og amming,
and adi ional elemen a y eaching by demons a ion. I hen in oduces he mo
s ecen command modes which
ha e been os e ed la e on by he use o a i icial in elligence echniques implemen ed on mo e powe ul
compu e s. In his con ex , we will conside speci ically he ollowing modes: in e ac i e p og amming based on
he g aphical-use -in e aces, oice-based, poin ing-on-image-based, ges u e-based, and inally b ain-
based
commands.
Keywo ds: human- obo in e ac ion, eleope a ion, speech-based commands, image-based commands, ges u e-
based command, b ain-compu e in e ace.
INTRODUCTION
In a mode n echnological sense, he i s obo s which
appea ed du ing he second wo ld wa we e se ial a m
manipula o s. They we e mainly dedica ed o elemanipula e
objec s in haza dous en i onmen s such as nuclea plan s and
o pe o m simple epe i i e indus ial asks. Du ing his
his o ical pe iod, he adi ional modes o command and
con ol we e elemanipula ion, o -line obo p og amming,
lead- h ough and each pendan (Low, 2006; Wallén, 2008).
La e on, since he se en ies, wi h he p og essi e
imp o emen o compu e pe o mances, se ial obo ha e
p o en o be e y e icien o a ious indus ial asks.
Mo eo e , since he eigh ies, pa allel obo s joined he domain
o indus y adding p incipally mo e p ecision and speed
compa ed o se ial obo s (Cecca elli, 2001; Gaspa e o and
Scale a, 2019).
Nex , since he nine ies, new shapes o obo ic sys ems
appea ed and con inuously e ol ed in s uc u es,
unc ionali ies and pe o mances (Zamalloa e al., 2017).
Di e en obo shapes a e expe ienced such as mobile obo s,
cable-based obo s, lexible obo s like snakes, a ious o he
animals-like obo s, d ones, and e en humanoid obo s
(Ramos, 2018). I is a ac ha mode n obo s a e becoming
popula and p o ed o be capable o achie ing ela i ely
complex asks in a ious en i onmen s including unknown,
haza dous and/o emo e a eas. Some examples o such sys ems
a e space explo a ion obo s as o e s, assis i e obo s o elde ly
and disabled people, su ge y obo s, e c. Humanoid obo s a e
pa icula ly ema kable because o hei esemblance o human
beings. They a e going o be used in social en i onmen s wi h he
capabili y o in e ac di ec ly wi h human beings. They will be
also used o escue ope a ions, du ing wa s as soldie s, e c
(Fo d, 2015).
In he cou se o his ex ensi e e olu ion o mode n obo ic
sys ems, adap i e and sma Human Robo In e ac ion (HRI)
ises una oidably o be essen ial o ensu ing he success o
pe o ming complex asks and missions. As a consequence, HRI
has eme ged and s ands mo e and mo e as a opic o a
pa amoun impo ance. In he same con ex , command modes
play a undamen al ole in any dedica ed amewo k o human-
obo in e ac ion as hey a e subs an ial ea u es by which
communica ion, in e ac ion, coope a ion and expec ed mu ual
unde s anding be ween humans and obo s become possible. In
ac , his opic is an essen ial key o success ul de elopmen o
mode n and u u is ic HRIS (She idan, 2016) and jus i ies he
in e es o p esen his o e iew abou he mos used command
modes o HRIS.
This pape b ie ly p esen s some o he mos ele an HRI
echniques ha a e used o gene a e obo commands. I includes
elemanipula ion ha e ol ed owa ds ele obo ics, o -line
obo p og amming, in e ac i e-based command, oice-based
2 / 11
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
command, ges u e-based command, poin ing on image-based command, and b ain-based in e ac ion and command.
TRADITIONAL MODES OF HRI
The e a e ac ually di e en echniques h ough which a
human can in e ac wi h and command obo s. Howe e , a he
beginning o he eme gence o obo ics, he e we e ew
elemen a y in e ac ion and command modes.
Teleope a ion
Teleope a ion cons i u es he i s mode o in e ac ion
and command ha has been employed in eleope a ing
obo ic sys ems. I s a ed a ound he 1945 wi h he i s
mas e -sla e manipula o designed o manipula e
adioac i e ma e ial in ho cells (She idan, 1992).
Teleope a ion is a manual con ol pe o med by an ope a o
who is usually si ua ed in a local si e con olling emo ely a
obo which is si ua ed in i s wo kspace en i onmen . The
mos common con igu a ion o eleope a ion is ha o he
mas e -sla e whe e he human ope a o (mas e ) di ec ly
con ols he emo e obo (sla e) by means o a hand
con olle . T adi ionally, he ask moni o ing was assu ed by
di ec ision i possible and by came as. Figu e 1 illus a es
he gene al o ganiza ion o a eleope a ion sys em p esen ed
in (She idan, 1992). On he le side, one can dis inguish he
human ope a o (mas e ) who is eleope a ing and
supe ising he ask. On he igh side, one can dis inguish
he subo dina e (sla e) which is execu ing he ecei ed
commands.
Howe e , his mode con inuously in ol ed he
ope a o ’s a en ion in a di ec coupling. I , hus, gene a ed
a igue and a bo ing eeling du ing epe i i e asks, leading
o cogni i e a igue o he ope a o . Besides ha , o he
d awbacks can a ec eleope a ion such as he weakness o
senso y eedback, he limi ed communica ion bandwid h, he
ope a o 's subjec i e expe ience and p esen wo k
condi ions. The e is also one issue ha e y se iously a ec s
and educes he pe o mance o eleope a ion and which is
ha d o add ess. I is he con ol ins abili y ha is caused by
ime delays o he eedback signal be ween he ope a o and
he elemanipula ed obo (Lichia dopol, 2007; She idan,
1992; Zaa i, 2000).
Thus, o o e come hese p oblems, i was necessa y o
inco po a e a ce ain deg ee o au onomy in o he sys em in
o de o elie he ope a o . So, wi h he con inuously
inc easing powe p ocessing o compu e s and IA, he
de elopmen o adi ional eleope a ion leaded o
ele obo ics and o supe iso y con ol sys ems. These las
echniques we e de eloped since he se en ies and we e
en iched wi h mo e HRI echniques, sma , adap i e and
mul imodal use in e aces (Oussalah and Zaa i, 2003; Zaa i
and Van B ussel, 1997). The mos common echniques which
ha e been used la e on o elie eleope a ion we e:
elep esence, elep og amming, semi-au onomous con ol,
in elligen assis ance o suppo ope a o s and augmen ed
eali y (Lichia dopol, 2007; Makha ae a and Va ol, 2020; W.
S. Kim e al., 1992).
O -line P og amming Mode
Con a ily o elemanipula ion echniques whe e he
human ope a o is ully engaged in he in e ac ion wi h he
obo ; o o -line p og amming echniques, he ask execu ion
is ully managed by an au oma ic con olle so ha he ope a o
is le almos in an obse e ole. The o -line obo
p og amming echnique is simila o compu e p og amming
languages which a e usually used o simula ion. I consis s o
w i ing a ex ual p og am (code) ha con ains he
speci ica ions o ask execu ion. The p og am is a so wa e
which is w i en independen ly o he obo cell (o -line). To
execu e he ask by he obo , he ope a o in e ac s wi h a
compu e om whe e he uploads he p og am and launches i s
execu ion. Once he p og am s a s, he compu e manages
au oma ically he ask by means o he obo con olle
acco ding o he p og am ins uc ions. In he beginning phase
o obo ics, he ope a o is le ou o he ask ope a ions unable
o in e ene. Ne e heless, while supe ising he ask
execu ion, he/she can only in e ene i necessa y by means o
an eme gency bu on o s op he ask execu ion by disabling he
powe supply.
Con a y o adi ional eleope a ion; ex ual
p og amming, while i is ully au oma ic, lacks lexibili y o
pe o ming complex asks which equi e some skills. I is no
p ac ical in uns uc u ed en i onmen s. Mo eo e , because o
he unce ain ies inhe en o modelling eal en i onmen s,
obo s and senso s. This mode is no applicable a all in
unknown and emo e en i onmen s. I is howe e well
adap ed o p og amming simple asks (Mi si e al., 2005; Pan
e al., 2012; Yong and Bonney, 1999).
O he echniques: eaching by demons a ing
Beside eleope a ion and o -line p og amming, he e a e
wo o he adi ional me hods ha we e used o acili a e he
p og amming o obo s o pe o ming some simple epe i i e
bu ela i ely complex manipula ion asks. They a e based on
eaching by demons a ion, and hey a e namely he lead
h ough me hod and each pendan me hod.
"Lead h ough me hod:" is a me hod ha is also e e ed o
as hand guidance p og amming me hod. I is an in ui i e
me hod ha in ol es he human ope a o o demons a e he
ask by pe o ming i manually by guiding he obo end-
e ec o . Du ing he demons a ion phase, he ope a o mo es
by means o his own hands he obo 's he end-e ec o and
guides he obo while pe o ming he ask. The obo
con olle s o es he ajec o y p o ided du ing he
demons a ion. This enables he obo o play back
au oma ically he demons a ed ask du ing he p oduc ion
cycle. The walk h ough me hod was usually app op ia e o
some ype o asks such as sp ay pain ing and a c welding
(A gall e al., 2009; Eakins e al., 2013; Qi and Zhang, 2009).
"Teach pendan me hod:" like he lead- h ough me hod,
each pendan me hod consis s o eaching he obo how o
pe o m he gi en ask, bu by means o a each pendan which
is used as an in e ace ool ha se es o guide he obo . The
each pendan is a hand-held de ice ha is used o con ol he
3 / 11
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
obo wi hou con ac and emo ely. I is p o ided in he o m
o a po able de ice like a able wi h bu ons, swi ches and
dials co esponding o speci ic unc ions designed o
con olling he obo .
Du ing he eaching o lea ning phase, he ope a o holds
and uses he each pendan o d i e he obo s ep by s ep o
any desi ed loca ions which a e equi ed o pe o ming he
ask. The ea e , he ele an poses a e s o ed, and inally,
du ing he p oduc ion cycle, he obo epea s au onomously
he mo emen s and ac ions ha lead o pe o m he ask a
speci ied speed. (Fukui e al., 2009; Joseph, 1998).
The lead h ough and each pendan me hods in ol e he
human ope a o in he ask, bu only du ing he eaching
phase which is pe o med ou o he p oduc ion cycle. A e
lea ning he p ocesses, he obo ac ions a e con e ed in o
ex ual p og ams which a e used he ea e o au oma ically
pe o m he ask (A gall e al., 2009).
INTERACTIVE COMMANDS BASED ON
GUI
I we conside he p e iously p esen ed adi ional
in e ac ion modes, we no ice ha he e was a need o ee he
ope a o om his con inuous engagemen du ing ull
eleope a ion and in e sely o gi e him he possibili y o
in e ene when needed du ing he ull au oma ic ask
execu ion o he p og amming mode. To his end, he
in e ac i e con ol mode, which has eme ged a ound 1995, has
p o ided an al e na i e o he use o in e ene du ing ask
execu ion and e en o swi ch om one command mode o
ano he . I was os e ed by he appa i ion o so wa e
in e acing acili ies. I has been made possible conjoin ly wi h
he de elopmen o he objec -o ien ed p og amming
echniques and hei co esponding languages such as C++, Ja a,
Py hon, e c. Basically, in e ac i e p og amming enables o buil
up G aphical Use In e aces (GUI) on compu e sc eens ha
con ain g aphical objec s like windows, panels, bu ons, slide s,
pop-up menus, e c. Each g aphical objec can be linked o a
unc ion ha can be execu ed each ime he use ac i a es he
co esponding objec om he GUI. The gene a ion o unc ions
can be done by mouse clicks on pa icula widge s, by yping
speci ic cha ac e keys o he compu e keyboa d o by sc een
ouch (Figu e 2). The GUI enables also he p esen a ion o da a
in di e en o ma s and s uc u es ( ex s, images, ideos)
(Appels al e al., 2018; Mye s, 1995).
F om he HRI poin o iew, he in oduc ion o in e ac i e
p og amming echniques was welcomed as i gi es he
possibili y o gene a e in e ac i ely commands ia GUI while he
obo is e en pe o ming ope a ions wi hou necessa ily
s opping i . I enables also he e lec ion o eedback in o ma ion
ia he GUI.
As shown in Figu e 2, many needed unc ions can be
p og ammed and launched ia he GUI. Posi ion commands,
join commands, eloci y commands, o ce commands, ei he
cons ained o no can be gene a ed in e ac i ely. Wi hin his
in e ac i e mode, he ope a o can ca y ou pick-and-place asks
wi h obo manipula o s o di ec a mobile obo o some
des ina ion. Fo ins ance, o achie e asks wi h his mode, he
ope a o di ec s he obo by a se ies o clicks on he app op ia e
bu ons.
In addi ion, his mode enabled o combine pu e eleope a ion
wi h pu e ex ual p og amming, p o iding a way o o e coming
hei inhe en limi a ions. These modes o eleope a ion, o -line
p og amming and in e ac i e p og amming can be combined
and sha ed o pe o m asks mo e e icien ly. This in e ac i e
mode has opened many ways o HRI and coope a ion (Ma ion
e al., 2017; Zaa i, 2000; Zendoui e al., 2018)
Figu e 1. Teleope a ion sys em
Figu e 2. In e ac i e commands
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
4 / 11
MODERN COMMAND MODES
Mode n command modes make he e olu ion o he ask
easie and mo e e icien . Ye , hey equi e mo e complex
sys ems o be implemen ed o hey need senso s, ela i ely
as compu e s and AI echniques. They need algo i hms ha
p ocess da a, ex ac ele an ea u es and impo an
in o ma ion as humans do. In he ollowing sec ions, we
p esen speech-based command, poin ing on image-based
command, ges u e-based command, and b ain-based
command.
Speech-Based Commands
I is well known ha speech is he mos na u al way o
communica e be ween humans o he wise alking o obo s a
he ea lie age o obo ics was like a d eam. Howe e , la e
on wi h he de elopmen o AI, human speech appea s o be
an in e es ing mean o command, con ol and communica e
wi h obo s.
S ill, using human speech o command obo s equi es
speech ecogni ion echniques, which s a ed in he ea ly
1950s. O cou se, ea ly sys ems had limi ed ocabula y, bu
mode n speech ecogni ion sys ems ha e e ol ed and a e
now widely a ailable in many domains. They ha e gained
use wi h in elligen assis an s, such as Amazon's Alexa,
Apple's Si i and Mic oso 's Co ana, Google’s Google
Assis an and o he s (Te zopoulos and Sa a zemi, 2020).
These sys ems a e enabling speech in e ac ion wi h
compu e s and o he de ices. They ha e capabili ies o
in e p e wha is said h ough speech ecogni ion sys ems
and e en may espond o ques ions o commands ac oss ex -
o-speech sys ems.
In pa icula , au oma ic speech unde s anding o
gene a ing obo commands is linked o he ield o
Au oma ic Speech Recogni ion (ASR). ASR is in ended o
con e human speech in o w i en ex s. Mos Speech-based
commands and in e ac ion be ween humans and obo s ha e
been de eloped wi h he ecogni ion o isola ed wo ds
because i is easie o be used as obo commands. Examples
o sen ences ha a e used by speech sys ems o command
obo s a e: go o ini ial posi ion, ind able, mo e le , e c.
Howe e , because o he complex na u e o oice signals, ASR
s ill emains in some ci cums ances a ela i ely ha d issue o
obo s o unde s and speech commands.
The p inciple o speech commands
The p inciple used o mos wo d ecogni ion sys ems can
be illus a ed in Figu e 3 and Figu e 4. I comp ises wo phases:
he lea ning phase and he ecogni ion phase. The lea ning
phase consis s o c ea ing a lis o wo ds which a e s o ed in o
a dic iona y as e e ence wo ds. The ecogni ion phase consis s
o iden i ying any new spoken wo d o one o he e e ence
wo ds s o ed in he dic iona y.
An example o ASR we ha e implemen ed was based on
he ollowing p ocedu e: any spoken wo d which is a
con inuous acous ic signal is ansla ed by he mic ophone in o
an elec ic con inuous signal. This con inuous elec ical signal
is hen sampled by a sound ca d. Some digi al ope a ions a e
hen applied such as p e-emphasis, Fas Fou ie T ans o m
(FFT), powe spec um, il e bank in eg a ion (Mel's Fil e ),
loga i hmic comp ession, Disc ee Fou ie T ans o m. The inal
ou pu is a se o coe icien s which a e called Mel F equency
Ceps al Coe icien s (MFCC) (Muda e al., 2000). MFCC a e he
main ea u es ha cha ac e ize a speech signal. They se e o
build he dic iona y o he obo commands ( e e ences) a e a
aining phase o he use . They se e also in he ecogni ion
phase o iden i y any new unknown obo command by
compa ing i o hose s o ed in he dic iona y.
In o de o ecognize any spoken wo ds conside ed as
possible obo commands, ASR is using a ious me hods o
classi ica ion such as Hidden Ma ko Model (Rabine , 1989),
VQ ec o quan i ica ion (Linde e al., 1980), and lea ning
echniques as Neu al Ne wo ks (Paul and Pa ekh, 2011).
Howe e , he Mul iLaye Pe cep on (MLP) is o special
impo ance o acous ic modeling in ASR (Pin o, 2010). As an
example in one o ou implemen a ion, he ole o he classi ie
is played by he MLP (Zaa i e al., 2015). I selec s he closes
e e ence wo d wi h espec o he spoken one. The scheme o
an ASR is ep esen ed in Figu e 4.
Figu e 3. Lea ning phase
Figu e 4. Recogni ion phase
5 / 11
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
Howe e , speech-based command is in e es ing o
handicapped people using o ins ance wheelchai s id he
ope a o ’s hands canno be used o i hey a e busy wi h some
o he asks. Wi h he ecen de elopmen o pe sonal and
humanoid obo s, his mode is also gaining mo e in e es .
Ne e heless, ARS applied o obo ic domain s ill su e s
om many ba ie s. I is sensi i e o ha d noisy
en i onmen s. The esul s depend on he pe sonal
cha ac e is ic o he ope a o ha can only be imp o ed by a
long aining. E icien success can be ob ained wi h limi ed
ocabula y and canno ma ch all and new con ex s
(Oussalah and Zaa i, 2003; Zaa i e al., 2015).
Commands by Poin ing on Images (Look- hen-Mo e)
The use o ision o command and con ol obo s is a e y
impo an elemen o enhancing obo applica ions. Today,
many comme cial obo ics in eg a ing ision sys ems a e
a ailable. Bu because o he sensi i i y and complexi y o
image p ocessing, he e iciency o such sys ems is mainly
limi ed o s a ic indus ial applica ions. Fo ele obo ics,
ision-based con ol echniques, as s essed in (She idan,
1992), make a powe ul se o ools o obo in e ac ion and
con ol especially in uns uc u ed and unknown
en i onmen s. These echniques do no equi e p e-
knowledge o models abou he objec s o wo k wi h. Wi h
hese echniques, he human ope a o designa es he objec s
o loca ions o in e es om he ecei ed images; a e wa ds
he sys em ex ac s hei co esponding ea u es and heads
owa ds hem in eal space. In all hese echniques, he
supe iso ’s ole, as obse ed in (She idan, 1992) “is limi ed
o concep ion and poin ing, and he ele- obo do he es ”.
In gene al, ision-based obo command sys ems can be
di ided in wo main olds ega ding he applied con ol
sys em a chi ec u e. The ea lie echnique is named look-
and-mo e and he o he one is isual se oing.
Image-Based as Look- hen-Mo e Commands
Because eal- ime image p ocessing was ela i ely slow,
he ea lie ision-based sys ems we e p incipally designed
o applica ions o ype "look- hen-mo e" which a e usually
dedica ed o s a ic en i onmen s. They we e also e e ed
some imes as poin -and-click o image-based commands
(Kim and S a k, 1989; Oussalah and Zaa i, 2003). F om he
con ol sys em iewpoin , hei con ol a chi ec u e is an
open loop one wi hou con inuous image p ocessing
eedback.
These sys ems include s e eo ision sys ems o cap u e
images. They wo k as ollows (Figu e 5). A scene is selec ed
by he ope a o which inside which he objec o in e es
appea s in bo h s e eo ision came as. The ope a o selec s
his objec in one image by poin ing on i wi h a poin ing
de ice such as a mouse click. The s e eo ision sys em g abs
au oma ically he le and igh images. Then a s e eo ision
algo i hm p oceeds hese images in o de o ex ac he
co esponding 3D coo dina es o he selec ed objec o
in e es in he eal wo d. Once, hese 3D coo dina es a e
es ima ed, hey a e sen as a posi ion command o he obo
o heading owa ds his loca ion o his objec . In ac , he
obo execu es his command by pe o ming “blind”
mo emen s which assumes ha he en i onmen emains
s a ic a e he obo has s a ed o mo e. I is an open-loop
app oach o a ype i e and o ge (Wang, 2016) . By epea ing
his p ocess, he ope a o can di ec easily he obo o pe o m
manipula ion asks and/o na iga ion.
Howe e , while his echnique is simple, i enables high
le el commands. I ep esen s an e icien way o gene a e
look- hen-mo e obo commands o s a ic en i onmen s.
A success ul implemen a ion o an au oma ic image-based
(look- hen-mo e) commands named click-and-mo e, has been
de eloped o six DOF manipula o s. I has been desc ibed in
(Oussalah and Zaa i, 2003; Zaa i and Van B ussel, 1997). I has
also been ex ended o mobile manipula o s (Zaa i, 2000) and
o pa allel cable-based obo s (Bouchemal and Zaa i, 2014). All
ou expe imen s ha e p o en ha i is a e y powe ul
echnique o a ious ypes o obo s in he ange whe e
calib a ion o he came as has been pe o med.
Visual Se oing Con ol
Wi h he inc easing powe o image p ocessing, i becomes
possible o apply machine ision o dynamic sys ems ac ing o
mo ing in a non-s a ic en i onmen and capable o acking
mo ing a ge s. This echnique, known as ision-based con ol
o isual se oing appea ed in 1979 (Co ke, 1994; Hans, 2018).
Visual se oing is dis inguished om look- hen-mo e since i
enables o con ol he obo 's mo ion using eal- ime eedback
based on ision senso s (Vah enkamp e al., 2008).
Ges u al-based Commands
Ges u e-based obo command is a echnique ha uses he
mo emen o some body pa s in o de o in e ac and
command obo s. Ges u es can be o any ype o body
mo emen s: head mo emen , hand ges u e, anybody
mo emen and e en acial exp essions. Also, ools mo ed by a
human ope a o such as pencils, lags, s icks, e c. can be used
o in e ac and command obo s.
As he many mode n modes o obo command, ges u e-
based command has been s udied, designed and implemen ed
by many au ho s. A su ey conce ning ges u e-based
in e ac ion is p esen ed in (Gal án-Ruiz e al., 2020; S. Mi a
and T. Acha ya, 2007). Technically, in mos applica ions,
ges u e-based in e ac ion equi es de ec ion and iden i ica ion
o ges u es as in ended obo commands. Ac ually, de ec ion
and iden i ica ion o ges u es a e mainly based on objec
ecogni ion and acking app oaches in ol ing CCD came a
senso s. As o isual se oing, se e al image p ocessing
me hods a e used o image ecogni ion such as ea u es-based,
appea ance-based, g adien -based, lea ning me hods, e c.
(Bake and Ma hews, 2004; Bonci e al., 2021; Lucas and
Kanade, 1981; Nea chou, 2011).
P inciple o Ges u e-Based commands
The p inciple o ges u e-based in e ac ion sys ems is
simple. I equi es senso s ha cap u e sequences o images
du ing he mo emen o a human body pa o o any ool
mo ed by he ope a o . These sequences o images a e hen
analyzed by an image p ocessing so wa e in o de o ack he
mo ion o some de ec ed elemen s o in e es . Once a signi ican

Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
6 / 11
mo emen is de ec ed and acked; he inal con igu a ion o
he ges u e is iden i ied. I is in e p e ed wi h espec o a
code ha links de ec ed mo emen s wi h co esponding
obo commands. This iden i ica ion o he ges u e is inally
sen as a co esponding obo command. Las ly, he obo
pe o ms he in ended ac ion o ask. Figu e 6 summa izes
he p inciple o ges u e-based obo command showing he
sequence o he designed and implemen ed main ope a ions
(Zaa i, 2021).
Applica ions o ges u e-based obo command a e
nume ous and can be adap ed and ex ended acco ding o
many needs and con ex s. Ges u e exp essions can be
employed in egions whe e he speech is useless. This can
happen, o ins ance, in noisy places, in unde wa e a eas,
and in emp y space whe e he medium canno con oy he
oice wa es as wi h as onau s (Liu e al., 2016). I can be also
used o lea ning by demons a ion and o imi a ion as
ep oducing ope a ions in medical ca e o ele-su ge y
(S aub e al., 2011). This command mode is also in e es ing in
some mili a y ac i i y o communica e and di ec emo e
eams, au onomous and unmanned sys ems (Ellio e al.,
2016). Mo eo e , i can be used in assis i e obo ics o
supe ising dea people, o su eillance o disabled people,
e c. (Bouchemal and Zaa i, 2013).
In addi ion, he ges u e-based in e ac ion can be designed
wi h con ac o wi hou con ac . Examples o ges u e
in e ac ion wi h con ac a e used o eaching on blackboa ds,
ables and o he suppo s. In hese si ua ions, ma ke s and
colou s can be employed o acili a e he acking o elemen s
o in e es (Bouchemal and Zaa i, 2013; Nea chou, 2011;
Sigalas e al., 2010). Mo eo e , wi h he e en o Co id-19
pandemic, ges u e-based con ol wi hou con ac is gaining a
special impo ance by a oiding ouch and con ac wi h
con amina ed people and objec s like doo handles,
machines, e c.
This command mode o e s he ad an age o eeing he
ope a o om he con ac wi h he compu e . Ne e heless,
some di icul ies which a e ela ed o image p ocessing and
en i onmen al issues can limi he capabili ies and
pe o mances o his command mode.
BRAIN-BASED INTERACTION AND
COMMANDS
Beside he a ie y o exis ing human-machine in e ac ion and
command echniques, hese las decades, biological ones ha e
opened new unp eceden ed pe spec i es owa ds e y
in e es ing and p omising inno a i e esea ches. Indeed, bio-
mechanical ac ions, myo-elec ic and bio-cybe ne ic echniques
can be used o in e ac ion, command and con ol o dynamical
sys ems such as obo s, wheelchai s, and e en pa alyzed body
pa s o human (Be na-Ma inez, 2011; Ma inek e al., 2021;
Rechy-Rami ez and Hu, 2015).
In his con ex , he mas e ing o he b ain ac i i y o human
beings o in e ac ing and commanding sys ems e eals o be a
e y a ac i e opic. Indeed, b ain ac i i y is he p ime sou ce
o gene a ing command signals. In he pas , unde s anding
mind hough s and de ec ing in en ions we e conside ed as a
kind o elepa hic capabili y which only exis s in science ic ion.
Howe e , wi h ecen ad ances in neu o-sensing echnologies,
his belie is inally u ning in o eali y (Nam e al., 2018). As a
esul , he ield o human- obo in e ac ion has been signi ican ly
en iched by he b ain-based in e ac ion and command
echniques. I is ac ually possible o a subjec , by means o B ain
Compu e In e aces (BCI), o de ec o gene a e commands ha
can be used o manipula e i ual o eal sys ems acco ding o
in en ions h ough he b ain ac i i y (Nicolas-Alonso and
Gomez-Gil, 2012).
Figu e 5. Look hen mo e commands
Figu e 6. Look hen mo e commands
7 / 11
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
P inciple o B ain-Compu e In e aces
C ow I is well es ablished ha human b ain ac i i y
consis s o emi ing wa es wi h ce ain pa e ns as a esul o
ex e nal s imuli o men al s a es (Buzsaki, 2006; Doelling and
Assaneo, 2021). BCIs a e speci ic de ices which a e capable
o cap u ing and measu ing b ain ac i i y, hen ex ac ing
in e es ing ea u es, and inally con e ing hese ea u es in o
ou pu s o in e ac wi h and command dynamic sys ems.
Thus, BCIs enable use s o in e ac wi h compu e s and o he
de ices by means o b ain ac i i y only (A onso e al., 2014;
A onso, 2013; Yonck, 2002). They p o ide a unique way o
communica ion be ween a human and a machine (o de ice)
wi hou any neu omuscula in e en ion (Abi i e al., 2017;
Wolpaw e al., 2002). Mo e gene ally, B ain–Machine
In e aces (BMIs) a e de ices ha ansla e neu onal
in o ma ion in o commands capable o con olling ex e nal
so wa e o ha dwa e such as compu e s, wheelchai s,
p os hesis, and obo s. Figu e 7 shows a scheme o a BCI
in e acing and con e ing b ain ac i i y o a use in o
command signals wi h possible eedback (do ed lines) o he
use .
The design p inciple o BCIs is simila o mos o he
in e aces such as speech-based, image-based, ges u e-based
ones. I consis s o con e ing basic signals in o commands
by means o a se o p ocessing ope a ions: de ec ing he
p ima y use 's signals, p ocessing i in o de o ex ac
ele an ea u es, classi ying he ob ained ea u es in o de o
decide o which p ede ined command i is supposed o ha e
been issued; hen his command is sen o execu ion (see
Figu e 8).
BCI echniques
Di e en echniques a e used o measu e b ain ac i i y
o BCIs. Mos BCIs use elec ical signals which a e de ec ed
using senso s placed in asi ely o non-in asi ely. The e a e
se e al echniques o nonin asi e BCI, such as EEG
(elec oencephalog aphy), MEG (magne oencephalog aphy),
o MRT ( unc ional magne ic esonance imaging:
omog aphy) (Fe ei a e al., 2008). Elec oencephalog aphy
(EEG) is a physiological me hod o eco d he elec ical ac i i y
gene a ed by he b ain ia elec odes placed on he scalp
su ace. The signal ampli ude is usually unde 100 μV and he
equency band o no mal EEG signals is usually abo e DC up
o 50 Hz (Ma inek e al., 2021).
In in asi e echniques, special de ices a e inse ed di ec ly
in o he human b ain by su ge y. In Semi-in asi e, de ices a e
inse ed in o he skull on he op o he human b ain, o di ec ly
on he co ex (called Elec oco icog aphy – ECoG); he su ace
o he b ain (signal ha ing abou 1- 2 mV o ampli ude (Fe ei a
e al., 2008). In gene al, non-in asi e echniques a e conside ed
as sa es , o low-cos ype o de ices and hen easies o
s udies. Howe e , he cap u ed a human b ain signals a e
weake compa ed o in asi e echniques which a e in di ec
con ac s wi h neu al cells (Fe ei a e al., 2008; S ey l e al.,
2016).
Applica ions and Pe spec i es o BCIs
F om he beginning, mos applica ions o he biological and
physiological con ol echniques we e o ien ed owa ds he
assis ance o handicapped and disabled people. The p og ess in
de eloping BCIs is p o iding a lo o hope o his communi y,
especially o hose wi hou muscula capabili y. Wha is
ema kable conce ning he BCIs is ha mos applica ions
dedica ed o medical applica ions can in ol e he use a he
same ime as a commande and as a subjec .
A e a aining pe iod, he use can lea n how o gene a e
commands by hough s, o con ol and ac i a e p os hesis
membe s o his own body pa s such as pa alyzed membe s
(Baniqued e al., 2021; Mane e al., 2020). Figu e 9 p esen an
ope a o wea ing a BCI o con olling a mechanical sys em.
Figu e 7. BCI in e acing b ain ac i i y o sys ems
Figu e 8. Global o ganiza ion o BCIs
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
8 / 11
Figu e 9. Ope a ion wi h BCI commanding a mechanical
sys em
BRIEF SUMMARY ABOUT AI AND HRI
Since he eme gence o he i s heo ies and app oaches o
AI, he e was a hope o in use hese echniques in pa icula o
obo s o imp o ing hei adap i i y and sma ness. The low
o echniques p o ided by AI such as symbolic compu a ions,
gene ic algo i hms, neu al ne s, uzzy logic, and machine
lea ning, has se ed and s ill se es o a emp ing o sol e and
o e come many issues ha limi obo ic applica ions. Ac ually,
AI has ed obo s by nume ous capabili ies wi h mo e o less
sa is ying success. Some ema kable ields a e speech
unde s anding, objec ecogni ion based on ision,
au onomous na iga ion, manipula ion asks, in e ac ions wi h
humans and o he en i ies, coope a ion, and so on (Pe ez e al.,
2018; Seme a o e al., 2021).
Bu ollowing he inc easing complexi y o obo ic sys ems
designed o au onomously pe o m asks and missions in
a ious en i onmen s; speci ic con ol a chi ec u es o
in elligen sys ems equi ing high le el lexibili y and
adap i i y was p oposed and implemen ed. One popula
example o such a chi ec u e is he h ee laye ed one which
combines e lexi e and eac i e beha io s as well as
delibe a i e capabili ies (A kin, 1998, p. 199; R. B ooks, 1986; S.
Ga cıa e al., 2018).
Howe e , conce ning he con ibu ion o AI o he
pa icula domain o HRI, some au ho s ha e analyzed and
discussed many ele an applica ions, challenging issues and
po en ial p omising pe spec i es (Feil-Sei e and Ma a ic, 2009;
Lemaignan e al., 2017; Seme a o e al., 2021; She idan, 2016).
In ac , he mos challenging goal o AI ela ed o HRI is s ill
how o design obo s ha can unde s and and in e p e human
beha io s and in en ions in dynamic and possibly unp edic ed
si ua ions in o de o ake app op ia e decisions?
To his end, con inuous e o s a e spen on esea ching
echniques and designing join cogni i e a chi ec u es o
imp o e HRI by ying o allow obo s gaining mo e
knowledge and au onomous de elopmen as human beings
do. E o s a e o ien ed o de eloping obo mechanisms o
enable de elopmen al lea ning and capabili y o acqui ing
skills (Lemaignan e al., 2017; Nicolescu and Ma a ic, 2005;
Seme a o e al., 2021). Beside supe ised and unsupe ised
lea ning, ein o cemen lea ning, deep lea ning;
de elopmen al au onomous lea ning is a e y impo an and
challenging app oach. I is a kind o a sel -lea ning compe ence
whe e he obo a emp o acqui e knowledge and skills by
con inuous obse a ion and imi a ion o people and o he
in elligen sys ems. As wi h humans, i equi es obse ing e en s
and si ua ions, classi ying and o ganizing hem, ex ac ing
ele an ea u es and in e ing ules and unde s anding
si ua ions and pa ne 's in en ions (A en s and G ei ans, 2022;
Nicolescu and Ma a ic, 2005).
Among he issues and pe spec i es is he need o make mo e
na u al he in e ac ion and he unde s anding be ween humans
and obo s as be ween humans hemsel es. This eques
cons i u es also one main objec i e o he 4 h indus ial
gene a ion which in ends o es ablish na u al coope a ion and
collabo a ion be ween humans and sma obo s named
some imes "cobo s" in o de o achie e common asks and
missions (A en s and G ei ans, 2022; Chak abo i e al., 2017;
Ja aid and Khan, 2021; Zamalloa e al., 2017).
Mo eo e , he ac ual and u u is de elopmen o di e en
kind o in elligen obo s and hei a ious possible in e ac ions
wi h humans is p essing owa ds a homogeneous ep esen a ion
o Human-Robo con ol a chi ec u es. The e is a se ious need o
de elop a uni ied con ol a chi ec u e o HRIS ha ele a es he
cogni i e compe encies o he obo s o app oxima e he le el o
humans and elimina es he e o e he dis inc ion be ween
humans and a i icial in elligen sys ems when wo king oge he
as eam membe s (Ha io and Adams, 2013; K äme e al., 2012;
Zaa i, 2021).
CONCLUSION
This pape has b ie ly desc ibed he mos command modes
in ol ed in HRIS. This includes he adi ional in e ac ion and
command modes which a e namely ele-manipula ion, o -line
obo compu e p og amming and lea ning by demons a ion
(lead- h ough and each pendan ). I hen in oduces he modes
which ha e been os e ed la e on by he conjunc ion o obo ics
wi h he eme gence o a i icial in elligence echniques and he
p o ision o powe ul compu ing machines. The ollowing
modes we e conside ed: in e ac i e commands based on GUI,
oice-based commands, poin ing on image-based commands,
ges u e-based commands, and inally b ain-based commands. In
addi ion, some ele an and challenging issues co esponding o
some command modes ha e been b ie ly discussed.
One can no ice ha he gene a ion o obo commands
ollows almos he same p ocess: de ec ing he in ended
command signal gene a ed by he ope a o by means o speci ic
senso s; hen analyzing and iden i ying his command w. . a
dic iona y; and inally o de ing he obo o execu ing his
command. The cons i u ion o he dic iona y ha con ains he
e e ences a e usually cons i u ed h ough a lea ning p ocess.
I s pe spec i es on he sho and mean e ms, he e is a need
o imp o e he command modes o make hem mo e lexible and
use iendly. These modes can also be combined unde a
mul imodal ope a o in e ace o p o ide mo e lexible
in e ac ion and con ol o obo s. The e is also a need o use
echniques o a i icial in elligence o enable a human- obo
coope a ion and coo dina ion as pa ne s. On he o he hand,
9 / 11
Abdelouahab Zaa i / J INFORM SYSTEMS ENG, 7(2), 14039
BCI is aking mo e a en ion o he capabili y i o e s o
con olling i ual as well as eal dynamic sys ems. P obably,
he mos impo an domain in he nea u u e is he
applica ions o BCI in o de o help handicapped and disabled
people using hei pa alyzed limbs, wheelchai s, p os hesis,
and se ice obo s.
REFERENCES
Abi i, R., Heise, G., Zhao, X., Jiang, Y., Abi i, F.A., 2017. B ain
compu e in e ace o ges u e con ol o a social obo :
An o line s udy. 2017 I anian Con e ence on Elec ical
Enginee ing (ICEE) 113–117.
A onso, A., Angélico Gonçal es, M., Lima, J., Co a, M., 2014.
UsaWeb. A model o usabili y e alua ion web
in e aces, in: Ibe ian Con e ence on In o ma ion
Sys ems and Technologies, CISTI. P esen ed a he
Ibe ian Con e ence on In o ma ion Sys ems and
Technologies, CISTI, p. 6.
h ps://doi.o g/10.1109/CISTI.2014.6876952
A onso, A.P., 2013. P oyec o de E aluación de In e aces —
PAI,. Uni e sidad de Vigo, Depa amen o de
In o má ica., Vigo.
Appels al, M., Michalak, J., Os e be g, M., 2018. Easy o Use
G aphical Use In e ace o Robo P og amming.
A en s, J., G ei ans, M., 2022. Sma Indus ial Robo Con ol
T ends, Challenges and Oppo uni ies wi hin
Manu ac u ing. Applied Sciences 12.
h ps://doi.o g/10.3390/app12020937
A gall, B.D., Che no a, S., Veloso, M., B owning, B., 2009. A
su ey o obo lea ning om demons a ion. Robo ics
and Au onomous Sys ems 57, 469–483.
h ps://doi.o g/10.1016/j. obo .2008.10.024
A kin, R.C., 1998. Beha io -based obo ics. MIT p ess.
Bake , S., Ma hews, I., 2004. Lucas-Kanade 20 Yea s On: A
Uni ying F amewo k. In . J. Compu . Vision.
Baniqued, P., S anye , E., Awais, M., Alazmani, A., Jackson, A.,
Mon-Williams, M., Mush aq, F., Hol , R., 2021. B ain-
Compu e In e ace Robo ics o Hand Rehabili a ion
A e S oke: A Sys ema ic Re iew. Jou nal o
Neu oEnginee ing and Rehabili a ion 18.
h ps://doi.o g/10.1186/s12984-021-00820-8
Be na-Ma inez, J., 2011. Robo ic Con ol Based On The Human
Ne ous Sys em. In e na ional Jou nal o A i icial
In elligence & Applica ions 2, 107.
h ps://doi.o g/10.5121/ijaia.2011.2409
Bonci, A., Fio i, S., Higashi, H., Tanaka, T., Ve dini, F., 2021. An
In oduc o y Tu o ial on B ain–Compu e In e aces
and Thei Applica ions. Elec onics 10, 560.
h ps://doi.o g/10.3390/elec onics10050560
Bouchemal, B., Zaa i, A., 2014. Image-based con ol o cable-
based obo s. In e na ional Jou nal o Con ol,
Au oma ion and Sys ems 12, 118–125.
h ps://doi.o g/10.1007/s12555-012-0445-y
Bouchemal, B., Zaa i, A., 2013. Ges u al and Image-Based
Con ol Combina ion o ehabili a ion applica ions using
cable-based obo s. P esen ed a he In e na ional
Con e ence on Technology o Helping People wi h
Special Needs, Riyadh, Kingdom o Saudi A abia.
Buzsaki, G., 2006. Rhy hms o he B ain. Ox o d uni e si y p ess.
Cecca elli, M., 2001. A His o ical Pe spec i e o Robo ics Towa d
he Fu u e. Jou nal o Robo ics and Mecha onics 13, 299–
313. h ps://doi.o g/10.20965/j m.2001.p0299
Chak abo i, T., Kambhampa i, S., Scheu z, M., Zhang, Y., 2017.
AI challenges in human- obo cogni i e eaming. a Xi
p ep in a Xi :1707.04775.
Co ke, P., 1994. High-Pe o mance Visual Closed-Loop Robo
Con ol.
Doelling, K., Assaneo, M., 2021. Neu al oscilla ions a e a s a
owa d unde s anding b ain ac i i y a he han he end.
PLoS biology 19, e3001234.
h ps://doi.o g/10.1371/jou nal.pbio.3001234
Eakins, W., Rossano, G., Fuhlb igge, T., 2013. Lead h ough
Robo Teaching. P esen ed a he IEEE Con e ence
Technology P ac ice Robo Applica ion, pp. 1–4.
Ellio , L.R., Hill, S.G., Ba nes, M.J., 2016. Ges u e-Based Con ols
o Robo s: O e iew and Implica ions o Use by
Soldie s. P esen ed a he Human Resea ch and
Enginee ing Di ec o a e, ARL, US A my Resea ch
Labo a o y, ARL-TR-7715.
Feil-Sei e , D., Ma a ic, M., 2009. Human-Robo In e ac ion, in:
Jou nal Abb e ia ion: Encyclopedia o Complexi y and
Sys ems Science. Sp inge , pp. 4643–4659.
Fe ei a, A., Celes e, W., aua cheein, F., Bas os, T., Sa cinelli-
Filho, M., Ca elli, R., 2008. Human-Machine in e aces
base on EMG and EEG applied o obo ic sys ems. Jou nal
o neu oenginee ing and ehabili a ion 5, 10.
h ps://doi.o g/10.1186/1743-0003-5-10
Fo d, M., 2015. Rise o he Robo s Technology and he Th ea o
a Jobless Fu u e. Basic Books, New Yo k, NY, US.
Fukui, H., Yonejima, S., Yamano, M., Dohi, M., Yamada, M.,
Nishiki, T., 2009. De elopmen o eaching pendan
op imized o obo applica ion. 2009 IEEE Wo kshop on
Ad anced Robo ics and i s Social Impac s 72–77.
Gal án-Ruiz, J., T a ieso, C., Teje a-Fe milch, A., Pinan-
Roesche , A., Es eban-He nández, L., Dominguez
Quin ana, L., 2020. Pe spec i e and E olu ion o Ges u e
Recogni ion o Sign Language: A Re iew. Senso s 20,
3571. h ps://doi.o g/10.3390/s20123571
Gaspa e o, A., Scale a, L., 2019. A B ie His o y o Indus ial
Robo ics in he 20 h Cen u y. Ad ances in His o ical
S udies 08, 24–35. h ps://doi.o g/10.4236/ahs.2019.81002
Hans, S., 2018. A B ie Compa a i e S udy o Visual Se oing
Sys ems. IJSRD - In e na ional Jou nal o Scien i ic
Resea ch & De elopmen 6, 2321–0613.
Ha io , C., Adams, J., 2013. Modeling Human Pe o mance o