Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/2.
1
And oid based au onomous mobile obo
Ge gely Nagymá é
Depa men o Mecha onics Op ics and Enginee ing In o ma ics,
Budapes Uni . o Technology and Economics
Budapes , Hunga y
nagyma e.ge gel[email p o ec ed]m
Abs ac —The sp eading o mobile obo s is ge ing mo e
signi ican nowadays. This is due o hei abili y o pe o m asks
ha a e dange ous, uncom o able o impossible o people. The
mobile obo mus be endowed wi h a wide a ie y o senso s
(came as, mic ophones, p oximi y senso s, e c.) and p ocessing
uni s ha makes hem able o na iga e in hei en i onmen . This
gene ally ca ied ou wi h unique, small se ies p oduced and hus
expensi e equipmen . This pape desc ibes he concep o a
mobile obo wi h a con ol uni in eg a ing he p ocessing and
he main senso unc ionali ies in o one mass p oduced de ice, an
And oid sma phone. The obo is able o pe o m asks such as
acking colo ed objec s o human aces and o ien i sel . In he
mean ime, i a oids obs acles and keeps he dis ance be ween he
a ge and i sel . I is able o e bally communica e wi .
Keywo ds— mobile obo , And oid sma phone, human-
machine in e ac ion
I. INTRODUCTION
The compu ing capaci y o oday’s mobile phones is many
imes g ea e han he one’s was used o Moon landing [1],
ye commonly used o en e ainmen pu poses. These pocke
compu e s a e equipped wi h many di e en kind o senso s
ha a e also used in obo ics. Ad anced obo s o en need
GPS, ine ial senso s and e-compass, o na iga ion and
o ien a ion, came as o image p ocessing, mic ophone as
audio inpu and wi eless connec i i y. These a e all included
in sma phones. This pape no only d aws a en ion o
sma phones o hei in eg a ed senso s, bu he eno mous
so wa e suppo and open sou ce solu ions.
The e ha e al eady been esea ches using sma phones in
obo ics. A common applica ion uses sma phones as use
in e aces o emo e con olling [2][3]. A semi au onomous
obo p esen ed by S ans ield and Bo hma is s ill emo e
con olled and has And oid de ice on bo h sides, bu on he
obo side i is used o in e acing an ex e nal dis ance senso
and also implemen s a simple collision a oidance algo i hm
[4]. A mobile obo p esen ed by Lim, Lee and Tewolde is
addi ionally o he p e ious obo uses he ine ial senso s o
he sma phone o imp o e indoo na iga ion capabili ies o
he obo [5]. Ano he obo akes ad an age o he came a
in eg a ed in he sma phone o implemen ing image
p ocessing o line ollowing [6].
Sma phone based mobile obo s a e used o educa ional
pu poses, whe e sma phone based image p ocessing is also
implemen ed [7] [8]. The e a e al eady obo pla o ms o
sma phones ha a e comme cial p oduc s wi h easy o use
de elope en i onmen designed o child en educa ional
p og amming [9] [10].
The p esen p ojec aimed o build a mobile obo ha is
capable o ul illing he ollowing objec i es while aking
ad an age o he ad anced unc ionali y o an e e yday
sma phone:
1) capable o acking human aces and colo ed objec
2) implemen s collision a oidance while ollowing i s
a ge
3) capable o e bal communica ion wi h humans,
ecei es o de s and answe s simple ques ions
Such obo ha can unde s and human ques ions and
in elligen ly answe hem seemed e y high ech a ew yea s
ago, bu oday his echnology is in e e yone’s pocke [11]
[12].
The obo is a ca -like ehicle wi h ul asonic dis ance
senso s and a 2 DOF consol o holding he And oid phone.
Two se o mo o s can change he yaw and pi ch angles o he
phone in 180°. A hi d se o is esponsible o s ee ing he
ehicle. Fig. 1 illus a es he mechanical cons uc ion o he
obo .
Fig. 1. Mechanical cons uc ion o he obo
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/2.
2
In he iew o o ganiza ion he pape has he ollowing
sec ions: Sec ion I was a sho in oduc ion. Sec ion II
desc ibes di e en possibili ies o connec ing he sma phone
o ex e nal ha dwa e. Sec ion III is abou he so wa e
solu ions o he obo . Sec ion IV sho ly desc ibes he
a chi ec u e o he so wa e. Sec ion V concludes he pape .
II. CONNECTING THE SMARTPHONE TO THE ROBOT
The p ojec was ca ied ou wi h a Samsung Galaxy Ace
S5830 ype sma phone. This was a mid- ange de ice
announced in 2011, Janua y. I has an 800 MHz ARM11
p ocesso and 278 Mb o RAM. The la es o icial so wa e
upda e o i was And oid 2.3.3 Ginge b ead. In he iew o
connec i i y his se up can ac as a USB sla e de ice. I can
also be connec ed h ough Blue oo h o an ex e nal ha dwa e.
Di ec Wi-Fi connec ion is only possible om And oid 4.0
and abo e [13].
Many mic ocon olle manu ac u e s o e now And oid
accesso y amewo ks o hei IC-s [14]. These es ablish
communica ion h ough USB be ween he And oid de ice and
he embedded sys em. In he USB communica ion he e is
always a hos con olle and a sla e de ice [14]. The hos
powe s he bus while usually he sla e de ice ep esen s
simple unc ionali y. De ices unning And oid 3.1 and abo e
can ac as USB hos con olle . Olde sys ems – such as in his
p ojec – can only ac as sla es. The e o e, USB
communica ion equi es an ex e nal mic ocon olle wi h USB
hos pe iphe al. This solu ion can bene i du ing longe
ope a ions o he obo , because he obo can cha ge he
And oid de ice.
Howe e , wi eless connec ion is always o e s wide
accessibili y. Wi h he abo e desc ibed se up, he
communica ion can be ca ied ou wi h Blue oo h as well. This
does no equi e Blue oo h module in he embedded sys em,
bu a USB-Blue oo h adap e can be used.
In he desc ibed sys em a Mic ochip PIC24FJ256DA
mic ocon olle was used. The mic ocon olle implemen s an
open sou ce amewo k called IOIO ha handles he USB
pe iphe al and suppo s he Blue oo h communica ion [15].
The amewo k comes wi h an And oid lib a y, which o e s
high le el unc ions o ini ializing and ope a ing digi al IO-s,
PWM ou pu s, UART pe iphe al and ime unc ions in he
mic ocon olle . This means ha unc ionali ies can be
p og ammed h ough he And oid p og am. Fig. 2 shows he
layou plan o he sys em.
Fig. 2. Layou o he ex e nal sys em. The cha ging cu en can be se wi h a
po en iome e . Blue oo h adap e equi es maximum cu en , while di ec
USB communica ion can be ope a ional wi h minimal cu en , hus sa es
obo ba e y
III. USED SOFTWARE SOLUTIONS AND OTHER POSSIBILITY
A. Image p ocessing
The main unc ions o he obo a e ace acking and colo
blob acking. These asks can be sol ed wi h ex e nal image
p ocessing so wa e lib a ies such as OpenCV [16] o wi h
na i e And oid unc ions.
1) Face acking
In he case o ace acking, he me hods desc ibed in his
sec ion can be compa ed in he pe spec i e o ame a e, and
he p ecision o ecogni ion. Spon aneous alse posi i e
ecogni ions can be il e ed ou wi h low pass il e . In case o
sho ime alse nega i e esul s du ing acking he obo will
look o he ace in he p e iously iden i ied egion o in e es .
2) Face acking wi h na i e And oid unc ions
Abo e And oid 4.0 he e is a Face de ec o unc ion in he
came a class [17]. This inds aces in he came a p e iew
image, hus i na i ely se es wi h he highes ame a e. The
unc ion’s e u n a ay con ains he posi ion o he aces in he
p e iew image. This ce ainly he mos op imized solu ion bu
was no ye a ailable due o he sys em se up.
Ano he na i e solu ion o ace de ec ion in And oid is he
indFaces unc ion, which is a ailable in e e y And oid
e sion [18]. This does no wo k on he li e p e iew image
bu needs a bi map image o p ocessing and is ela i ely slow.
3) Face acking wi h OpenCV
The OpenCV ace de ec ion uses Viola-Jones objec
de ec ion algo i hm [18], which is a Haa - ea u e based
cascade classi ie [20]. The ope a ion and aining o he
de ec o is desc ibed in de ails by Viola and Jones in hei
wo k on apid objec de ec ion [19]. OpenCV comes wi h
many p e- ained classi ie o ace, eyes, smile e c. These a e
s o ed in XML iles and can be simply loaded in o he
p og am. The OpenCV p o ides a wide scale o algo i hm
cus omiza ion compa ed o he p e iously men ioned ace
de ec o s. The image o be p ocessed can be downscaled,
which educes p ocessing ime pe ame. Also he size o he
smalles de ec able ace can be gi en. The smalle he ace we
wan o de ec he mo e imes he algo i hm has o scan he
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/2.
3
image wi h inc easingly smalle de ec ion window. By a ying
hese wo ea u es an op imum can be ound whe e he obo
can sense aces in an accep able dis ance, while p o iding
su icien ame a e.
4) Colo blob acking
The inpu image o he colo blob acke algo i hm is
coded in RGB colo space. In he i s s ep his is con e ed
in o HSV colo space. HSV colo model is a me hod o de ine
colo s acco ding o he h ee basic ea u es o he colo : hue,
sa u a ion and luminance [21]. In he p esen applica ion he
obo ollows a colo ed objec . The changes o ligh condi ions
due o he en i onmen and obo o ien a ion a ec he
de ec ed colo s. These changes in RGB colo space a ec all
he h ee colo ea u e, whe eas in HSV colo space hese
mos ly a ec he luminance componen o he colo . This
ea u e o he HSV colo space allows a mo e useable colo
based segmen a ion. The hue o he colo can be mo e
speci ied, while a wide ole ance can be se o he luminance
ea u e.
The algo i hm unde -samples he image i.e. educes
esolu ion o educe p ocessing ime. C ea es a new bina y
ma ix whe e only ills hose posi ions, whe e he
co esponding pixel alls be ween he allowed h esholds in
hue, sa u a ion and luminance. This bina y mask is expanded
o he o iginal size o he image. A bo de ollowing algo i hm
de eloped by Suzuki and Abe [22] inds he con ou s a ound
he mask and e u ns an a ay wi h he bounding ec angles o
he indi idual con ou s. The obo wan s o ollow he la ges
objec ha ma ches he a ge colo , he e o e akes he la ges
con ou as he ep esen a ion o he ollowed objec . I
calcula es he di e ence be ween he cen e poin o he
de ec ed colo blob and he cen e o he p ocessed image as
he egula o ’s e o (Fig. 3). The e o signal o he objec
ollowe con olle is calcula ed simila ly du ing ace acking
wi h he la ges , hence closes ace.
Table I. summa izes he ame a e esul s be ween he
expe imen ed colo blob and ace de ec o me hods in espec
o hei di e en pa ame e se ups.
Fig. 3. Sma phone sc een du ing colo acking
TABLE I. FPS VALUES WITH DIFFERENT TRACKING ALGORITHMS
(THE VALUES ARE IN FRAME/SECOND)
And oid na i e ace de ec o
0,75
OpenCV ace de ec ion
Face size
40%
30%
20%
10%
Image a io
100%
5.1
4.2
2.2
1.2
50%
7.3
6.3
4.78
3.75
30%
9.2
7.3
5.1
4.9
20%
9.11
9.3
9.25
5.86
Colo blob de ec ion
4,75
B. Speech ecogni ion and e bal esponse
Speech ecognize se ice and ex - o-speech engine a e
basic se ices in And oid, and can be easily implemen ed in
any p og am. The speech ecognize equi es in e ne
connec ion, because he p ocessing is done in he cloud [23]. I
e u ns a se o he possibly hea d wo d combina ions. The
gi en o de o he obo is alida ed i he e u ned se con ains
i o some hing e y simila . The obo answe s o each
ins uc ion using he ex - o-speech engine, and no i ies i he
ins uc ion was misunde s ood. I can also answe s o some
basic ques ions om a p ep og ammed answe bank.
The e bal o de s a e also used o changing be ween he
ace acking and objec acking. Fo a speci ied o de he
obo akes a colo sample om he objec held on o i s
came a and will ollow i . The speech ecognize can be
igge ed wi h a sho whis le.
IV. SOFTWARE ARCHITECTURE
The so wa e builds on pa allel unning h eads. The wo
main h eads a e he image p ocessing and he mic ocon olle
in e ace h ead ha eads he ex e nal senso s and d i es he
obo . These wo h eads communica e o each o he h ough
an in e media e con olle objec . The e a e wo mino h eads
esponsible o whis le de ec ion.
The analog digi al con e e module o he mic ophone is
capable o 44100 Hz sampling equency on 16 bi s. One
h ead eco ds hese samples and eeds hem o he whis le
de ec o h ead in 2048 sample big packages. The de ec o
han applies a Fas Fou ie T ans o ma ion on he da ase .
Spikes be ween 1 and 3 kHz a e conside ed as whis les. When
a whis le is de ec ed he speech ecognize se ice is being
launched, bu i s he eco ding has o be s opped o elease
he mic ophone as sys em esou ce.
The in e media e con olle objec is esponsible o he
e en ual acking o he de ec ed objec s. I also implemen s a
s a e machine o enhanced collision a oidance algo i hm.
The image p ocessing h ead does no supply he da a wi h
ixed sampling a e, hus he sys em clock is used o es ablish
he elapsed ime o he e ical and ho izon al PID
con olle s.
Fig. 4 summa izes he so wa e o ganiza ion.
Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2.
DOI: 10.17667/ iim.2015.1-2/2.
4
Fig. 4. So wa e a chi ec u e
V. CONCLUSION
This pape p esen ed a possible usage o sma phones in
mobile obo ics as complex p ocessing uni s and senso a ays.
Usage o sma phones in obo ic applica ions can ex end he
capabili ies o obo s compa ed o con en ional embedded
sys ems. Howe e , wi h mid- ange phones he esponse ime
o he sys em will be oo long. Simul aneous unning o
ad anced unc ions migh esul s phase shi on he eal- ime
beha io . Highe -end sma phones can ex end he embedded
compu e s o mobile obo s by aking ad an ages o all
bene i s o mass p oduc ion and open sou ce in he aspec s o
cos s, so wa e and ha dwa e de elopmen .
ACKNOWLEDGMENT
The au ho would like o hank o Ri a M. Kiss and Pé e
Tamás o hei commen s, which g ea ly imp o ed his pape .
REFERENCES
[1] Eldon C. Hall, (1996), Jou ney o he Moon: The His o y o he Apollo
Guidance Compu e , Res on, Vi ginia, USA: AIAA, p. 196, ISBN 1-
56347-185-X
[2] C. Pa ga, Xiaoou Li, Wen Yu, “Sma phone-Based Human Machine
In e ace wi h Applica ion o Remo e Con ol o Robo A m”, Sys ems,
Man, and Cybe ne ics (SMC), 2013 IEEE In e na ional Con e ence on
Oc . 2013, pp 2316-2321
[3] S. W. Moon, Y. J. Kim, H. J. Myeong, C. S. Kim, N. J. Cha, D. H.
Kim, “Implemen a ion o sma phone en i onmen emo e con ol and
moni o ing sys em o And oid ope a ing sys em-based obo
pla o m”, Ubiqui ous Robo s and Ambien In elligence (URAI), 2011
8 h In e na ional Con e ence on No 2011, pp 211-214
[4] N. S ans ield, B. Bo hma, “And oid based semi-au onomous collision
a oidance obo ”, Robo ics and Mecha onics Con e ence (RobMech),
2013 6 h, pp 135-139
[5] J. Lim, Seok Ju Lee, G. Tewolde, J. Kwon, “Ul asonic-senso
deploymen s a egies and use o sma phone senso s o mobile obo
na iga ion in indoo en i onmen ”, Elec o/In o ma ion Technology
(EIT), 2014 IEEE In e na ional Con e ence on June 2014, pp 593-598
[6] Nolan He ge , William Keyes, and Chao Wang, "Sma phone-based
Mobile Robo na iga ion" Depa men o Elec ical and Compu e
Enginee ing, Ca negie Mellon Uni e si y, May 9 h, 2012
[7] C. K o i sch, C. Hinge ; M. Me dan, G. Koppens eine , “Sma phone
d i en con ol o obo s o educa ion and esea ch”, Robo ics,
Biomime ics, and In elligen Compu a ional Sys ems
(ROBIONETICS), 2013 IEEE In e na ional Con e ence on No . 2013,
pp 148-154
[8] D. Yang, J.-K. Lim ; Y. Choi, “Ea ly childhood educa ion by hand
ges u e ecogni ion using a sma phone based obo ” Robo and Human
In e ac i e Communica ion, 2014 RO-MAN: The 23 d IEEE
In e na ional Symposium on Aug. 2014, pp 987-992
[9] Romo webpage: h p://www. omo i e.com/ Accessed in No 2014
[10] Sma bo webpage: h p://www.o e d i e obo ics.com/ Accessed in
Aug 2014
[11] Apple Inc. h p://www.apple.com/iphone/ ea u es/si i.h ml. Accessed:
Oc 2014
[12] Google Mobile. h p://www.google.com/mobile/ oice-ac ions
Accessed: Oc 2014
[13] And oid De elope Si e – Wi-Fi Pee - o-Pee
h p://de elope .and oid.com/guide/ opics/connec i i y/wi ip2p.h ml
Accessed: No 2014
[14] Flowe s, D.: And oid accesso y de elopmen wi h he Google Open
Accesso y F amewo k. ELEKTRONET (2012, No embe ), pp 24-25.
[15] IOIO Documen a ion h ps://gi hub.com/y ai/ioio/wiki Accessed: Oc
2014
[16] OpenCV Use Si e: h p://openc .o g/ Accessed: Oc 2014
[17] And oid De elope Si e, Came a Face T acking
h p://de elope .and oid.com/guide/ opics/media/came a.h ml# ace-
de ec ion Accessed: Oc 2014
[18] And oid De elope Si e, Face De ec o class
h p://de elope .and oid.com/ e e ence/and oid/media/FaceDe ec o .h
ml Accessed: Oc 2014
[19] Viola, P.; Mi subishi Elec . Res. Labs., Camb idge, MA, USA ; Jones,
M., “Rapid objec de ec ion using a boos ed cascade o simple ea u es”
Compu e Vision and Pa e n Recogni ion, 2001. CVPR 2001.
P oceedings o he 2001 IEEE Compu e Socie y Con e ence on Vol 1
pp 511-518
[20] OpenCV documen a ion,
h p://docs.openc .o g/ unk/doc/py_ u o ials/py_objde ec /py_ ace_de
ec ion/py_ ace_de ec ion.h ml Accessed: Sep 2013
[21] Li Shuhua, Guo Gaizhi, “The applica ion o imp o ed HSV colo space
model in image p ocessing” Fu u e Compu e and Communica ion
(ICFCC), 2010 2nd In e na ional Con e ence on (Volume:2 ) 21-24
May 2010
[22] S.Suzuki, K.Abe., "Topological s uc u al analysis o digi al bina y
image by bo de ollowing," CVGIP.30(l): 32-46, 1985.
[23] Robe McMillan: How Google Re ooled And oid Wi h Help F om
You B ain, h p://www.wi ed.com/wi eden e p ise/2013/02/and oid-
neu al-ne wo k/ Accessed: Sep , 2013