Android based autonomous mobile robot

Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/ iim.2015.1-2/2. 1 And oid based au onomous mobile obo Ge gely Nagymá é Depa men o Mecha onics Op ics and Enginee ing In o ma ics, Budapes Uni . o Technology and Economics Budapes , Hunga y nagyma e.ge gel[email p o ec ed]m Abs ac —The sp eading o mobile obo s is ge ing mo e signi ican nowadays. This is due o hei abili y o pe o m asks ha a e dange ous, uncom o able o impossible o people. The mobile obo mus be endowed wi h a wide a ie y o senso s (came as, mic ophones, p oximi y senso s, e c.) and p ocessing uni s ha makes hem able o na iga e in hei en i onmen . This gene ally ca ied ou wi h unique, small se ies p oduced and hus expensi e equipmen . This pape desc ibes he concep o a mobile obo wi h a con ol uni in eg a ing he p ocessing and he main senso unc ionali ies in o one mass p oduced de ice, an And oid sma phone. The obo is able o pe o m asks such as acking colo ed objec s o human aces and o ien i sel . In he mean ime, i a oids obs acles and keeps he dis ance be ween he a ge and i sel . I is able o e bally communica e wi . Keywo ds— mobile obo , And oid sma phone, human- machine in e ac ion I. INTRODUCTION The compu ing capaci y o oday’s mobile phones is many imes g ea e han he one’s was used o Moon landing [1], ye commonly used o en e ainmen pu poses. These pocke compu e s a e equipped wi h many di e en kind o senso s ha a e also used in obo ics. Ad anced obo s o en need GPS, ine ial senso s and e-compass, o na iga ion and o ien a ion, came as o image p ocessing, mic ophone as audio inpu and wi eless connec i i y. These a e all included in sma phones. This pape no only d aws a en ion o sma phones o hei in eg a ed senso s, bu he eno mous so wa e suppo and open sou ce solu ions. The e ha e al eady been esea ches using sma phones in obo ics. A common applica ion uses sma phones as use in e aces o emo e con olling [2][3]. A semi au onomous obo p esen ed by S ans ield and Bo hma is s ill emo e con olled and has And oid de ice on bo h sides, bu on he obo side i is used o in e acing an ex e nal dis ance senso and also implemen s a simple collision a oidance algo i hm [4]. A mobile obo p esen ed by Lim, Lee and Tewolde is addi ionally o he p e ious obo uses he ine ial senso s o he sma phone o imp o e indoo na iga ion capabili ies o he obo [5]. Ano he obo akes ad an age o he came a in eg a ed in he sma phone o implemen ing image p ocessing o line ollowing [6]. Sma phone based mobile obo s a e used o educa ional pu poses, whe e sma phone based image p ocessing is also implemen ed [7] [8]. The e a e al eady obo pla o ms o sma phones ha a e comme cial p oduc s wi h easy o use de elope en i onmen designed o child en educa ional p og amming [9] [10]. The p esen p ojec aimed o build a mobile obo ha is capable o ul illing he ollowing objec i es while aking ad an age o he ad anced unc ionali y o an e e yday sma phone: 1) capable o acking human aces and colo ed objec 2) implemen s collision a oidance while ollowing i s a ge 3) capable o e bal communica ion wi h humans, ecei es o de s and answe s simple ques ions Such obo ha can unde s and human ques ions and in elligen ly answe hem seemed e y high ech a ew yea s ago, bu oday his echnology is in e e yone’s pocke [11] [12]. The obo is a ca -like ehicle wi h ul asonic dis ance senso s and a 2 DOF consol o holding he And oid phone. Two se o mo o s can change he yaw and pi ch angles o he phone in 180°. A hi d se o is esponsible o s ee ing he ehicle. Fig. 1 illus a es he mechanical cons uc ion o he obo . Fig. 1. Mechanical cons uc ion o he obo Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/ iim.2015.1-2/2. 2 In he iew o o ganiza ion he pape has he ollowing sec ions: Sec ion I was a sho in oduc ion. Sec ion II desc ibes di e en possibili ies o connec ing he sma phone o ex e nal ha dwa e. Sec ion III is abou he so wa e solu ions o he obo . Sec ion IV sho ly desc ibes he a chi ec u e o he so wa e. Sec ion V concludes he pape . II. CONNECTING THE SMARTPHONE TO THE ROBOT The p ojec was ca ied ou wi h a Samsung Galaxy Ace S5830 ype sma phone. This was a mid- ange de ice announced in 2011, Janua y. I has an 800 MHz ARM11 p ocesso and 278 Mb o RAM. The la es o icial so wa e upda e o i was And oid 2.3.3 Ginge b ead. In he iew o connec i i y his se up can ac as a USB sla e de ice. I can also be connec ed h ough Blue oo h o an ex e nal ha dwa e. Di ec Wi-Fi connec ion is only possible om And oid 4.0 and abo e [13]. Many mic ocon olle manu ac u e s o e now And oid accesso y amewo ks o hei IC-s [14]. These es ablish communica ion h ough USB be ween he And oid de ice and he embedded sys em. In he USB communica ion he e is always a hos con olle and a sla e de ice [14]. The hos powe s he bus while usually he sla e de ice ep esen s simple unc ionali y. De ices unning And oid 3.1 and abo e can ac as USB hos con olle . Olde sys ems – such as in his p ojec – can only ac as sla es. The e o e, USB communica ion equi es an ex e nal mic ocon olle wi h USB hos pe iphe al. This solu ion can bene i du ing longe ope a ions o he obo , because he obo can cha ge he And oid de ice. Howe e , wi eless connec ion is always o e s wide accessibili y. Wi h he abo e desc ibed se up, he communica ion can be ca ied ou wi h Blue oo h as well. This does no equi e Blue oo h module in he embedded sys em, bu a USB-Blue oo h adap e can be used. In he desc ibed sys em a Mic ochip PIC24FJ256DA mic ocon olle was used. The mic ocon olle implemen s an open sou ce amewo k called IOIO ha handles he USB pe iphe al and suppo s he Blue oo h communica ion [15]. The amewo k comes wi h an And oid lib a y, which o e s high le el unc ions o ini ializing and ope a ing digi al IO-s, PWM ou pu s, UART pe iphe al and ime unc ions in he mic ocon olle . This means ha unc ionali ies can be p og ammed h ough he And oid p og am. Fig. 2 shows he layou plan o he sys em. Fig. 2. Layou o he ex e nal sys em. The cha ging cu en can be se wi h a po en iome e . Blue oo h adap e equi es maximum cu en , while di ec USB communica ion can be ope a ional wi h minimal cu en , hus sa es obo ba e y III. USED SOFTWARE SOLUTIONS AND OTHER POSSIBILITY A. Image p ocessing The main unc ions o he obo a e ace acking and colo blob acking. These asks can be sol ed wi h ex e nal image p ocessing so wa e lib a ies such as OpenCV [16] o wi h na i e And oid unc ions. 1) Face acking In he case o ace acking, he me hods desc ibed in his sec ion can be compa ed in he pe spec i e o ame a e, and he p ecision o ecogni ion. Spon aneous alse posi i e ecogni ions can be il e ed ou wi h low pass il e . In case o sho ime alse nega i e esul s du ing acking he obo will look o he ace in he p e iously iden i ied egion o in e es . 2) Face acking wi h na i e And oid unc ions Abo e And oid 4.0 he e is a Face de ec o unc ion in he came a class [17]. This inds aces in he came a p e iew image, hus i na i ely se es wi h he highes ame a e. The unc ion’s e u n a ay con ains he posi ion o he aces in he p e iew image. This ce ainly he mos op imized solu ion bu was no ye a ailable due o he sys em se up. Ano he na i e solu ion o ace de ec ion in And oid is he indFaces unc ion, which is a ailable in e e y And oid e sion [18]. This does no wo k on he li e p e iew image bu needs a bi map image o p ocessing and is ela i ely slow. 3) Face acking wi h OpenCV The OpenCV ace de ec ion uses Viola-Jones objec de ec ion algo i hm [18], which is a Haa - ea u e based cascade classi ie [20]. The ope a ion and aining o he de ec o is desc ibed in de ails by Viola and Jones in hei wo k on apid objec de ec ion [19]. OpenCV comes wi h many p e- ained classi ie o ace, eyes, smile e c. These a e s o ed in XML iles and can be simply loaded in o he p og am. The OpenCV p o ides a wide scale o algo i hm cus omiza ion compa ed o he p e iously men ioned ace de ec o s. The image o be p ocessed can be downscaled, which educes p ocessing ime pe ame. Also he size o he smalles de ec able ace can be gi en. The smalle he ace we wan o de ec he mo e imes he algo i hm has o scan he Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/ iim.2015.1-2/2. 3 image wi h inc easingly smalle de ec ion window. By a ying hese wo ea u es an op imum can be ound whe e he obo can sense aces in an accep able dis ance, while p o iding su icien ame a e. 4) Colo blob acking The inpu image o he colo blob acke algo i hm is coded in RGB colo space. In he i s s ep his is con e ed in o HSV colo space. HSV colo model is a me hod o de ine colo s acco ding o he h ee basic ea u es o he colo : hue, sa u a ion and luminance [21]. In he p esen applica ion he obo ollows a colo ed objec . The changes o ligh condi ions due o he en i onmen and obo o ien a ion a ec he de ec ed colo s. These changes in RGB colo space a ec all he h ee colo ea u e, whe eas in HSV colo space hese mos ly a ec he luminance componen o he colo . This ea u e o he HSV colo space allows a mo e useable colo based segmen a ion. The hue o he colo can be mo e speci ied, while a wide ole ance can be se o he luminance ea u e. The algo i hm unde -samples he image i.e. educes esolu ion o educe p ocessing ime. C ea es a new bina y ma ix whe e only ills hose posi ions, whe e he co esponding pixel alls be ween he allowed h esholds in hue, sa u a ion and luminance. This bina y mask is expanded o he o iginal size o he image. A bo de ollowing algo i hm de eloped by Suzuki and Abe [22] inds he con ou s a ound he mask and e u ns an a ay wi h he bounding ec angles o he indi idual con ou s. The obo wan s o ollow he la ges objec ha ma ches he a ge colo , he e o e akes he la ges con ou as he ep esen a ion o he ollowed objec . I calcula es he di e ence be ween he cen e poin o he de ec ed colo blob and he cen e o he p ocessed image as he egula o ’s e o (Fig. 3). The e o signal o he objec ollowe con olle is calcula ed simila ly du ing ace acking wi h he la ges , hence closes ace. Table I. summa izes he ame a e esul s be ween he expe imen ed colo blob and ace de ec o me hods in espec o hei di e en pa ame e se ups. Fig. 3. Sma phone sc een du ing colo acking TABLE I. FPS VALUES WITH DIFFERENT TRACKING ALGORITHMS (THE VALUES ARE IN FRAME/SECOND) And oid na i e ace de ec o 0,75 OpenCV ace de ec ion Face size 40% 30% 20% 10% Image a io 100% 5.1 4.2 2.2 1.2 50% 7.3 6.3 4.78 3.75 30% 9.2 7.3 5.1 4.9 20% 9.11 9.3 9.25 5.86 Colo blob de ec ion 4,75 B. Speech ecogni ion and e bal esponse Speech ecognize se ice and ex - o-speech engine a e basic se ices in And oid, and can be easily implemen ed in any p og am. The speech ecognize equi es in e ne connec ion, because he p ocessing is done in he cloud [23]. I e u ns a se o he possibly hea d wo d combina ions. The gi en o de o he obo is alida ed i he e u ned se con ains i o some hing e y simila . The obo answe s o each ins uc ion using he ex - o-speech engine, and no i ies i he ins uc ion was misunde s ood. I can also answe s o some basic ques ions om a p ep og ammed answe bank. The e bal o de s a e also used o changing be ween he ace acking and objec acking. Fo a speci ied o de he obo akes a colo sample om he objec held on o i s came a and will ollow i . The speech ecognize can be igge ed wi h a sho whis le. IV. SOFTWARE ARCHITECTURE The so wa e builds on pa allel unning h eads. The wo main h eads a e he image p ocessing and he mic ocon olle in e ace h ead ha eads he ex e nal senso s and d i es he obo . These wo h eads communica e o each o he h ough an in e media e con olle objec . The e a e wo mino h eads esponsible o whis le de ec ion. The analog digi al con e e module o he mic ophone is capable o 44100 Hz sampling equency on 16 bi s. One h ead eco ds hese samples and eeds hem o he whis le de ec o h ead in 2048 sample big packages. The de ec o han applies a Fas Fou ie T ans o ma ion on he da ase . Spikes be ween 1 and 3 kHz a e conside ed as whis les. When a whis le is de ec ed he speech ecognize se ice is being launched, bu i s he eco ding has o be s opped o elease he mic ophone as sys em esou ce. The in e media e con olle objec is esponsible o he e en ual acking o he de ec ed objec s. I also implemen s a s a e machine o enhanced collision a oidance algo i hm. The image p ocessing h ead does no supply he da a wi h ixed sampling a e, hus he sys em clock is used o es ablish he elapsed ime o he e ical and ho izon al PID con olle s. Fig. 4 summa izes he so wa e o ganiza ion. Recen Inno a ions in Mecha onics (RIiM) Vol. 2. (2015). No. 1-2. DOI: 10.17667/ iim.2015.1-2/2. 4 Fig. 4. So wa e a chi ec u e V. CONCLUSION This pape p esen ed a possible usage o sma phones in mobile obo ics as complex p ocessing uni s and senso a ays. Usage o sma phones in obo ic applica ions can ex end he capabili ies o obo s compa ed o con en ional embedded sys ems. Howe e , wi h mid- ange phones he esponse ime o he sys em will be oo long. Simul aneous unning o ad anced unc ions migh esul s phase shi on he eal- ime beha io . Highe -end sma phones can ex end he embedded compu e s o mobile obo s by aking ad an ages o all bene i s o mass p oduc ion and open sou ce in he aspec s o cos s, so wa e and ha dwa e de elopmen . ACKNOWLEDGMENT The au ho would like o hank o Ri a M. Kiss and Pé e Tamás o hei commen s, which g ea ly imp o ed his pape . REFERENCES [1] Eldon C. Hall, (1996), Jou ney o he Moon: The His o y o he Apollo Guidance Compu e , Res on, Vi ginia, USA: AIAA, p. 196, ISBN 1- 56347-185-X [2] C. Pa ga, Xiaoou Li, Wen Yu, “Sma phone-Based Human Machine In e ace wi h Applica ion o Remo e Con ol o Robo A m”, Sys ems, Man, and Cybe ne ics (SMC), 2013 IEEE In e na ional Con e ence on Oc . 2013, pp 2316-2321 [3] S. W. Moon, Y. J. Kim, H. J. Myeong, C. S. Kim, N. J. Cha, D. H. Kim, “Implemen a ion o sma phone en i onmen emo e con ol and moni o ing sys em o And oid ope a ing sys em-based obo pla o m”, Ubiqui ous Robo s and Ambien In elligence (URAI), 2011 8 h In e na ional Con e ence on No 2011, pp 211-214 [4] N. S ans ield, B. Bo hma, “And oid based semi-au onomous collision a oidance obo ”, Robo ics and Mecha onics Con e ence (RobMech), 2013 6 h, pp 135-139 [5] J. Lim, Seok Ju Lee, G. Tewolde, J. Kwon, “Ul asonic-senso deploymen s a egies and use o sma phone senso s o mobile obo na iga ion in indoo en i onmen ”, Elec o/In o ma ion Technology (EIT), 2014 IEEE In e na ional Con e ence on June 2014, pp 593-598 [6] Nolan He ge , William Keyes, and Chao Wang, "Sma phone-based Mobile Robo na iga ion" Depa men o Elec ical and Compu e Enginee ing, Ca negie Mellon Uni e si y, May 9 h, 2012 [7] C. K o i sch, C. Hinge ; M. Me dan, G. Koppens eine , “Sma phone d i en con ol o obo s o educa ion and esea ch”, Robo ics, Biomime ics, and In elligen Compu a ional Sys ems (ROBIONETICS), 2013 IEEE In e na ional Con e ence on No . 2013, pp 148-154 [8] D. Yang, J.-K. Lim ; Y. Choi, “Ea ly childhood educa ion by hand ges u e ecogni ion using a sma phone based obo ” Robo and Human In e ac i e Communica ion, 2014 RO-MAN: The 23 d IEEE In e na ional Symposium on Aug. 2014, pp 987-992 [9] Romo webpage: h p://www. omo i e.com/ Accessed in No 2014 [10] Sma bo webpage: h p://www.o e d i e obo ics.com/ Accessed in Aug 2014 [11] Apple Inc. h p://www.apple.com/iphone/ ea u es/si i.h ml. Accessed: Oc 2014 [12] Google Mobile. h p://www.google.com/mobile/ oice-ac ions Accessed: Oc 2014 [13] And oid De elope Si e – Wi-Fi Pee - o-Pee h p://de elope .and oid.com/guide/ opics/connec i i y/wi ip2p.h ml Accessed: No 2014 [14] Flowe s, D.: And oid accesso y de elopmen wi h he Google Open Accesso y F amewo k. ELEKTRONET (2012, No embe ), pp 24-25. [15] IOIO Documen a ion h ps://gi hub.com/y ai/ioio/wiki Accessed: Oc 2014 [16] OpenCV Use Si e: h p://openc .o g/ Accessed: Oc 2014 [17] And oid De elope Si e, Came a Face T acking h p://de elope .and oid.com/guide/ opics/media/came a.h ml# ace- de ec ion Accessed: Oc 2014 [18] And oid De elope Si e, Face De ec o class h p://de elope .and oid.com/ e e ence/and oid/media/FaceDe ec o .h ml Accessed: Oc 2014 [19] Viola, P.; Mi subishi Elec . Res. Labs., Camb idge, MA, USA ; Jones, M., “Rapid objec de ec ion using a boos ed cascade o simple ea u es” Compu e Vision and Pa e n Recogni ion, 2001. CVPR 2001. P oceedings o he 2001 IEEE Compu e Socie y Con e ence on Vol 1 pp 511-518 [20] OpenCV documen a ion, h p://docs.openc .o g/ unk/doc/py_ u o ials/py_objde ec /py_ ace_de ec ion/py_ ace_de ec ion.h ml Accessed: Sep 2013 [21] Li Shuhua, Guo Gaizhi, “The applica ion o imp o ed HSV colo space model in image p ocessing” Fu u e Compu e and Communica ion (ICFCC), 2010 2nd In e na ional Con e ence on (Volume:2 ) 21-24 May 2010 [22] S.Suzuki, K.Abe., "Topological s uc u al analysis o digi al bina y image by bo de ollowing," CVGIP.30(l): 32-46, 1985. [23] Robe McMillan: How Google Re ooled And oid Wi h Help F om You B ain, h p://www.wi ed.com/wi eden e p ise/2013/02/and oid- neu al-ne wo k/ Accessed: Sep , 2013