Control Mouse Cursor by Head Movement: Development and Implementation

Applied Medical In o ma ics O iginal Resea ch Vol. 40, No. 3-4 /2018, pp: 39-44 39 [ Con ol Mouse Cu so by Head Mo emen : De elopmen and Implemen a ion Hussein Ali ALHAMZAWI* Facul y o In o ma ics, Uni e si y o Deb ecen, Kassai S ee , no. 26, 4028 Deb ecen, Hunga y. E-mail: alio .mezhe @gmail.com; * Co esponding au ho : Tel.: +36304390299 Recei ed: Feb ua y 7, 2018 /Accep ed: Augus 29, 2018/ Published online: Sep embe 5, 2018 Abs ac As he e is an excellen ad ancemen in he echnology in ecen yea s, he e has been much imp o emen in a ious ields o compu ing such as Human Compu e In e ace (HCI), Compu e Vision and Pe cep ual Use In e ace (PUI). The inpu o he compu e s has sensed in o ma ion abou he physical p ope ies o he use , places, o hings. Fo example, a compu e mouse ope a es by mo ion impa ed by he use ’s hand. Many ac i e esea che s ha e been wo king on inding an al e na i e solu ion o con ol compu e mouse like by using a inge , eye, hand/palm, ma ked glo es, e c. These all echniques may no be sui able o physically disabled people. The sys em o con ol he mouse cu so by he mo emen o he head o a pe son who is posi ioned in on o he moni o o a mic ocompu e is p esen ed in his pape . A he op o a moni o is a colo ideo came a wi h he s anda d Na ional Tele ision Sys em Commi ee (NTSC) ou pu ha cap u es use images. These images a e p ocessed using digi al image p ocessing echniques, and he inal objec i e is o p opo ionally ela e he use 's head mo emen s (iden i ied by a blue ci cle pain ed on a assel) wi h he mouse cu so . The cap u ed o he colo ed images a e 320 pixels by 240 wide by high. P elimina y esul s ha e shown e iciency in con olling he posi ion o he cu so on he compu e sc een. Keywo ds: Quad iplegic; Segmen a ion; Bina iza ion; Digi al Image P ocessing (PDI) echniques In oduc ion An es ima ed wo million people li ing wi h spinal co d inju ies (SCI) and e e y 49 minu es ano he su e s new inju y [1]. O his o al, 48.7% a e he quad iplegic. To acili a e he accomplishmen o hei daily ac i i ies and o p o ide a mo e p oduc i e and happy li e, a home, a school o wo k, i is u gen o c ea e new u ensils, de ices, and equipmen o hese use s. Mo eo e , oday he compu e a means o leisu e, s udy, wo k and communica ion, new ools ha e been c ea ed o imp o e hei accessibili y [2] and make i mo e e icien in he new asks ha a ise. Resea che s ha e ied in he ield o obo ics and human-compu e in e ac ion o con ol mouse mo emen s using ideo de ices. Almos all esea che s use di e en me hods o make he e en clickable. E dem e al. [3] use inge acking o con ol mouse mo emen . The mouse bu on o e he implemen a ion o he sc een ha has been de ined so ha a click e en when he use 's hands passed h ough he a ea. Zhou, Hailing, Lijun Xie, and Xuliang Fang [4] used inge ips o con ol he mouse poin e and click. This me hod depends on he in ensi y o he image and is equi ed o hold he mouse poin e on he desi ed poin o a sho pe iod. Lien, Chu-Feng, Used ano he me hod o clicking. He used he mo ion o he humb ( om humbs-up posi ion o is ) o ma k a clicking e en humb and he mo emen o he hand o make a pa icula hand sign mo e o mo e mouse poin e [5]. In his wo k, use heade mo emen s a e se in he mouse inpu . The webcam is se o ake pic u es con inuously. In he use 's head, he e mus be a blue colo ed segmen so i would Hussein A. ALHAMZAWI 40 Appl Med In o m 40(3-4) Sep embe -Decembe /2018 be isible o he came a when aking a pic u e . This colo o he pixel image is de ec ed, and he pixel posi ion is se in he mouse inpu . Depending on he size o he image aken by he came a he pixel posi ion in he image will no ha e a co espondence wi h sc een esolu ion. The e o e a ious scaling echniques a e used. HeadMouse® [6], om O igin Ins umen s Co p., has as p incipal componen a 94×56×13 mm USB de ice which con ains an op ical senso and In a ed LEPs. Some Disposable e lec will be a ached o he head o he use ; an in a ed op ical senso will be de ec ed his ma ks as a do s. A mouse poin e mo emen on he compu e is coming om he p opo iona e ansla ion o he head mo emen s. On he op o he compu e sc een, he de ice can be ins alled. Whe e a “pe ec ule humb is o place he HeadMouse so ha he eye le el is nea , and look di ec ly a he poin . F om his, a maximum ange o mo ion will p o ide o he use . F om he O igin Ins umen s, he ollowing ex ensions can be pe o med using Mouse click: • On a headse , a sip/pu swi ch is moun ed • Dwell clicking wi h he so wa e D agge By using hi d-pa y p oduc s (e.g., swi ches), a mouse clicks can also be pe o med. The windows so wa e D agge 95$, he sip/pu swi ch cos s 295$ and he main componen , he Head Mouse Ex eme cos s 995$. The disad an age o he HeadMouse a e he need o ex a equipmen and he p ices o his ex a-equipmen s ha a e expensi e. Sma -Na ®, o Na u alPoin Inc., uses LEDs posi ioned on he on o he use ; he ligh emi ed is acqui ed by a came a ha is on he moni o , and an algo i hm calcula es he posi ion o he cu so [7]. The disad an age o his me hod is:  E a ic acking, cu so "jumps" a ound he sc een.  In some case So wa e loads and uns and g een indica o ligh blinks, bu he cu so does no mo e.  In some cases, Unable o p oceed pas d i e signing du ing ha dwa e ins all.  In some case, All o some o he Ho Keys will no wo k. Wi h he impossibili y o using he uppe limbs, quad iplegic people a e dep i ed o he use o pe sonal compu e s. The p ima y objec i e o his p ojec was o c ea e a con ol sys em o mouse cu so by mo ing he head o a use posi ioned in on o a compu e o assis hese pa ien s in using compu e s and help him/he in his/he ou ine li e. Me hodology A con ol sys em o mouse cu so has been designed by mo ing he head o a use posi ioned in on o a compu e unning Windows 95 (o highe ) ope a ing sys em. On he moni o , a colo ideo came a (s anda d NTSC) is placed ha cap u es images o he head o he use and, using digi al image p ocessing (DIP) echniques, con e s hem in o in o ma ion necessa y o he mouse cu so o mo e in he eques ed di ec ion. In he de eloped design, a spo s- ype ape is placed on he use 's head (Headband). In he cen e o his ape, he e is a blue pain ed ci cle, which is he e e ence poin ha he camco de should look o when cap u ing an image. This sea ch o he cen oid is pe o med using PDI echniques. Once he coo dina e is loca ed, he mouse cu so is mo ed o i s new posi ion, which is p opo ional o he cen oid displacemen o he ci cle in he cap u ed image. Figu e 1 illus a es he implemen ed sys em. Any mo emen o he head, ho izon al o e ical, mo es he ci cle ixed on he es e . As he camco de is ecei ing images om he use , his shi will be e lec ed in he images, causing he cen oid o he ci cle o change coo dina es. This change will p opo iona ely a ec he posi ion o he mouse cu so on he compu e sc een. The simple mouse click is emula ed when he use si s a leas n seconds wi hou mo ing he head and double-clicking when n + 1 seconds is le . The de eloped applica ion was assembled in C++ p og amming language. A colo ideo came a WAT 202B om Wa ec Inc. was used. Mo eo e , a ideo signal cap u e boa d, CPH050 ame g abbe om T iew Inc., unning on an IBM / PC wi h Pen ium IV p ocesso , 1,700 GHz, and 256 Mby es o RAM. The cap u ed images we e 320 pixels by 240 (wide by heigh ), moun ed on he RGB colo model. Con ol Mouse Cu so by Head Mo emen : De elopmen and Implemen a ion [ Appl Med In o m 40(3-4) Sep embe -Decembe /2018 41 Figu e 1. The sys em o con ol he mouse by mo emen o he head. The blue ci cle placed on he use is he e e ence o he images cap u ed by he ideo came a on he moni o . DIP echniques de e mine i s cen oid in he image p opo ionally mo es he mouse cu so on he sc een o he desi ed posi ion. The algo i hm depic ed in Figu e (2) was implemen ed. Figu e 2. Flow cha o he sys em. Fo he de ec ion o head mo emen s o he use , a webcam was used ha will wo k as a senso . The head mo emen s o he use a e cap u ed a a ixed ame a io and esolu ion which is ob ained by he ha dwa e o he came a. By changing he HSV alue he ame a io and esolu ion can be changed i equi ed. These ope a ions a e iden i ied on he cap u ed images wi h he Open CV. The cap u ed ideo is di ided in o indi idual image ames based on he ames pe second o he came a, and indi idual image ames a e p ocessed. Colo ed images a e o med by he combina ion o n in o ma ion channels (n=3) [8]. The scanned images always he RGB, whe e R ep esen s he ed channel, G he g een channel, and B he blue channel. Howe e , he RGB ype is no bes sui ed o segmen a ion and o his eason, con e sion om RGB o HSI is pe o med, whe e H ep esen s he hue o he colo , S is sa u a ion, and I is in ensi y (b igh ness). The pixel ma ix alue is gi en in deg ees, whe e 0° means ed, 60° yellow, 120° g een, and so on. To ind egions o he image wi h pixels o he blue colo , we only need o look o hues ha ha e alues close o 240º. A new image is hen gene a ed, whe e blue pixels a e ma ked as whi e and he o he s as black pixels. F om a bina ized image whe e whi e pixels a e ea ed as objec s and black pixels as backg ound, he echnique p oposed by Hough [9] and Pa ke [10], now called he Hough T ans o m, is ecognized as being one o he mos e ec i e in loca ing exis ing geome ic shapes in images [11]. Ci cula geome ic shapes can be iden i ied by Equa ion (1). Acqui ing he eal- ime ideo om he webcam con e ideo in o he image Blue colo segmen a ion Loca ion o he ci cle and mouse cu so s mo emen and e en Hussein A. ALHAMZAWI 42 Appl Med In o m 40(3-4) Sep embe -Decembe /2018 𝑝 = 𝑘√k2− c , o k2− c ≥ 0 (1) whe e p = dis ance, θ = he pola coo dina e o he cen oid o he ci cle ela i e o he posi ion (0, 0) o he image, uppe le co ne ,k = xcos(θ) + ysin(θ) and c = x2+y2- 2, = he adius o he ci cle. The (x,y) coo dina e e e s o any whi e pixels in he image. The a iable ep esen s he adius o he ci cle o be loca ed which mus a y in a p ede e mined in e al, acco ding o he dimensions o he image. The dis ance p and he angle θ ep esen he pola coo dina e o he cen oid o he ci cle ela i e o he posi ion (0, 0) o he image, uppe le co ne . The angle should also a y wi hin a speci ic ange. The esul o his ans o m is a 3D a ay, whe e one dimension is ep esen ed by , ano he by θ and he las by p. In his a ay, he coo dina e elemen s ( , θ, p) ha ha e highe alues a e iden i ied as ci cles o adius , and he pola coo dina e o he cen oid (θ, p). P ocessing is e mina ed when con e ing he coo dina e o he pola ep esen a ion o ec angula [12,13]. The ci cle localiza ion algo i hm by he Hough T ans o m had he ollowing inpu pa ame e s: 10 ≤ ≤ 25 pixels and 15 ≤ θ ≤ 75 deg ees. The de eloped sys em was es ed only in he labo a o y on people wi hou any mo o p oblems and espec i ely wi h spinal co d inju y. Resul s and Discussion The e lec ion o he head mo emen on he mouse cu so on he sc een is depic ed in Figu e 3. Fo he click o he mouse, n = 2 s was adop ed. The p ocessing ime o all sys em s eps was less han 67ms, wi h a ame cap u e a e o 15/s. Figu e 3. The e lec ion o he head mo emen on he mouse cu so on he sc een. A dx di e ence in he cen oid is ela ed o aαdx o se a he cu so , and a di e ence dy causes a βdy o se . The cons an s α and β de e mine he sensi i i y o he sys em. The igh and le images a e e e sed because he camco de cap u es mi o images. The p oposed sys em cap u es he image h ough a webcam so, i he web came a canno cap u e he image accu a ely, hen he ope a ion canno pe o m. In some cases due o he p esence o o he colo s in he backg ound, an e o can occu a ec ing he h eshold alues and o he pa ame e s o he sys em. Fu he mo e, he sys em migh un slow due o he high esolu ion o he came a [14]. The use should always si nea e o he desk op, because o ex e nal ha dwa e mouse and esolu ion o he came a. I he mouse is damage in any si ua ion, hen cos will inc ease. The esul s o he e alua ions pe o med a e p esen ed in Table 1 and 2. The es s we e conduc ed on h ee ideos om ow olun ee s he i s one was heal hy, he second was spinal co d. Al hough he accu acy o he sys em is ou s anding in con olled en i onmen s, we ind many d awbacks and limi a ions when use beha io and he en i onmen is unp edic able:  High e o a io on as mo ion anima ion. Con ol Mouse Cu so by Head Mo emen : De elopmen and Implemen a ion [ Appl Med In o m 40(3-4) Sep embe -Decembe /2018 43  I en i onmen al ligh s a e changing, alse segmen a ion o blue colo is esul ing in poo a ali y classi ica ion o click e en s. Table 1. Accu acy o blue ci cle de ec ion # To al ames # De ec ed ames Accu acy (%) Video1 892 801 88.72 Video2 750 698 91.23 Video3 960 932 96.22 A e age Accu acy (%) 95 Table 2. Accu acy o click e en de ec ion #To al ames #Click e en ames #De ec ed ames Accu acy (%) Video1 889 62 51 80.36 Video2 755 44 40 87.13 Video3 953 92 81 87.1 A e age Accu acy (%) 84.86 The main ad an ages o he p oposed sys em a e as ollow:  Help people who ha e damage in hei a ms o use he compu e .  The Vi ual Keyboa d is no es ic ed o he QWERTY ouch- yping pa adigm; adjus men s can be made o he so wa e o i o he ouch- yping pa adigms as well.  No d i e s a e necessa y; i can use a Plug and Play de ice.  Po abili y.  Cos -e ec i eness. Ha ada e al. [14], and espec i ely Spo ka e al. [16] epo ed he used o oice con ol o mouse poin e bu hei app oaches a e alid jus when he used ha e oice. Ou p oposed me hod add essed o hose subjec s who did no ha e any oice con ol. Func ionali ies o he p esen ed con ol mouse cu so sys em a e o de ec and ack he objec , hen ge he posi ion o he head and con e his head’s mo emen o cu so coo dina es. De ec ion and acking he blue ci cle in headband using he low- esolu ion came a wi h he cheap equipmen and easy algo i hm o ge igh and almos p ecisely esul in ha help he people ha ha e unable o mo e hem uppe limb o using he compu e as he heal hy people is p esen ed in his pape . We in oduced a new me hod in ou pape o con olling a mouse cu so by using a blue segmen a ion placed on he use ’s o ehead. Two me hods o cu so con ol, eloci y-con ol, and posi ion-con ol. We also compa ed along wi h desk op Mouse. The mean h oughpu s o he desk op mouse cu so we e 4.42 bps, while h oughpu s o he eloci y con ol modes and posi ion we e 1.07 bps and 1.61 bps espec i ely. Noise in ick disclosu e was a de e mina ion o he posi ion-con ol p ocess. Howe e , in o de o speed con ol is less conjec u al, pe o mance mode con ol speed is wo se han he posi ion con ol p ocess. Imp o ed mo emen o he posi ion-con ol would be he bes way o deal wi h noise de ec ion ma k, o imp o ed accu acy and dec easing he ji e . The e is o app oach o deal wi h his p oblem: Fo a mo e e ec i e way o de ec signs and de e mine he di ec ion o he ma k, and Ge a be e smoo hing algo i hm ha educes he noise a e he o ien a ion has been calcula ed. Con lic o In e es The au ho decla es ha he ha e no con lic o in e es . Hussein A. ALHAMZAWI 44 Appl Med In o m 40(3-4) Sep embe -Decembe /2018 Re e ences 1. NSCIA. Abou Spinal Co d Inju y. 2003. [online]. A ailable om: www.spinalco d.o g (Accessed 12/10/2017). 2. The F eedom o Connec ! © 2012, No h Dako a Assis i e. [online] A ailable om: h p://ndipa .o g/compu e -access (Accessed 14/08/2018) 3. E dem A, E dem E, Ya dimci E, A alay Y, Ce in AE. Compu e ision based mouse. 2002 IEEE In e na ional Con e ence on Acous ics, Speech, and Signal P ocessing, O lando, FL, 2002, pp. IV-4178- IV-4178. doi: 10.1109/ICASSP.2002.5745637. 4. Zhou H, Lijun X, Xuliang F. Visual mouse: Si de ec ion and PCA ecogni ion. 2007 In e na ional Con e ence on Compu a ional In elligence and Secu i y Wo kshops (CISW 2007), Heilongjiang, 2007, pp. 263-266. doi: 10.1109/CISW.2007.4425487. 5. Lien C-F. Po able Vision-Based HCI–A Real ime Hand Mouse Sys em on Handheld De ices. [online] A ailable om: h ps://www.csie.n u.edu. w/~p93007/p ojec s/ ision/ ision_hci_p93922007.pd (Accessed 12/10/2017) 6. HeadMouse® Nano Wi eless Head Con olled Mouse [online]. A ailable om: h p://www.o in.com/access/headmouse/ (Accessed 14/08/2018) 7. Sma -Na ® [online]. A ailable om: h ps://www.na u alpoin .com/sma na / (Accessed 14/08/2018) 8. Gonzalez RC, Woods RE. Digi al Image P ocessing. Bos on, Addison-Wesley Publishing, 1993. 9. Hough PVC. Me hod and means o ecognizing complex pa e ns.U.S. Pa en 3,069,654, Dec. 18, 1962. 10. Pa ke JR. Algo i hms o Image P ocessing and Compu e Vision. New Je sey, Wiley Compu e Publishing, 1997. 11. B adski G, Kaehle A. Lea ning OpenCV: Compu e ision wi h he OpenCV lib a y. O'Reilly Media, Inc., USA, 2008. 12. Bloch I, Hen i M. Fuzzy ma hema ical mo phologies: a compa a i e s udy. Pa e n Recogni ion 1995;28(9):1341-1387. 13. Homma K, Takenaka E. An image p ocessing me hod o ea u e ex ac ion o space-occupying lesions. J Nucl Med. 1985;26(12):1472-1477. 14. Be ke M, Gips J, Fleming P. The came a mouse: Visual acking o body ea u es o p o ide compu e access o people wi h se e e disabili ies. EEE T ansac ions on Neu al Sys ems and Rehabili a ion Enginee ing 2002;10(1):1-10. doi: 10.1109/TNSRE.2002.1021581 15. Ha ada S, Landay JA, Malkin J, Li X, Bilmes JA. The ocal joys ick:: e alua ion o oice-based cu so con ol echniques. P oceedings o he 8 h in e na ional ACM SIGACCESS con e ence on Compu e s and accessibili y, 2006, pp. 197-204. 16. Spo ka AJ, Ku niawan SH, Sla ík P. Acous ic con ol o mouse poin e . Uni e sal Access in he In o ma ion Socie y 2006;4(3):237-245.