Preliminary results for a monocular marker-free gait measurement system

Ad ances in Elec ical and Elec onic Enginee ing 218 PRELIMINARY RESULTS FOR A MONOCULAR MARKER-FREE GAIT MEASUREMENT SYSTEM J. Cou ney, A. de Pao School o Elec ical, Elec onic and Mechanical Enginee ing Na ional Uni e si y o I eland, Dublin, Bel ield, Dublin 4, Rep. o I eland. E-mail: Jane.Cou ney@di .ie Summa y This pape p esen s esul s om a no el monocula ma ke - ee gai measu emen sys em. The sys em was designed o physical and occupa ional he apis s o moni o he p og ess o pa ien s h ough he apy. I is based on a no el human mo ion cap u e me hod de i ed om model-based acking. Tes ing is pe o med on wo monocula , sagi al- iew, sample gai ideos – one wi h bo h he en i onmen and he subjec ’s appea ance and mo emen es ic ed and one in a na u al en i onmen wi h un es ic ed clo hing and mo ion. Resul s o he modelling, acking and analysis s ages a e p esen ed along wi h s anda d gai g aphs and pa ame e s. 1 INTRODUCTION 1.1 Mo i a ion This pape is p esen ed as pa o a gai measu emen design p ojec . The goal o his p ojec is o design a sys em, o he use o occupa ional and physical he apis s, which would cap u e and analyse human gai . Cu en me hods o gai measu emen in ol e complex ma ke sys ems and a mul iple came a se -up, he eby equi ing a dedica ed gai labo a o y and ained he apis s, making he sys ems cumbe some and di icul o use. He e we ha e designed a simple single-came a sys em which is no only accu a e bu also has a low p ocessing ime and can be used emo e om he ilming loca ion, he eby elimina ing he need o pa ien s o a el o a gai labo a o y. I is hoped ha he simplici y o he sys em will encou age bo h he apis s and pa ien s o pa icipa e in gai s udies and make he mos o he echnology a ailable. 1.2 Ma ke -based Sys ems Ma ke -based sys ems a e s ill he mos eadily used me hod o gai analysis. Howe e , hey a e ex emely di icul o use and p oblema ic, equi ing speci ic equipmen and expe ise. This makes he sys ems less po able and less accessible ou side o a gai labo a o y which can be a signi ican issue when pa ien s a e oo unwell o a el. In addi ion, a gai labo a o y can be a e y in imida ing en i onmen , which can make pa ien s eel uncom o able – a majo issue pa icula ly when dealing wi h young child en and elde ly pa ien s. In o de o a oid ma ke mo emen , ma ke s canno be placed on clo hing as his will mo e ela i e o he join s and bones being ma ked. The pa ien mus be s ipped and he ma ke s a ached di ec ly o hei skin. This again adds o he discom o ha pa ien s can eel in a gai labo a o y en i onmen . Many gai pa ien s a e elde ly s oke ic ims who do no eel a all com o able walking in hei unde wea . One o he majo mo i a ions o ou design was o elimina e he need o s ip he pa ien . E en wi h expe ience and knowledge, ma ke placemen is s ill a di icul ask. The posi ion o he ma ke s will ha e a signi ican e ec on he ou pu o he sys em. E en sligh inaccu acies, pa icula ly a ound he join s, can cause he sys em o ail. Once he ma ke s ha e been placed accu a ely o begin wi h, hey mus be kep in posi ion as he subjec walks. They mus no in e e e wi h he eedom o mo emen o he pa ien and he sys em mus be in ulne able o ma ke occlusion. The equipmen equi ed o measu emen can be uncom o able o wea and can ha e a signi ican e ec on he subjec ’s eedom o mo emen . Ac i e ma ke sys ems equi e a ansmi e , powe supply and wi ing o be wo n by he pa ien as hey walk. Passi e ma ke s a e o en moun ed on special suppo s o p o uding s icks o make hem mo e isible in he image. Again, his can es ic he subjec ’s mo emen and limi s he posi ions in which ma ke s can be placed ( o ins ance, p o uding ma ke s canno be placed on he inside o he subjec ’s legs as hey will easily be knocked while walking). This cumbe some equipmen is no only di icul o a ach bu could ha e a signi ican e ec on he mo emen o he subjec . While passi e ma ke sys ems a e less in usi e, hey equi e mo e ma ke s o compensa e o hei ulne abili y o occlusion. Since passi e ma ke s a e e lec i e, hey e ec i ely disappea when hey a e blocked om he in a- ed ligh . This happens qui e egula ly du ing walking as he subjec ’s a m na u ally swings back and o h, he eby occluding any ma ke s a ound he pel ic egion. All o hese issues make a ma ke -based sys em di icul o design and ope a e. Ins ead, we en isage a sys em ha will be so simple o use ha pa ien s could be moni o ed in any local clinic, hospi al, su ge y o e en in hei own homes. In ac , occlusion is an issue e en wi h ma ke - ee sys ems. Many a emp s a designing ma ke - ee sys ems ha e been based on ea u e de ec ion and acking [1] o on appa en mo ion [2]. A he poin o c osso e o he legs, du ing he swing phase o gai , he image ea u es become less well de ined and i is di icul o iden i y any appa en mo ion. Al hough his is no echnically occlusion, he esul is he same: he P elimina y esul s o a monocula ma ke - ee gai measu emen sys em 219 acking cues a e los . In de eloping a new ma ke - ee sys em, his is a majo conside a ion. 1.3 Ma ke - ee Gai Analysis The e a e cu en ly many esea ch g oups s i ing o de elop he i s ully au oma ed ma ke - ee sys em. The e a e, in ac , al eady some comme cially a ailable ma ke - ee sys ems, e.g. [3]. Howe e , so a , none o he a ailable sys ems is comple ely au oma ed and hey s ill equi e a gai labo a o y en i onmen , se e al measu emen s o he subjec and/o manual in e en ion a a ious s ages in o de o ope a e eliably. While hese sys ems may su ice in o he applica ions, hey ha e no been eadily emb aced by he apis s as a be e al e na i e o ma ke -based sys ems in moni o ing pa hological gai . Al hough i may s ill be necessa y o use speci ic equipmen o acqui e ce ain da a, e.g. o ce- pla e measu emen s, we eel ha he e is no eason ha gai kinema ics, he in o ma ion gleaned om ideo da a, canno be measu ed emo e om a gai labo a o y. The g ea es challenge om a mo ion analysis poin o iew lies in analysis o he lowe limbs in he sagi al plane. In ac , his is also whe e he mos use ul in o ma ion is ga he ed o diagnosis and in e p e a ion o gai da a. In some basic ma ke - ee sys ems, he subjec is equi ed o wea di e en colou ed s ockings on each leg o dis inguish he wo legs om each o he , e.g. [4]. This s ays om he goal o a non-in usi e and comple ely ma ke - ee sys em. The di icul ies in he sagi al plane s em om he simila i y in appea ance and p oximi y o he wo legs and om he speed change du ing he swing phase o gai . As he legs c oss du ing he gai cycle, he image o he mo ing leg becomes blu ed and he mo ing leg becomes indis inguishable om he s a iona y leg. Wi h s anda d mo ion acking echniques, his can lead o mo ion ec o s ha ing e oneous ze o alues. This is why sagi al gai analysis is such a challenge when a emp ing au oma ed mo ion acking. Many cu en ma ke - ee echniques a e s ill being imp o ed upon in his a ea. Re iew pape s [5] and [6], co e ing he en i e a ea o human mo ion cap u e including ma ke -based and ma ke - ee analysis, can be consul ed o a mo e ho ough su ey o he cu en s a e o human mo ion esea ch. 1.4 Design Goals Wi h he in e es s o bo h pa ien s and he apis s in mind, we ou lined he ollowing goals o he p ojec : • The sys em mus be comple ely au oma ed equi ing no excessi e measu emen s o he pa ien and no manual in e en ion. • The sys em mus be simple o use. • The ou pu will be a comple e se o sagi al plane gai g aphs and pa ame e s. • The inpu will be a single AVI ile con aining a ilm o he pa ien walking. • The gai ideo can be ilmed in any easonable en i onmen wi hou signi ican es ic ions. • The subjec can be ully clo hed in app op ia e clo hing. • The subjec can walk eely. In addi ion, because his mo ion measu emen sys em has a speci ic applica ion, we can apply ce ain es ic ions o ou expec a ions: • I is easonable o expec adequa e ligh ing and con as in he ilming en i onmen . • The da a will be ilmed om a s a iona y came a. • The subjec will walk om one side o he came a iew plane o he o he . • The subjec will be ully isible in all ames om head o oe. • Clo hing will no hide he subjec ’s leg ou line, o example, ski s may no be wo n. • The heigh o he subjec is known. In designing ou sys em, we ied o mee as many o ou goals as possible whils minimising he es ic ions on he sys em. An ini ial design a emp was made p e iously bu he di icul y a he c osso e o he legs du ing he gai cycle could no be o e come and he un acked leg had o be manually emo ed om each ame in o de o p o ide esul s [7]. Since hen, howe e , a me hod has been disco e ed which ou lines he acked leg in each ame, he eby dis inguishing i om he un acked leg. This me hod has been in eg a ed in o he o e all sys em and adap ed o make i comple ely au oma ed and obus . The esul is a ully au oma ed acking sys em ha succeeds in eaching ou ou lined goals. 2 IMPLEMENTATION 2.1 Segmen a ion The human mo ion cap u e me hod used he e is based on a me hod in oduced by Nyogi and Adelson [8]. In hei ‘XYT’ me hod, ideo ames om a s a iona y came a a e s acked o c ea e a 3D block wi h wo o i s dimensions ep esen ing ho izon al and e ical di ec ions and i s hi d dimension being ime (See Figu e 1). Fig. 1: The XYT block o he gai labo a o y sequence. Ad ances in Elec ical and Elec onic Enginee ing 220 A pic u e o he mo emen in he ideo is ob ained by slicing he block in he XT di ec ion. In he XT slice, he s a iona y backg ound appea s as e ical lines and objec s mo ing ho izon ally ac oss he came a plane appea as diagonal lines. This image is pa icula ly use ul o ecognizing and analysing walking because o an in e es ing cha ac e is ic o he leg mo ion. In he case o a sagi al iew o a human walking in on o a s a iona y came a, a dis inc ly ecognisable b aided pa e n is obse ed in slices a ound he leg heigh (see Figu e 2). The b aids a e o med by he pe iodic mo ion o he legs as hey pass h ough he swing and s ance phases o he gai cycle. While he legs appea e y close o each o he in he XY plane, causing occlusion and in e e ence, in he XT plane hey a e e y clea ly dis inguishable. I hese wo pa e ns can be ou lined sepa a ely in he slice, he wo legs would be sepa a ed om one ano he h oughou he ideo sequence. Fig. 2: The pe iodic pa e n o he legs in mo ion in a sample XT slice. This ou lining was achie ed using an au oma ically ini ialised snaking algo i hm. A he ankle heigh o he subjec , an XT slice was ob ained and an ini ial app oxima e o he snake was i o he b aided image. This ini ial empla e was hen wa ped o ollow he pa e n’s edges. Because o he simila i y o he b aided pa e ns a locally connec ed slices, his ini ial snake i ing was pe o med only once. A e ha , he snake i ing p ocess was epea ed a each slice om he ankle o he head o he subjec and he esul o he snake i ing om he p e ious heigh was used as he ini ialisa ion a he nex . This esul ed in a comple e ou line o he acked egion, which is held h oughou he ideo sequence (see Figu e 3). Fig. 3: The ou line o he a ea o in e es in a sample ame. 2.2 Ellipse Fi ing Once he ou line o he a ea being acked has been ob ained and he body has been segmen ed, he ou line segmen s can be used as inpu s o he ellipse- i ing algo i hm. The ou line ob ained om he slice- by-slice snake algo i hm is di ided using he body segmen a ion in o he acked body pa s: he head, o so, high and shank. Each o hese segmen ou lines is passed o he ellipse- i ing algo i hm and an ellipse is a ached o each one independen ly. So a , he body pa s in each ame ha e been posi ioned independen o each o he and independen o hei loca ions in p e ious ames. Howe e , his could cause anomalies in he esul s. In many human mo ion acking algo i hms, he segmen s a e posi ioned subjec o cons ain s and a e dependen on he loca ions o hei p edecesso in he hie a chical ee s uc u e o he body. This is a good way o a oiding unlikely posi ioning bu i is p one o s aying. Since each posi ioning is dependen on i s own p e ious s a e and on he cu en s a e o i s connec ed body pa s, one bad i would p opaga e h ough he image and h ough he image sequence causing he acking o ail. While ou algo i hm a ely su e s om s aying and eco e s quickly when i does, i can po en ially esul in nonsensical conclusions. Using ou di ec ellipse- i ing me hod, he esul will be he bes - i ellipse wi h no limi a ions. This means ha , while we can ob ain a good es ima e o he posi ion and o ien a ion o he body pa , he size o he pa may a y om ame o ame and he ela i e angles wi h o he body pa s could be un easonable. In o de o o e come his p oblem, we apply cons ain s a e he ini ial app oxima e i has been acqui ed h ough di ec ellipse i ing. Fi s ly, he dimensions o each ellipse a e se o he a e age o e he sequence. While he e may be some change in he appa en dimensions as he pe son mo es sligh ly owa d o away om he came a o as muscles lex, i is easonable o assume ha he change will be insigni ican . Nex , he angles o he ellipses a e empo ally smoo hed using a 1D Gaussian il e . This has an indis inguishable e ec on he isual esul s bu i ensu es ha he body pa s a e no o a ing a un easonable speeds om ame o ame and i imp o es he gai g aphs. Las ly, he ela i e angles a e checked o ensu e ha he join angles a e easonable. The inal s age o he sys em design is he ex ac ion o he gai da a om he acked body model. Because he ellipses con ain in o ma ion abou he dimensions and o ien a ions o he limb segmen s, he g aph da a o he high and shin angles can be ex ac ed di ec ly om he isual esul s. The lexion angles a e he angles o he join s and so can be calcula ed as he di e ence o he body segmen angles. A ull se o sagi al iew gai g aphs along wi h a ew signi ican gai pa ame e s a e p esen ed he e. The me hod used in implemen ing his sys em is desc ibed ully in [9]. P elimina y esul s o a monocula ma ke - ee gai measu emen sys em 221 3 RESULTS 3.1 Sys em The sys em was implemen ed on a PC wi h a 2.66GHz Pen ium®4 p ocesso wi h 1GB o RAM. The gai labo a o y ideo images we e ilmed wi h an analogue ideo came a and we e cap u ed using an ATI All-in-Wonde ®128 P o ideo cap u e ca d. The na u al-en i onmen ideo images we e ilmed wi h a USB2.0 webcam. Some success ul p elimina y es s ha e been done using wo synch onised USB2.0 webcams wi h a iew o c ea ing a ully in eg a ed 3D sys em in he u u e. The p og am was implemen ed in Mic oso Visual C++ ® 6.0. The gai esul s we e g aphed in MATLAB®. 3.2 Inpu Da a The i s ideo was ilmed in a gai labo a o y using a high quali y camco de and acquisi ion ca d. The subjec is wea ing i ed spo swea wi h he legs mos ly ba e and is walking wi hou a m swing. The second ideo was ilmed wi h a s anda d webcam in a no mal en i onmen , al hough he backg ound is kep da k o ensu e ha he subjec is clea ly isible in con as . The subjec is d essed in e e yday clo hes and walking eely wi h a m swing. In bo h ideo clips, he da a was cap u ed a 30 ames pe second as a 320 x 240, 24-bi uncomp essed RGB AVI. Highe esolu ion would gi e be e esul s bu would conside ably slow he sys em. The sample ideo sequences used he e a e bo h app oxima ely ou seconds long (126 ames) o alling app oxima ely 28MB. P ocessing ime is less han one minu e. 3.3 Visual Resul s The isual da a is use ul o gauging he success o he algo i hm and i could also be used o c ea e an a a a o mimic he gai in a i ual 3D en i onmen . This is a e y angible o m o ou pu bu is only ully ealisable wi h comple e 3D gai da a i.e. including he ans e se and co onal planes o mo emen and including pel is and ankle da a. He e, we ha e concen a ed on he sagi al plane and pa icula ly on he main lowe limb a ea (i.e. he high and shank), as his is he mos challenging egion in he acquisi ion o gai in o ma ion. Shown in Figu e 4 a e some sample ames om he wo ideo sequences wi h he simple ellipse body model a ached. No e he signi ican di e ence in pic u e quali y and con as be ween he wo sequences ye despi e his, he e is li le di e ence in he accu acy o he model i ing. Howe e , because he leg is no di ec ly isible in he second sequence, he e is an una oidable ambigui y wi h ega d o he dimensions o he limb segmen s. In he g aphical esul s, we plo he o ien a ion o he segmen s h oughou he sequence and so his ambigui y will no a ec he gai measu emen s – ano he ad an age o his ellipse-based me hod. Fig. 4: Sample ames om he gai labo a o y ideo sequence and he na u al-en i onmen ideo sequence, showing he simple ellipse model a ached o he images o he subjec s. 3.4 Gai Da a As his sys em concen a es on he main lowe limb sec ions, we ha e p esen ed g aphed da a o he o a ion and ela i e angles o he high and shank. These a e ypical gai g aphs used in gai kinema ics. The da a in Figu e 5 is p esen ed om oo -s ike ( he poin when he heel o he acked leg s ikes he g ound) o oo -s ike. (a) (b) Fig. 5: Gai da a acqui ed om (a) he gai labo a o y sequence and (b) he na u al en i onmen sequence. Ad ances in Elec ical and Elec onic Enginee ing 222 Along wi h he isual and g aphical esul s, a ew signi ican gai pa ame e s a e no mally acqui ed as pa o ull sagi al gai in o ma ion. These a e he walking eloci y, he s ide leng h and he cadence. The s ide leng h is simply he leng h o wo s eps ( he s ep leng h and walking eloci y acquisi ion me hods a e desc ibed p e iously) and he cadence is he numbe o s eps pe minu e. The alues o he sample ideo sequences a e p esen ed below. Table 1: Gai pa ame e s o bo h ideo sequences Gai Labo a o y Sequence Na u al En i onmen Sequence Uni s Subjec ’s Heigh 175 180 cm Walking Veloci y 111 117 cm/sec S ide Leng h 175 170 cm Cadence 87 96 s eps/min 4 DISCUSSION Cu en ly, he e a e ew comme cial ma ke - ee sys ems in use in gai analysis. While ma ke - ee sys ems a e becoming mo e common in o he a eas such as spo s science and anima ion, he accu acy and de ail equi ed o gai analysis makes his sys em design pa icula ly challenging. In esea ching he cu en s a e o a ai s, we disco e ed ha he g ea es obs acles in designing ei he ma ke -based o ma ke - ee sys ems lie in sagi al-plane acquisi ion. This is also whe e he mos in o ma ion can be gleaned in diagnosing and analysing pa hological gai . Thus, i was decided in his p ojec o concen a e on designing a eliable ma ke - ee sys em o moni o ing sagi al- plane mo emen in he gai cycle. We ha e achie ed ou ou lined goals o building a comple ely au oma ed, po able gai measu emen sys em o use in sagi al-plane gai analysis. This sys em will allow pa ien s’ gai o be eco ded in a elaxed and con enien en i onmen wi hou he need o a ained he apis o be p esen . Thus, he he apis s can use hei aluable expe ise in diagnosing gai a he han spending hei ime mas e ing and using cumbe some ma ke sys ems. I is hoped ha his sys em will be used by he apis s and pa ien s in he Na ional Rehabili a ion Hospi al, Dublin whe e i was de eloped, and will pe haps become pa o a mo e comple e gai labo a o y design in he u u e. REFERENCES [1] Segen J., Pingali S., ‘A came a-based sys em o acking people in eal ime’, P oceedings o he In e na ional Con e ence on Pa e n Recogni ion, pp. 63-67, Vienna, Aus ia, Augus 1996. [2] Cai Q., Mi iche A., Agga wal, J.K., ‘T acking human mo ion in an indoo en i onmen ’, P oceedings o he 2nd In e na ional Con e ence on Image P ocessing, ol. 1, pp. 215-218, Washing on D.C., Oc obe 1995. [3] Mik omak: www.mic omak.com [4] Chaudha i A.M., B agg R.W., Alexande E.J., And iacchi T.P., ‘A ideo-based, ma ke less mo ion acking sys em o biomechanical analysis in an a bi a y en i onmen ’, BED Bioenginee ing Con e ence, Vol. 50, pp. 777-778, 2001. [5] Agga wal J.K., Cai Q., ‘Human mo ion analysis: a e iew’, Compu e Vision and Image Unde s anding, ol. 73, no. 3, pp. 428-440, 1999. [6] Ga ila D.M., ‘The isual analysis o human mo emen : a su ey’, Compu e Vision and Image Unde s anding, ol. 73, no.1, pp. 82-98, 1999. [7] Cou ney J., Applica ion o Digi al Image P ocessing o Ma ke - ee Analysis o Human Gai . Mas e s Thesis, Na ional Uni e si y o I eland, Dublin, Janua y 2001. [8] Nyogi S., Adelson E., ‘Analysing and ecognising walking igu es in XYT’, P oceedings o he IEEE Con e ence on Compu e Vision and Pa e n Recogni ion, pp. 469–474, 1994. [9] Cou ney J., Ma ke - ee Human Mo ion Cap u e wi h Applica ions in Gai Analysis. PhD Thesis, Na ional Uni e si y o I eland, Dublin, July 2005.