A novel design of an augmented reality based navigation system & its industrial applications

ACTA IMEKO ISSN: 2221-870X Ma ch 2018, Volume 7, Numbe 1, 57-62 ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 57 A No el Design o an Augmen ed Reali y Based Na iga ion Sys em & i s Indus ial Applica ions Timo ei Is án E dei, Zsol Molná , Nwachukwu C. Obinna, Géza Husi Uni e si y o Deb ecen, Elec ical Enginee ing and Mecha onics Depa men H-4028 Deb ecen, Ó eme ő s ee 2-4 Sec ion: RESEARCH PAPER Keywo ds: augmen ed eali y; i ual eali y; au oma ed guided ehicle; KUKA obo ics Ci a ion: Timo ei Is án E dei, Zsol Molná , Nwachukwu C. Obinna, Géza Husi, A No el Design o an Augmen ed Reali y Based Na iga ion Sys em & i s Indus ial Applica ions, Ac a IMEKO, ol. 7, no. 1, a icle 10, Ma ch 2018, iden i ie : IMEKO-ACTA-07 (2018)-01-10 Sec ion Edi o : Lo enzo Ciani, Uni e si y o Flo ence, I aly Recei ed Decembe 12, 2017; In inal o m Janua y 16, 2018; Published Ma ch 2018 Copy igh : © 2018 IMEKO. This is an open-access a icle dis ibu ed unde he e ms o he C ea i e Commons A ibu ion 3.0 License, which pe mi s un es ic ed use, dis ibu ion, and ep oduc ion in any medium, p o ided he o iginal au ho and sou ce a e c edi ed Funding: The p ojec was suppo ed by Uni e si y o Deb ecen, Elec ical Enginee ing and Mecha onics Depa men Co esponding au ho : Timo ei Is án E dei, e-mail: imo eie [email protected] 1. INTRODUCTION Main enance ac i i ies a e in eg al wi hin an indus ial se ing. The e iciency o hese ac i i ies is associa ed wi h he o al p oduc i i y o an indus ial p ocess/machine. A highly e icien main enance policy/s a egy usually esul s in ela i ely high le els o plan p oduc i i y. Augmen ed Reali y AR and Vi ual Reali y VR may be inco po a ed in o a main enance s a egy. AR and VR echnology would enhance main enance ac i i ies and acili a e somewha complex asks. A main enance s a egy includes a ange o ac i i ies and may be ca ego ized in o adminis a i e, echnical and managemen p ocesses. AR echnology con ains digi al da a, as well as o he echnical de ails, and is able o p o ide in o ma ion abou indus ial machine y-equipmen , wi hou he need o equipmen disassembling. In his ega d, we employed AR echnology in de eloping a unique na iga ion sys em o eplace/ educe he ins alla ion cos s o adi ional Au oma ic Guided Vehicles (AGV) na iga ion sys ems. The p oposed AR sys em consis s o a came a, which obse es he QR-Code/Ma ke s, and a p ocessing uni . Augmen ed Reali y enables isualisa ion o any da a and in o ma ion, as well as con ol o a unning p ocess. I means i is possible o ead a ious da a wi hin any equipmen , du ing i s ope a ion and in eal- ime. This acili a es analysis o “black box” sys ems. 2. AR IMAGE PROCESSING - OPENCV Ou main goal was de eloping a Cybe -Physical Sys em (CPS) o an indus ial obo labo a o y and ind ways o implemen ing In e ne o Things (IoT). The depa men obo ic labo a o y includes a KUKA KR5 medium payload indus ial welding obo , a KUKA KR3, FANUC spide and a Sony Sca a SRX-611 [1]. The da a and pa ame e s ha e been encoded in o QR-Codes and o he unique ma ke s. In he i s es we use hese codes as posi ion ma ke s and an ins uc ion se o con ol a p o o ype AGV obo . The AGV uses an IP came a as ision senso s. The Building Mecha onic Resea ch Cen e ea u es se e al unc ional IP came as, which we e in e aced o he CPS sys em. A compu e se es as he CPS sys em p ocessing uni . The compu e has an In el Co e i7 CPU, 16 GB DDR3 RAM, 2 TB HDD & 2 × NVIDIA GeFo ce GTX 650 Ti. On he o he hand we chose OpenCV, which is a ision lib a y so wa e wi h nume ous algo i hms, o image p ocessing [1]. In o de o implemen he AR algo i hm, a special plugin/lib a y known as ARma was equi ed. AR gi es an in o ma i e iew o he wo ld, and he main ABSTRACT This pape p oposes a design o an augmen ed eali y based na iga ion sys em, as well as in es iga e i s po en ial a eas o applica ion wi hin he indus y. Wi h he ascen o Indus y 4.0 (IoT), sys ems such as Augmen ed Reali y and Vi ual Reali y bene i om he a ailabili y o bulk senso y da a. ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 58 pa s include a came a, a QR-Code [2] wi h hidden in o ma ion and an image p ocessing so wa e/ha dwa e o decode a QR- Code and de ec he posi ion o he code in eal- ime (Figu e 1). The p oposed AR sys em is equi ed o ack he mo ion o a poin in a p ede e mined 3D coo dina e sys em. Mo ion acking in ol es he measu emen o an objec eloci y and o ien a ion (Figu e 2). This objec acking me hod is a igo ous one, because i s you need o design he 3D model in a 3D CAD so wa e, and a e i you need o dec ease he polygon numbe o he 3D model. Al hough his has o be done wi h ci cumspec ion. In ou case we used wo N idia GeFo ce GTX 650 Ti g aphics ca ds. The main pa ame e s o hese g aphics ca ds a e he maximum polygon numbe and maximum image ame d awings pe second. Rega ding he unc ioning o he g aphics ca d, i is wo h o ecall ha compu e -gene a ed images a e made up o iangles. Fo example, a squa e will be made up by wo quad an iangles. In case o complex geome ical o ms, like a sphe e o cu ed su aces, hese iangles a e so small and dense, ha in appea ance hey do no bo he he use , in o he wo ds, hey canno be seen. The bigge he numbe o he iangles, he smalle hey a e, and he be e he quali y o ou 3D model: his is a equi emen o be me in o de o ou d aw compe i o s. The goal is o ha e 60 pic u e e eshmen s pe second, his is a alue ha canno be no iced by he human eye, and i gi es cons an mo emen in case o highe speeds as well. In he case o he g aphics ca d, he pe o mance can also be cha ac e ized in iangle/seconds. This means he iangle (polygon) numbe o a gi en pic u e is educed by 1, he polygon d awings will be educed by 60 each second. In case o bigge numbe s, his implies a signi ican lessening o demand o he calcula ions. As we can see in Figu e 3, in he beginning we ied unning he sys em wi h 50000 polygons, which showed a nice g aphic, bu because o he dynamic p ope ies o he model, his g aphic was useless, since he e eshing o he image d opped o 25 pic u es/second. Wi h he polygon numbe being d opped d as ically o 1000 polygons, he model became g aphically useless. On he o he hand, by u ning o he limi a ions o he g aphics ca d (V-sync) he nume ical accomplishmen su passed he alue o 2300 pic u e e eshmen s. By unning mo e es s, he op imal polygon numbe came o be 25000. Wi h his numbe , he 3D model is sui able and a s able 60 pic u e e eshmen s pe second can also be a ained and i pe mi s o he objec s o be loaded beside he obo a m. High polygon numbe alues a e compu a ionally expensi e. We designed all 3D models in 3D CAD so wa e, and a e ha , we dec eased he numbe s o he polygons o make ou p og am un mo e e icien ly [3]. The nex s ep o ou esea ch was designing he AGV p o o ype. The AR AGV na iga ion sys em would include ajec o y planning and acking, as well as collision a oidance. The mos impo an cha ac e is ic o he obo is ha he e a e no senso s on he obo i sel , bu i uses only he IP came as o he obo ics labo a o y o ajec o y acking. Fi s , we designed he AGV concep in a 3D CAD modelling so wa e and made a pa lis wi h he necessa y equipmen [4]. This 3D model is shown by Figu e 4. The ehicle guide pa is composed o an A duino Nano panel. This makes up he equi ed PWM signal o he se o mo o s and communica es wi h he ESP Wi-Fi module h ough communica ion lines. Figu e 1. ARma and OpenCV – wo king. Figu e 2. Came a iew & posi ion de ec ion o objec . Figu e 3. Polygon educ ion. Figu e 4. AGV p o o ype obo ; le panel: 3D model; igh panel: pic u e. ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 59 In addi ion, wo con inuous o a ing se o mo o s make he shell o he ehicle mo e. The communica ions a e ensu ed by he ESP Wi-Fi module which, a e he powe -supply ol age is supplied, au oma ically connec s o he Wi-Fi ou e and ansmi s he ecei ed commands o he A duino Nano Panel. Besides his, on he AVG a Li-Ion accumula o wi h wo cells was placed. Two DC-DC s ep-down con e e s p o ide ol age s abiliza ion. The s ep-down con e e s ha e high powe con e sion a e, e en 90% e iciency is eachable, which is why hey we e chosen. The block diag am o he AGV obo is shown on Figu e 5. The co e o he AGV p o o ype obo is an A duino Nano, wo LM2596 DC s ep down con e e and a Wi-Fi-Se ial module. The weigh o he obo wi h ba e ies is 483 g ams. Two Li- Ion ba e ies we e used o powe supply, each p o iding 3.6 V ol age and 3600 mAh capaci y. Two o a ing se omo o s [5] we e used o d i e. They can deli e 0.27 Nm o que a 6 V powe . The maximum cu en consump ion is o 200 mA each. The high ba e y capaci y and he low powe consump ion o he AGV p o o ype obo make possible a long ba e y li e. Du ing he es s, depending on he ope a ing speed and load, he con inuous ope a ion was 8 o 10 hou s. 3. ROBOT LABORATORY AS AR ISOLATED ENVIRONMENT The Robo Labo a o y o he Building Mecha onic Resea ch Cen e se es as an AR isola ed en i onmen , and i is a pa o he building su eillance and secu i y sys em ne wo k. Fu he mo e, we can use he ins alled IP came as as ision senso using a desk op pc which can s o e he eco dings and which is powe ul enough o make CUDA based image p ocessing/decoding [6]. Thus, i is possible o use QR-Code/Ma ke s placed in he ield o ision o he came as and hus making possible o iden i y hei posi ions. Once he iden i ica ion is done, we can add o he commands o he QR-Code/Ma ke s o pe o m asks. The diag am in Figu e 6 shows he s uc u e o he Robo Labo a o y in a con ol ask. The sys em consis s o IP came as and hei QR-Code and he managed de ice. Images o IP came as a e p ocessed by he cen al high pe o mance compu e . On he con olled de ice, in ou case he AGV obo , no de ec o is ound: i s posi ion is de e mined by he came as in he Robo Labo a o y. IP came as a e a ailable on bo h loo s o he en i e Building Resea ch Cen e, whose images a e a ailable om he cen al compu e and p ocessed. This allows he AGV obo o ope a e in any pa o he building, i necessa y [7]. The IP came as used in he Lab a e ype TCIP- LP o213WDRMDN and ha e he ollowing pa ame e s: • Sony 1/3, 1.3 Mpx. P og essi e Exmo CMOS senso • 1280x1024 (SXGA); H.264, MPEG4, M-JPEG • Lux (Au oma ic In a ed LED Ligh ing below 2 Lux) • 2D noise educ ion and sensi i i y enhancemen • Mo ion De ec ion and Ala m • Two-way audio and mul icas • Buil -in Asphe ical Day / Nigh Lens ( 3.8 9.5 mm) • 12 V supply ol age [8]. The came as cha ac e is ics made hem sui able o he con olling o he AGV obo . E en hough, his also comes wi h ce ain se backs. One o hese is connec ed wi h he au oma ic ligh adjus men . In he case o he came as used i wo ks like his: i uses all o he ligh powe o he acqui ed image o compu e a alue, which is hen aken as e e ence o u he adjus men s. The o he se -back comes om he used lens. They gi e a wide poin o iew, bu in he same ime dis o he edges o he image. A e he designing and building o he AGV p o o ype obo , we needed o es he sys em de ec ion s abili y. In ha aspec we analysed he de ec ion abili y o all ins alled IP came as in he Robo Labo a o y. In his con ex we made measu emen s o he illumina ion which is he mos impo an hing in de ec ion. The g aph below illus a es he measu able illumina ion in he cen e o he Lab depending on he numbe o ligh s on. The measu emen s ha e shown (Figu e 7) ha he e mus be a leas 200 lux abo e he exposu e o he a ea, so ha he TCIP-LP o213WDRMDN came a can de ec he codes. The came a can see lowe b igh ness bu he image is no clea enough o ecognize he codes. In addi ion, in he absence o illumina ion, he came a will au oma ically u n on he in a ed LED. The in a ed ligh di ac s om di e en su aces di e en ly om o he isible ligh , gi ing simply a black and Figu e 6. AR isola ed en i onmen . Figu e 5. AGV obo block diag am. ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 60 whi e image, making i unsui able o code ecogni ion. 4. AR BASED NAVIGATION SYSTEM In he i s pa o ou p ojec we ound a sui able place o es he sys em which was he obo ics labo a o y in he Building Mecha onic Resea ch Cen e. I s dimensions a e: wid h: 4.3 m; leng h: 6.5 m; heigh : 2.8 m. We designed he QR-Codes/Ma ke s and p in ed hem ou in di e en sizes and placed hem in he igh o de o make he pa h o he obo . The ask o he AGV p o o ype obo is o lea e he “S a ” posi ion, go o he 1s posi ion and wai un il he KUKA KR5 obo ic a m goes o he posi ion o he AGV and pu s on i he aluminium cube (Figu e 8). A e i each he 2nd, 3 d 4 d, 5 d posi ion and he Home posi ion wi h he ca go. We p in ed a code o iden i ying he AGV’s o ien a ion. The dis ance be ween he posi ions, he ime equi ed o make he dis ance and he a e age speed a e calcula ed in Table 1. Dis ances ha e been de e mined by he so wa e du ing ou e planning. The eco ding o ime was de e mined by he ime elapsed be ween ouching he wo a ge s. The e a e di e ences be ween a e age speed alues. The eason is ha o he s aigh pa hs, he applied algo i hm simply de e mines he pa h and he obo mo es wi hou o a ions. Fo pa hs wi h obs uc ions such as he 2-3 ou e, he algo i hm bypasses he obs acles, de e mining whe e he obo needs o o a e, which akes ime and dec eases he a e age speed. The IP came as o Robo Labo a o y see he codes and s eam da a o he no ebook ia Wi-Fi. On he desk op compu e , OpenCV and ARma ecognize he posi ion o he code and decode i , send he in o ma ion o he AGV p o o ype obo which plans and acks he ajec o y. The algo i hm used o design he ou e is he Rapidly- Explo ing Random T ee (RRT), o iginally w i en o humanoid obo s o design mo ion. The RRT algo i hm has ad an age in many cases, p o iding a solu ion o 2D and 3D ou e planning. The ope a ion o RRT can easily be desc ibed. The algo i hm g ows om he ini ial con igu a ion o " oo " as a ee in he sea ch space o de ec he a ge con igu a ion. I i canno ind a iable ou e, i will con inue o sea ch in he open space, while aking in o accoun he c i e ia (ba ie s, limi s, e c.). The size o he " oo s" can be de e mined by he g ow h ac o . The RRT algo i hm will con inue o sea ch un il i inds he op imal pa h be ween he s a and des ina ion con igu a ions. In he i s es we launched/p og ammed he KUKA KR5 indus ial obo manipula o wi h ou elec onic PCB, he TiMo boa d, o con ol he obo g ippe . Fu he mo e, we es ablished a new Wi-Fi ne wo k wi h a TP-Link ou e o make he Wi-Fi signal s onge in he Robo Labo a o y. This was e y impo an because we sen all signals ough Wi-Fi and we wan ed o a oid he lag o he s eam o he mobile phone. The s uc u e o he communica ion ne wo k used o es ing is shown on Figu e 9. Figu e 7. Illumina ion o he cen e o he Robo Labo a o y . Table 1. Dis ance, ime, speed alues. Pa h Dis ance [cm] Time [s] Speed [cm/s] S a – 1 94 41.5 2.27 1 – 2 46 28.6 1.6 2 – 3 87 67 1.3 3 – 4 75 30.2 2.48 4 – 5 98 40.8 2.4 5 – Home 53 24.7 2.15 Figu e 8. Robo Labo a o y as AR en i onmen . Figu e 9. Ne wo k communica ion sys em. ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 61 Fo he i s ime we no iced a lag in he Desk op compu e . Fo una ely, we could se he dedica ed N idia g aphic ca d o calcula e ins ead o he co e i7 CPU which was slowe in his ins ance. When we began he i s es , he AGV p o o ype obo was in he “S a ” posi ion, hen i s a ed o go o he 1s posi ion and wai ed he e un il he KUKA KR5 mo ed o he AGV posi ion and opened he g ippe o pu he aluminium cube on he AGV. The nex s ep was o each he 3 d posi ion ma ke and a oid he collisions wi h he o he objec s. To ealize his we made codes wi h o he 3D models and we se he a oidance dis ance in he ARma so wa e. Knowing his da a, he AGV could a oid hem du ing he mo ion (Figu e 10). When he AGV eached he HOME posi ion i s opped. The p ojec ask equi ed he e-c ea ion o he KUKA KR5 g ipping sys em o be sui able o ma e ial handling. To he end a Japanese, Humph ey H040-4E2 ype p e-d i en elec o- pneuma ic bis able al e has been selec ed, which uns om 2 o 7 ba and 24 ol s [9]. The con ol al e was ins alled in o a g ippe GRIP GmbH. To con ol he g ippe , we used ou own sel -de eloped TiMO Boa d con olle sys em, which is sui able o con olling indus ial pneuma ic/hyd aulic al es and has al eady been p o en o con ol smalle p oduc ion line cells [10]. TiMo Boa d was designed o Indus y 4.0 / IoT asks. The o he pa o he TiMo boa d is a Raspbe y Pi B+ minimized desk op compu e [11] and he Timo ei-Robo ics Linux dis ibu ion Os which was ins alled on i . The s uc u e o he p og ammable con ol sys em is shown on Figu e 11. When p og amming he con ol panel (TiMo Boa d), he p og am was w i en on he Timo ei-Robo ics Os, in s anda d PLC ladde diag am. The TiMo boa d con ol panel has been ins alled on he pneuma ic g ipping elec ic ci cui o he KUKA KR5 indus ial obo . 5. AR APPLICATION AREAS & DEVELOPMENT In he i s pa o ou p ojec we es ed he AR based na iga ion sys em success ully, bu he e a e many o he ways o use QR-Codes and AR echnology in de elopmen o in educa ion. In he indus ial obo ics labo a o y we ins alled obo s o a ious ypes. Main enance o all hese obo is ela i ely di icul , because we need o egis e all o hem. We collec ed all he da ashee s, uploaded o ou p i a e ne wo k sys em and embedded hem in o QR-Codes. The co e o he p i a e ne wo k sys em is a Raspbe y Pi [12] mini desk op compu e , which has a Raspbian Linux dis ibu ion sys em and a 64GB SDHC ca d. When someone needs in o ma ion abou a obo in he obo labo a o y he has o decode he QR-Code and in ha momen we ecei e a no i ica ion ha someone decoded one o ou dynamic QR-Code, so ha we can see on a diag am which code ha e been decoded (Figu e 12). On he o he hand, we can use he QR-Codes as AR ma ke s in o de o isualize he axes posi ions o KUKA KR5 o se dynamically i s posi ion in he wo kspace (Figu e 13). The AR echnology is also able o es s andalone sys ems and i ems. We de eloped a CAD g ippe o he KUKA KR5, we educed he polygon numbe o i and moun ed “ i ually” Figu e 12. QR-Code decoding - moni o ing. Figu e 13. AR code o se posi ion & axis isualiza ion. Figu e 11. P og ammable Con ol Sys em – TiMo Boa d. Figu e 10. AGV P o o ype i s es . ACTA IMEKO | www.imeko.o g Ma ch 2018 | Volume 7 | Numbe 1 | 62 on o he obo a m. Fu he mo e, he codes can be used o es ablish in e ac i e connec ions be ween each o he . In one o ou es s we used ano he ma ke which consis s o a column 3D model (Figu e 14). When we mo ed he eal KUKA KR3 wi h he i ual moun ed g ippe , i could push away he i ual column (Figu e 15). 6. CONCLUSIONS The Augmen ed Reali y echnique is po en ially e ec i e because we can se up a i ewall sys em p o ec ing in o ma ion embedded in he codes. Fu he mo e, we can enc yp commands o obo s as we did o command a p o o ype AGV. Mo eo e , we can upg ade he obo s i ually and pe o m es s o collision a oidance, g ippe eplacemen es ing, e c. Du ing he es s we used GPU image p ocessing because he in eg a ed VGA was no as enough o calcula ions. Un o una ely, we can clea ly say o hese es s we had o use a ela i ely as p ocessing compu e because i is compu a ionally expensi e. The sys em ha we c ea ed does no only ecognize QR Code in ead-only in o ma ion, bu can also ecognize hese codes as so-called Ma ke s ha can ep esen he s a us/posi ion o an AGV obo o he axes o obo a ms. This has enabled he con ol o AGV obo and he a achmen o i ual AR de ices o physical machines. The AR based na iga ion sys em o he AGV is ully au onomous so ha no human in e en ion is equi ed. The AR echnology equi es huge knowledge in 3D modelling, polygon educ ion and image p ocessing. Fu he imp o emen s a e unde way o enhance hese a eas o applica ion. ACKNOWLEDGEMENT The wo k is suppo ed by he EFOP-3.6.1-16-2016-00022 p ojec . The p ojec is co- inanced by he Eu opean Union and he Eu opean Social Fund. REFERENCES [1] N.C. Obinna, T.I E dei, Zs. Molná , G. Husi, LabVIEW Mo ion Planning and T acking o an Indus ial Robo ic Manipula o (KUKA KR5 a c): Design, Modelling, and Simula ing he Robo 's Con olle Uni - Elec ical Enginee ing and Mecha onics Con e ence, 2017.03.12. [2] Mau icio Ma engoni, High Le el Compu e Vision Using OpenCV, G aphics, Pa e ns and Images Tu o ials (SIBGRAPI- T), 2011 24 h SIBGRAPI. [3] J. Rubins ein, Signal Delay in RC T ee Ne wo ks, IEEE T ansac ions on Compu e -Aided Design o In eg a ed Ci cui s and Sys ems, 1983. [4] T.I. E dei, Zs. Molná , N.C. Obinna, T. Gönczi, Indus ial KUKA Robo Manipula o in Simula ion En i onmen and Posi ion Read-back, 1s In e na ional Symposium on Small-scale In elligen Manu ac u ing Sys ems (SIMS 2016). [5] Zs. Molná , G. Husi, P o o ípus alka észek elkészí ésé e ki ejlesz e CNC á ezé lése és á elügyele e, 10.17667/ iim.2017.si/2. [6] Pa allax se o mo o pa ame e s o icial manual, h ps://www.pa allax.com/p oduc /900-00008 2017.04.14. – 15:35. [7] N. Zla ano , CUDA and GPU Accele a ion o Image P ocessing, IEEE Compu e Socie y, 2016.03. [8] TCIP-LP o213WD-O ical Manual {o line} 2017.02.24., 18:42). [9] N.C. Obinna, T.I E dei, Zs. Molná , G. Husi, Su eillance and Secu i y Sys em in he Building Mecha onics Resea ch Cen e - Con e ence: 1s In e na ional Symposium on Small-scale In elligen Manu ac u ing Sys ems (SIMS 2016). [10] Humph ey H040-4E2 – h ps://www.asse a.com/pa s/humph ey-h040-4e2 2017.03.30. – 16:34. [11] T.I. E dei, Zs. Molná , G. Husi, Sajá ejlesz ésű ezé lő endsze Open-Sou ce alapokon - IJEMS.2016.2.13. [12] E. Up on, Raspbe y Pi - IEEE Compu e Socie y 31.10.2013, 10.1109/MC.2013.349. Figu e 14. Vi ual AR i ems. Figu e 15. Vi ual collision.