The Reduction Problem in CUDA and Its Simulation with P Systems

The Reduc ion P oblem in CUDA and I s Simula ion wi h P Sys ems Rodica Ce e chi1, Miguel ´ Angel Ma ´ınez-del-Amo 2, Ma io J. P´e ez–Jim´enez2 1Facul y o Ma hema ics and Compu e Science Uni e si y o Bucha es 14 Academiei s . 010014 Bucha es , Romania E-mail: [email p o ec ed], [email p o ec ed] 2Resea ch G oup on Na u al Compu ing Depa men o Compu e Science and A i icial In elligence Uni e si y o Se illa A da. Reina Me cedes s/n, 41012 Se illa, Spain E-mail: [email p o ec ed], [email p o ec ed] Summa y. We in oduce P sys ems wi h dynamic communica ion g aphs which simu- la e he unc ioning o he CUDA a chi ec u e when sol ing he pa allel educ ion p ob- lem. 1 In oduc ion In oduced in [13], P sys ems a e powe ul compu a ional de ices, wi h a high de- g ee o pa allelism, whose unc ioning is inspi ed by biological p ocesses a he le el o he cells, and o hei memb anes ([13],[14]). Among hese p ocesses, ew i ing and communica ion play an impo an ole. I is o in e es o compa e P sys ems wi h o he , classical, pa allel pa adigms. We ha e begun such a s udy in he o m o simula ing pa allel classical a chi ec- u es wi h P sys ems, in pa icula , he pe ec shu le a chi ec u e [4], [5] and he mesh a chi ec u e [6]. The educ ion p oblem, being one o he mos simple, p ima y ones o be sol ed in di e en con ex s, was used as an illus a ion. In [7] some gene al guidelines we e de eloped along which a wide class o pa - allel a chi ec u es can be simula ed wi h P sys ems. P sys ems wi h dynamic communica ion g aphs we e in oduced. In hei unc ioning, ew i ing s eps and communica ion s eps a e sepa a ed and made mo e isible. They di e om he sys ems in oduced in [3] in ha he communica ion g aphs a e inspi ed by he pa icula pa allel ne wo k a chi ec u e being simula ed. CUDA s ands o Compu e Uni ied De ice A chi ec u e [15, 10], and is a ech- nology p op ie a y o NVIDIA Co p. Since i s in oduc ion in 2007, CUDA has 92 R. Ce e chi, M. ´ Angel Ma ´ınez-del-Amo , M.J. P´e ez–Jim´enez allowed p og amme s o ake ad an age o he inhe en pa allel a chi ec u e o GPUs, which anges om 240 co es (Tesla C1060, eleased on 2008) o 2880 co es (Tesla K40, eleased on 2013). This is pe o med by using a h eaded, sha ed- memo y, abs ac ed model o he GPU, which is implemen ed by C/C++ ex en- sions. CUDA has helped o es ablish GPU compu ing [9] as a sub- amewo k o High Pe o mance Compu ing. In ac , i has been success ully applied o a b oad spec um o esea ch a eas, including Sys ems Biology and Popula ion Dynamics [12], and Memb ane Compu ing [1, 2, 11], among o he s. In he p esen pape we p opose o simula e wi h P sys ems he educ ion p oblem as sol ed in CUDA. Sec ion 2 is de o ed o he p esen a ion o se e al imp o ed e sions o sol ing he educ ion p oblem in CUDA. Sec ion 3 is de o ed o he p esen a ion o i s simula ion wi h P sys ems wi h dynamic communica ion g aphs. 2 Sol ing he Reduc ion P oblem in CUDA The GPU (G aphics P ocesso Uni ) is he co e o g aphics ca ds. A GPU o- day con ains housands o compu ing p ocesso s de o ed o g aphics. Howe e , no el echniques enable p og amme s o ake ad an age o his highly pa allel a - chi ec u e o scien i ic compu ing. These a e called GPGPU (Gene al Pu pose compu ing on he GPU) [9]. A new e a o GPGPU s a ed wi h he in oduc ion o CUDA (Compu e Uni ied De ice A chi ec u e) [15, 10] by NVIDIA. I o e s a p og amming model ha abs ac s he GPU a chi ec u e o p og amme s, so i is enough o lea n some ex ensions o C/C++ language (CUDA ex ensions), whe eas he CUDA d i e will execu e he code on he GPU. In he ollowing sec ions we will in oduce some concep s and e minology o CUDA which a e necessa y o unde s and he wo k p esen ed in his pape . 2.1 CUDA p og amming model The CUDA p og amming model assumes ha he CPU (o hos ) akes con ol o he execu ion low, and pe mi he GPU (o de ice) o un many ins ances o he same code in pa allel. This code is called ke nel, and i is execu ed by a g id o h eads. Typically, a g id is composed o housands o h eads, since he c ea ion o a su icien numbe o h eads o use all ha dwa e esou ces equi es a la ge amoun o da a pa allelism. The h eads a e a anged wi hin he g id in a wo-le el hie a chy, as seen in Figu e 1. A he highe le el, each g id consis s o one o mo e h ead blocks. A he lowe le el, each block is o ganized as a h ee dimensional a ay o h eads. All blocks in a g id ha e he same numbe and o ganiza ion o h eads. Each block is iden i ied by a wo dimensional iden i ie , and each h ead wi hin i s block by a h ee dimensional iden i ie (ID). The e o e, any h ead can be unequi ocally iden i ied by he union o bo h h ead and h ead The Reduc ion P oblem in CUDA and I s Simula ion wi h P Sys ems 93 block iden i ie s. The execu ion o h eads inside a block can be synch onized by ba ie ope a ions ( sync h eads()), and h eads o di e en blocks can be synch onized only by inishing he execu ion o he ke nel. Fig. 1. Th eading model in CUDA. Th eads a e execu ed in a g id, and hey a e o ga- nized in blocks. The memo y hie a chy is explici ly and manually managed in CUDA. This memo y model is composed in se e al le els, each one o e ing di e en speeds and s o age p ope ies. We highligh he wo mos impo an ones: global memo y and sha ed memo y. Global memo y is he la ges bu he slowes memo y in he sys em. I is accessed by he hos (whe e he inpu and ou pu da a a e alloca ed) and by any h ead in execu ion. Sha ed memo y is he smalles bu as es memo y. I is accessed by h eads belonging o he same block. No mally, pe o mance o CUDA applica ions depends on how much sha ed memo y is exploi ed. Thus, an e icien way o s uc u e an algo i hm is as ollows: 94 R. Ce e chi, M. ´ Angel Ma ´ınez-del-Amo , M.J. P´e ez–Jim´enez 1. The h eads o each block ead i s co esponding da a po ion om global memo y o sha ed memo y (which is ine i able because he hos only can pu he da a in global memo y). 2. Th eads wo k wi h he da a di ec ly on he sha ed memo y. 3. Th eads copy hese da a back o global memo y (so he hos can e ie e he esul ). 2.2 Mode n GPU a chi ec u e The GPU a chi ec u e has e ol ed in he las yea s, o e ing e en mo e com- pu e capabili ies. In gene al e ms, i consis s o a scalable p ocesso a ay, o ga- nized in S eaming Mul ip ocesso s (SMs) o S eaming P ocesso s (SPs, o co es). The numbe o hem depends on he GPU. SMs a e based on he SIMT (Single- Ins uc ion Mul iple-Th ead) model. Basically, in he SIMT model all he h eads execu e he same ins uc ion on di e en piece o da a. SMs c ea e, manage, sched- ule and execu e h eads in g oups o 32 h eads (which is he b anching g anula i y o NVIDIA GPUs). This se o 32 h eads is called wa p, and each SM can han- dle many o hem. Indi idual h eads o he same wa p mus s a oge he a he same p og am add ess. Howe e , hey a e ee o b anch and execu e indepen- den ly, bu a cos o se ializa ion and pe o mance (in ac , SIMT is eally applied o he wa p). I a wa p is b oken (because o b anching o memo y s all), he eal pa allelism in CUDA is no achie ed. 2.3 Pe o mance conside a ions Al hough CUDA p og amming model is lexible enough o un any kind o algo- i hm, he achie ed pe o mance depends on how he p og amme had designed he code, and on he a ge GPU unning he p og am. A CUDA p og amme has o pe ec ly know he CUDA p og amming model, bu also he idea o he GPU a chi ec u e, since i p o ides he es ic ions o be conside ed in o de o achie e peak pe o mance. The e a e se e al s a egies o accomplish i . Nex , we s and ou wo o hem: •Emphasize pa allelism: he wa p is he b anching g anula i y on CUDA; ha is, he pa allelism uni . Thus, wa ps mus be maximized wi h ac i e h eads, bu minimizing b anch di e gence be ween h ead: hey mus be execu ing he same ins uc ion simul aneously o each peak pe o mance. •Exploi memo y bandwid h: he peak bandwid h o using bo h global and sha ed memo ies is achie ed mainly by an access pa e n: coalesced access o con igu- ous (aligned) memo y posi ions. Da a is ans e ed om memo y o he GPU ha dwa e in blocks, which is o med by con iguous by es in memo y. Thus, we mus maximize hese blocks wi h he access o con iguous memo y add esses by con iguous h eads wi hin a wa p. The Reduc ion P oblem in CUDA and I s Simula ion wi h P Sys ems 95 2.4 Pa allel Reduc ion in CUDA The educ ion p oblem consis s in applying an ope a o o a se o elemen s. Le us assume a se o nelemen s {a1, . . . , an}, and he bina y and associa i e, educ ion ope a o ⊕. The esul o applying educ ion o he se o elemen s is ano he elemen a=a1⊕a2⊕. . . an. Reduc ion is a well-known p imi i e in Pa allel Compu ing, since i esides inside many impo an algo i hms. Fo example, i can be used o compu e he sum o he maximum o an a ay o numbe s. Nowadays, educe is pa o he mos used algo i hm in Big Da a and No-SQL da a bases, which is Map-Reduce. A common way o sol e his p oblem in pa allel is by using a ee, in which pa ial solu ions a e compu ed o each he inal one. The ime complexi y o his solu ion is O(logn). The p ocess is summa ized in igu e 2. Fig. 2. Scheme o a pa allel educ ion solu ion The mos popula CUDA implemen a ions o he educ ion p imi i e can be ound on he speech gi en by Ha is [8] in 2007. This documen in oduces se en ke nels om a didac ic pe spec i e, in a pe o mance-inc easing o de . Nex , we discuss he i s ou ones. In e lea ed add essing Le us assume ha he inpu a ay o elemen s is o size n, and ha he maximum amoun o h eads pe block is n . Tha means ha we will need o use nb =n n h ead blocks o p ocess he whole a ay. Howe e , i each block compu e educe o n elemen s, wha would we do wi h he nb pa ial esul s? The answe is o ha e a second ke nel, wi h a single block ha ing nb h eads, which will compu e 96 R. Ce e chi, M. ´ Angel Ma ´ınez-del-Amo , M.J. P´e ez–Jim´enez again educe. I is s aigh o wa d o add mo e ke nels in his way un il ha ing nb < n . In wha ollows, we will assume ha he size o he inpu a ay o elemen s is less han n (maximum numbe o h eads pe block). This will mean ha jus one h ead block is enough o compu e educe. We will dis ega d he second ke nel o pa ial esul s. A nai e implemen a ion o he ee solu ion in CUDA is o launch n h eads and co espond each one wi h an elemen . Fi s , each h ead wi h e en ID (called ac i e h eads) will compu e he pa ial esul wi h he nex elemen , and he p ocess will con inue by hal ing he numbe o ac i e h eads. This p ocess, called educe0 (in e lea ed add essing), is shown in Figu e 3 (whi e-a owed h eads a e inac i e). Fig. 3. Reduce0: in e lea ed add essing. The main d awbacks o his solu ion a e: •Wa ps a e no ul illed. Since he add essing is in e lea ed, wa ps can be illed by ac i e h eads up o he hal . The e o e, we a e no maximizing memo y. •Access o memo y is also in e lea ed, so he peak bandwid h canno be eached. Sequen ial add essing A solu ion o he d awbacks ound in in e lea ed add essing is o compac he memo y accesses o con iguous h eads. In o de o implemen his app oach, i The Reduc ion P oblem in CUDA and I s Simula ion wi h P Sys ems 97 will be necessa y o change he ee o he educe p imi i e solu ion. Now, he i s hal o h eads will access o he i s hal o he a ay o compu e he pa ial solu ions wi h he second hal . I can be seen ha he access is coalesced in his way, as well as wa ps a e also ul illed. Figu e 4 shows he scheme o his app oach, called educe 3, sequen ial add essing. Again, whi e-a owed h eads a e inac i e in he beginning o he p ocess. Fig. 4. Reduce3: sequen ial add essing. As i can be no iced, he main d awback o his solu ion is ha hal o he h eads a e idle on i s loop i e a ion, wha is a was e o esou ces. Fi s add du ing load The hi d app oach, called educe4 i s add du ing load, is based on aking ad- an age o he h eads which a e inac i e a he beginning o educe3. The idea is o hal e he numbe o blocks, and o use all he h eads a he beginning o apply he educ ion ope a ion o he co esponding elemen wi h he elemen ha would ha e co esponded o he a oided ex a block. Tha is, educe4 will compu e i s he a ay {a0,8, a1,9, a2,10, a3,11, a4,12, a5,13, a6,14, a7,15}, and p oceed as in educe3. 98 R. Ce e chi, M. ´ Angel Ma ´ınez-del-Amo , M.J. P´e ez–Jim´enez 3 The Simula ion wi h P Sys ems In his sec ion we use he o mal ools de eloped in [7] o p oduce a s aigh o wa d simula ion wi h P sys ems o he educ ion p oblem sol ed in CUDA as p esen ed in sec ion 2. In [7] only SIMD machines we e conside ed, and algo i hms o which communica ion ook place only ia a ne wo k o communica ion and no ia a sha ed memo y. The p esen case is di e en , we ha e communica ion ia sha ed memo y. As a i s s ep we cons uc o he CUDA model a P sys em Π(C) in he spi i o Theo em 5 o [7]. The sys em mus e lec in i s memb ane s uc u e he pa icula CUDA a chi ec u e. As a second s ep, we ollow Theo em 7 o [7], and cons uc Π(C, Y ) o Ya educ ion algo i hm. We mus speci y o each algo i hm he speci ic sequence o pai s (g aph, ules) which compose Rµ(Y). Le G aphs deno e he se o all possible g aphs ha ing n e ices labeled P1,· · · , Pn. Ha ing ixed he e ices, each elemen o G aphs will be uniquely iden i ied by he speci ic se o edges. A dis inguished elemen o G aphs is he iden i y g aph, deno ed in he sequel Id: ( he se o e ices is ixed as men ioned abo e) he se o edges is de ined as Id ={(i, i)|1≤i≤n}. Ano he dis inguished elemen o G aphs is he o al g aph, deno ed G o al: he se o edges is de ined as G o al ={(i, j)|1≤i, j ≤n}. One can also conside he s ic o al g aph, deno ed G+ o al: wi h se o edges de ined as G+ o al ={(i, j)|1≤i, j ≤n, i 6=j}=G o al Id. We ecall om [7] he ollowing wo de ini ions. De ini ion 3.1 A P sys em wi h dynamic communica ion g aphs is a cons uc Π=< V, P1,· · · , Pn, Rµ>, whe e P1,· · · , Pna e elemen a y memb anes, and Vis an alphabe o symbols used o codi y he con en s o he memb anes. Rµis a se o pai s [g aph, ules], wi h g aph ∈G aphs and such ha : (i) i g aph ⊆Id hen i s associa ed ules a e ew i ing ules; (ii) i g aph ⊆G+ o al hen i s associa ed ules a e communica ion ules. De ini ion 3.2 A P sys em wi h dynamic communica ion g aphs will be called wi h ini e sequen ial suppo i he se Rµis bo h ini e and o ally o de ed, i.e., i i is a ini e sequence. The Reduc ion P oblem in CUDA and I s Simula ion wi h P Sys ems 99 Le Vbe an alphabe o symbols wi h which we will codi y he con en s o he memb anes. An in ege nwill be codi ied as an(nappa i ions o he symbol a, wi h a∈V). We assume we sol e he educ ion p oblem o a bina y commu a i e and associa i e ope a ion ∗, and we assume we can compu e n∗minside a memb ane by ew i ing: i.e. we ha e symbols a, b ∈Vand a ew i ing ule ∗(a, b) such ha ∗(a, b)(anbm) = an∗m. We associa e a hie a chical memb ane s uc u e o he CUDA componen s in he ollowing manne : (1) each h ead is an elemen a y memb ane; (2) each block is a memb ane con aining he elemen a y ones associa ed o i s h eads; (3) he global memo y is a sepa a e elemen a y memb ane. The p esen a ion o his memb ane s uc u e is µ= (B0(P00, P01,· · · P0n),· · · , Bk(Pk0, Pk1,· · · Pkn), Mg), whe e n= 2 is he numbe o h eads pe block, Mgis he global memo y mem- b ane, Pij is he memb ane co esponding o h ead jo block i, and when we eason inside a block he subsc ip co esponding o he block may be omi ed. I we ha e a se Rµo pai s (g aph, ules) o obey he condi ions o he de i- ni ion, hen he cons uc Π(C) = (V, (B0(P00, P01,· · · P0n),· · · , Bk(Pk0, Pk1,· · · Pkn), Mg), Rµ) = (V, µ, Rµ) is a P sys em wi h dynamic communica ion g aphs. He e a discussion may s a , compa ing he P sys ems de ised in [7] o SIMD- Xmachines, and a po en ial simila candida e o he CUDA pa adigm. Such a candida e depends on a good de ini ion o Rµ, o , a leas , he o mula ion o c i e ia o ’admissible’ candida es. We open he way o his discussion, which is also a e lec ion on he powe and he limi a ions o he o malism in oduced in [7], by simula ing he sol ing o he educ ion p oblem in CUDA. Mo e p ecisely, in he ollowing we cons uc se s Rµ(Y) wi h he p ope y ha Π(C, Y )=(V, µ, Rµ(Y)) is a P sys em wi h dynamic communica ions g aph, wi h ini e sequen ial suppo , which simula es Y, a educ ion algo i hm among he ones p esen ed in Sec ion 2. The admissible communica ion g aphs will ha e o e lec he communica ion p ope ies o CUDA. We will ha e communica ion edges be ween Mgand each h ead memb ane Pji o simula e he eading om and he w i ing o global mem- o y. Be ween he indi idual elemen a y memb anes which simula e he h eads we can ha e communica ion only inside he same block, so he communica ion g aph will ha e sepa a e connec ed componen s o each block. Rew i ing ules inside memb anes will be associa ed o subg aphs o Id and communica ion ules o subg aphs o G+ o al. We use he sho hand no a ion (A, B, x) o symbol x a eling on an o ien ed edge (A, B) om A o B, (G, x) o symbol x a elling on all o ien ed edges o G,