Comparative Analysis of OpenACC Compilers

Compa a i e Analysis o OpenACC Compile s Daniel Ba ba, A u o Gonzalez-Esc ibano, and Diego R. Llanos ? Uni e sidad de Valladolid, Depa amen o de In o ma ica, Valladolid, Spain {daniel|a u o|diego}@in o .u a.es Abs ac . OpenACC has been on de elopmen o a ew yea s now. The OpenACC 2.5 speci ica ion was ecen ly made public and he e a e some ini ia i es o de eloping ull implemen a ions o he s anda d o make use o accele a o capabili ies. The e is much o be done ye , bu cu - en ly, OpenACC o GPUs is eaching a good ma u i y le el in a ious implemen a ions o he s anda d, using CUDA and OpenCL as backends. N idia is in es ing in his p ojec and hey ha e eleased an OpenACC Toolki , including he PGI Compile . The e a e, howe e , mo e de elop- men s ou he e. In his wo k, we analyze di e en a ailable OpenACC compile s ha ha e been de eloped by companies o uni e si ies du - ing he las yea s. We check hei pe o mance and ma u i y, keeping in mind ha OpenACC is designed o be used wi hou ex ensi e knowledge abou pa allel p og amming. Ou esul s show ha he compile s a e on hei way o a easonable ma u i y, p esen ing di e en s eng hs and weaknesses. 1 In oduc ion OpenACC is an open s anda d which de ines a collec ion o compile di ec i es o p agmas o execu ion o code blocks on accele a o s like GPUs o Xeon Phi cop ocesso s. OpenACC aims o educe bo h he equi ed lea ning ime and he pa alleliza ion o sequen ial code in a po able way [1]. OpenACC speci ica ion is cu en ly on i s 2.5 e sion [2], which has been eleased ecen ly. OpenACC was ounded by N idia, CRAY, CAPS and PGI, bu now he e is a la ge lis o conso ium membe s, bo h om he indus y and academy, including he Oak Ridge Na ional Labo a o y, he Uni e si y o Hous on, AMD, and he Edinbu gh Pa allel Compu ing Cen e (EPCC), among o he s. The Co po a e O ice s a e, a he ime o w i ing his pape , om N idia, Oak Ridge Na ional Labo a o y, CRAY and AMD. Academic membe ships a e a ailable o in e es ed ins i u ions. The e a e se e al compile s suppo ing OpenACC. The PGI Compile ( om he Po land G oup, which is a subsidia y o N idia o some ime now) is be- ing dis ibu ed as pa o he N idia OpenACC Toolki , unde a ee 90-day ?This esea ch has been pa ially suppo ed by MICINN (Spain) and ERDF p og am o he Eu opean Union: HomP og-He Sys p ojec (TIN2014-58876-P), CAPAP-H5 ne wo k (TIN2014-53522-REDT), and COST P og am Ac ion IC1305: Ne wo k o Sus ainable Ul ascale Compu ing (NESUS). Thanks also o D . Hec o O ega and D . Ja ie F esno o hei hough ul commen s, help, and encou agemen . 2 ial license, a ee annual academic license, o a comme cial license. The PGI Compile uses CUDA o OpenCL as backend. CAPS en e p ise, a p o ide o so wa e and se ices o he High Pe o mance Compu ing communi y, also de- eloped a compile which suppo s CUDA and OpenCL. Howe e , he company is no longe in business and i s de elopmen is no a ailable anymo e. CRAY Inc. has i s own OpenACC compile , a ailable wi h hei compu e s. I is e- po ed o be one o he mos ma u e comme cial compile s. Howe e , hey ha e no ye o e ed an academic o ial license o he pu poses o s udies like his one. Pa hscale Inc., a compile and mul ico e so wa e de elope , also made an OpenACC implemen a ion on hei ENZO compile . Un o una ely, a e p i a e communica ions hey we e eluc an o allow us o use hei compile o his s udy. The e a e also se e al academic a emp s o de eloping an OpenACC com- pile . In pa icula , OpenUH [3] de eloped a he Uni e si y o Hous on, and accULL [4] om Uni e sidad de La Laguna (Spain). Bo h o hem a e a ailable o ee o anyone in e es ed. This wo k p esen s a s udy abou he le el o suppo o OpenACC in he a ailable compile s, examining hei s eng hs and weaknesses, and gi ing in- sigh s on hei pe o mance. We analyze he le el o ma u i y o each compile , in e ms o hei com- ple eness in he suppo o he s anda d and obus ness, using each compile ’s documen a ion o check wha pa s o he speci ica ion ha e been implemen ed. We check he suppo o OpenACC compile di ec i es wi h he help o a bench- ma k sui e, de eloped by he Edinbu gh Pa allel Compu ing Cen e, which will be desc ibed la e . Ano he aspec o es is he ela i e pe o mance o he gene a ed code. Fo his, we would wan o un mo e complex applica ions and measu e he di e ences be ween he pe o mance o he execu able code gene a ed by each compile . The ideal si ua ion would be o es applica ions as close o eal wo ld p oblems as possible, a oiding syn he ic code agmen s. Since he use o OpenACC is no common ye , we ha e o ely on exis ing benchma ks. A his poin , compa ing he esul s om OpenACC code wi h CUDA o OpenCL di ec implemen a ions migh seem app op ia e, bu po ing he di e en bench- ma ks o hese languages makes he esul dependen on human in e e ence, as he de elope ’s abili y o CUDA p og amming impac s on he pe o mance. The expe imen s conduc ed in his wo k we e ca ied ou using se e al bench- ma ks. Fi s , he EPCC benchma k sui e which con ains a g oup o 13 ke nels po ed o OpenACC, called “Le el 1” benchma ks. This benchma k sui e also con ains h ee eal applica ions called “Himeno”, “27s encil”, and “le co e” [5]. We ha e also used he Pa hscale po o he Rodinia benchma k [6]. We also wan ed o use he OpenACC Valida ion Tes sui e [7] de eloped by he Uni e - si y o Hous on, bu a his momen ha ool is only a ailable o OpenACC membe s. 3 Ou conclusion is ha he di e en compile s a e on hei way o a easonable ma u i y. Howe e , he e is a numbe o ea u es no ully implemen ed ye by some o he compile s. The es o his pape is o ganized as ollows. Sec ion 2 desc ibes he selec ed compile s. Sec ion 3 shows he cha ac e is ics o he benchma k sui es chosen, enume a ing some p oblems encoun e ed when compiling hem wi h he compil- e s selec ed. Sec ion 4 con ains he esul o ou analysis, in e ms o comple eness o he OpenACC ea u es suppo ed, obus ness o compile implemen a ions, and ela i e pe o mance o he gene a ed code. Finally, Sec ion 5 concludes ou pape . 2 A ailable Compile s In he in oduc ion we men ioned se e al compile s. In his sec ion we desc ibe wi h mo e de ail he compile s we we e able o ob ain and use o his s udy, and we will discuss hei ins alla ion pa icula i ies on ou Linux based pla o m. 2.1 PGI Compile The PGI Compile [8] is being de eloped by The Po land G oup, being owned by N idia. This compile is widely used in webina s, wo kshops, and con e ences. The PGI Compile is, a he ime o w i ing his pape , a ailable o download as pa o he OpenACC Toolki om N idia. This oolki includes a 90-day ee ial, he possibili y o acqui ing an academic license o a whole yea , o buying a comme cial license. 2.2 accULL The accULL [4] compile de eloped by Uni e sidad o La Laguna (Spain) is an open sou ce ini ia i e. accULL consis s on a s uc u e o wo laye s con aining YaCF [9] (Ye ano he Compile F amewo k) and F angollo [10], a un ime li- b a y. YaCF ac s as a sou ce- o-sou ce ansla o while F angollo wo ks as an in e ace p o iding he mos common ope a ions ound in accele a o s. 2.3 OpenUH The OpenUH [3] compile , de eloped by he Uni e si y o Hous on (USA) is ano he open sou ce ini ia i e. I makes use o Open64, a discon inued open- sou ce op imizing compile . 3 Benchma k Desc ip ion This sec ion desc ibes he di e en benchma ks used in ou wo k, enume a ing he main cha ac e is ics ha make hem in e es ing o his s udy, and any issue de ec ed du ing hei compila ion wi h he h ee compile s s udied. 4 3.1 EPCC OpenACC Benchma ks This benchma k sui e [11] has been de eloped by he Edinbu gh Pa allel Com- pu ing Cen e (EPCC). The benchma ks a e di ided in h ee ca ego ies: “Le el 0”, “Le el 1” and “Applica ions”. The compiled p og am launches all he bench- ma ks in he sui e sequen ially. By de aul , he numbe o epe i ions is en, and he esul is he a e age o each benchma k. Time is measu ed in mic oseconds in double p ecision, using he OpenMP unc ion omp ge w ime(). We desc ibe b ie ly he benchma ks included bellow: Le el 0 Le el 0 includes a collec ion o small benchma ks ha execu e single hos and accele a o ope a ions, such as memo y ans e s. Le el 1 Le el 1 benchma ks [12] consis on a se ies o BLAS- ype ke nels. They a e based on Polybench [13] and Polybench/GPU ke nels. They measu e he pe o mance o execu ing hose codes. These benchma ks a e un on he CPU i s in o de o ha e esul s o compa e hose ob ained on he GPU. A b ie desc ip ion o he di e en issues ound while unning his sui e o hese compile s ollows. OpenUH: The ollowing benchma ks canno be compiled due o unsuppo ed p agmas p esen in hei code: ke nels i : P oblems using #p agma ke nels i (0). pa allel p i a e: P oblems decla ing pa ams as p i a e. pa allel i s p i a e: P oblems decla ing pa ams as i s p i a e. le co e: P oblems wi h non scala poin e s. himeno: P oblems wi h non scala poin e s. accULL: The e is a p oblem ela ed o a unc ion poin e in he hos p og am. The compile , du ing he sou ce o sou ce ansla ion modi ies he syn ax o he unc ion poin e . A double (* es )( oid) is con e ed o a double *( es ( oid)). This was sol ed by manually co ec ing his change in he in e media e C code gene a ed, e-compiling he objec ile and copying i o he main di ec o y o link all he objec iles again. No wa ning om any o he p agmas was de ec ed so we we e able o un all he benchma ks. 3.2 Rodinia OpenACC Rodinia is a benchma k sui e o he e ogeneous compu ing [14,15]. I includes applica ions and ke nels o mul ico e CPU o GPU applica ions. The e is an e o o po exis ing Rodinia benchma ks o OpenACC. Pa h- scale [6] is wo king on his. We ha e es ed hei Rodinia e sion commi ed o Gi Hub on Ap il 25, 2014. Mos o he sui e wo ks wi h PGI, bu OpenUH and accULL ha e many p oblems o compile mos o he es s. We ha e been able o success ully compile he ollowing benchma ks con ained in he sui e wi h wo o mo e compile s: gaussian, nw, lud, c d, ho spo , pa h inde , and s ad2. 5 4 E alua ion In his sec ion we analyze he OpenACC compile s, using bo h documen a ion and expe imen a ion. We use each compile ’s documen a ion o check he com- ple eness o OpenACC ea u es suppo ed. Then we use he EPCC benchma ks o check bo h obus ness and ela i e pe o mance. Finally we check h ead-block size sensibili y, measu ing he impac on pe o mance o di e en geome ies. 4.1 Expe imen al Se up We used a N idia GTX Ti an Black o un he expe imen s. This GPU con ains 2880 CUDA co es wi h a clock a e o 980Mhz and 15 SMs. I has 6GB o RAM, and Compu e Capabili y 3.5. The hos is a Xeon E5-2690 3 wi h 12 co es a a clock a e o 1.9GHz, and 64GB in ou 12GB modules. The PGI compile is he one con ained in he N idia OpenACC Toolki , e - sion 15.7-0, published in Jul 13, 2015. We used OpenUH e sion 3.1.0 (published in No embe 4, 2015), based on Open64 e sion 5.0 and using GCC 4.2.0, p e- buil , downloaded om he High Pe o mance Compu ing Tools g oup websi e [16]. accULL is e sion 0.4alpha (published in No embe 28, 2013), downloaded om Uni e sidad de La Laguna’s esea ch g oup “Compu aci´on de Al as P es a- ciones” [17]. 4.2 Comple eness o OpenACC Fea u es Suppo ed F om each compile documen a ion we ge some insigh on he comple eness o he OpenACC ea u es suppo ed. F om his in o ma ion, we can conclude ha he OpenACC s anda d is no ully implemen ed ye by any o he a ailable compile s. The e is wo k o be done, bu he h ee compile s a e a a espec able ma u i y le el. 4.3 Robus ness and P agma Implemen a ion The EPCC Benchma k sui e con ains se e al benchma ks o es ing OpenACC di ec i es. These benchma ks a e con ained in he “Le el 0” g oup, which has been desc ibed in he p e ious sec ion. Table 1 con ains he esul s ob ained o he h ee compile s. In his sec ion we enume a e he p oblems wi h each benchma k and we explain he esul s ob ained, including he o e head o he di e en p agma implemen a ions. Excep o Upda e hos ,Ke nels In oc., and Pa allel In oc., he ime shown is he di e ence be ween execu ing and no execu ing each p agma. When he o e head is ze o (o he p agma is no implemen ed), he imes a e e y simila , wi h minimal s ochas ic a ia ion. These a ia ions may p oduce a e y small nega i e esul when calcula ing he di e ence. When di e ences in ime a e on he o de o ens o mic oseconds (posi i e o nega i e), i can be assumed ha he e is no di e ence in ime be ween he di e en e sions es ed in ha benchma k. 6 Table 1: EPCC le el 0: di ec i e’s o e head (in µsec), 1 MB da ase EPCC L0 PGI OpenUH accULL Ke nels i -37.50 Fail 4.54 Pa allel i -30.76 -0.48 1237.02 Pa allel p i a e -21.94 Fail 51.09 Pa allel 1s p i Fail Fail -213.83 Ke nels comb. -1.67 -108.43 -127.17 Pa allel comb. -0.05 -2.74 33.38 Upda e hos 478.63 373.22 548.77 Ke nels In oc. Fail 12.76 2398.20 Pa allel In oc. 31.81 13.47 1377.88 Pa allel educ . -14.85 -164.41 -2168.12 Ke nels educ . -8.49 -172.31 -2009.11 PGI The e was a p oblem wi h he “Ke nels In oca ion” benchma k: I e- u ned an inco ec esul . The code was no being pa allelized and he p agmas we e igno ed because i wasn’ speci ically s a ed ha he i e a ions we e inde- penden . This could be sol ed adding he keywo d es ic o he poin e o he clause independen o he p agma. The “ke nels i ” and “pa allel i ” esul s a e e y simila , and in bo h cases he esul s indica e ha he code wi h he p agma is sligh ly as e han he one wi hou i , e en hough bo h a e being un on he hos . In [5] i was s a ed ha his could be because o op imiza ions done by he compile while o a e p ocessing he p agmas. The “pa allel p i a e” benchma k shows ha he c ea ion o p i a e a iables o each h ead unning he loop is sligh ly as e han he alloca ion o de ice memo y. “Ke nels combined” shows a e y small di e ence o ime be ween w i ing wo p agmas ins ead o a combined one, he o me being sligh ly as e han he la e al hough he di e ence is almos negligible. The same occu s o he “pa allel combined” benchma k, he di e ence being smalle in his case. Finally “Pa allel educ ion” and “Ke nels educ ion” show ha PGI has e y li le o e head o he educ ion clause. In [5] i is s a ed ha he PGI compile does he educ ion e en i i is no anno a ed. This could explain he e y small di e ence in bo h benchma ks. OpenUH We go some e o s du ing compila ion o he “Ke nels i ”, “Pa - allel p i a e” and “Pa allel 1s p i a e” benchma ks so hey a e igno ed in his analysis. Howe e , he “Pa allel i ” di ec i e is suppo ed and he di e ence be- ween using he p agma o un code on he hos o unning i di ec ly is almos negligible. “Ke nels combined” shows an o e head o he combined p agma e sus he sepa a ed e sion. Howe e , his is no he case o he “Pa allel combined” benchma k, whe e he di e ence is much smalle . The in oca ion o ke nels and 7 pa allel di ec i es a e e y simila . And o bo h o hem, he educ ion adds a simila o e head. This migh be ela ed o OpenUH assuming loops o be independen inside ke nels egions. accULL No e o s we e shown while compiling o unning he benchma ks wi h accULL. The e is a big di e ence be ween he wo e sions con ained in he “Ke nels i ” and he “Pa allel i ” benchma ks, whe e he ke nels di ec i e e sion has a e y small o e head compa ed o he non-anno a ed code. This o e head is e y la ge in he pa allel di ec i e e sion. This is explained by he accULL de elope s in [5] whe e hey say ha he absence o a loop clause in he pa allel di ec i e is causing he loop o be execu ed sequen ially in each h ead. The e o e, his clause is no co ec ly suppo ed, as we unde s and om he OpenACC Speci ica ion ha he loop should be execu ed only on he hos . Robus ness Summa y The o e all esul s indica e ha some o he clauses a e no implemen ed ye , bu he h ee compile s a e in hei way o a easonable ma u i y le el and, since he mos used di ec i es a e wo king, hey can ac ually be used o code pa alleliza ion using OpenACC. 4.4 Rela i e Pe o mance o Gene a ed Code In his sec ion we analyze he pe o mance o he gene a ed code desc ibing he impac o p agmas o e head in accULL. Pe o mance measu emen is di- ided in o da a mo emen , whe e we analyze he esul s o he da a mo emen benchma ks in Le el 0 o EPCC OpenACC Benchma k Sui e, and execu ion pe - o mance, using Le el 1 and Applica ion Le el o EPCC OpenACC Benchma k Sui e, and selec ed benchma ks om Rodinia. E ec o P agmas O e head in accULL Some esul s om he Le el 0 o he EPCC Benchma k Sui e show a pe o mance impac in oduced by some clauses and di ec i es in he accULL gene a ed code. Ke nels and Pa allel in oca ions in accULL ha e a highe o e head han o he compile s. This is due o he un ime calls and i is specially no iceable in he educ ion clause. These o e heads accumula ion does no ha e a signi ican impac o complex ke nels, o launching he same ke nel o e and o e again. Howe e , his could be a p oblem when unning simple ke nels o many di e en small ke nels. This is he main eason behind he o e all esul s showing a wo se pe o mance o he accULL compile in his analysis. Da a Mo emen Da a mo emen pe o mance can be measu ed in ou bench- ma ks om he Le el 0 o he EPCC OpenACC benchma k sui e. We ha e launched 10 epe i ions o hose benchma ks wi h da asizes o 1 kB, 1 MB, 10 MB, and 1 GB. The esul s can be seen in Tables 2, 3, 4, and 5. 8 Table 2: EPCC da a mo emen esul s (in µsec), 1 kB da ase . Whi e cells highligh he bes esul s. no m. is he no malized esul using PGI as e e ence. Da a M mn PGI OpenUH accULL 10 eps, 1kB ime no m. ime no m. ime no m. Con igH2D 30.827 1.0 322.699 10.47 338.218 10.97 Con igD2H 14.686 1.0 323.319 22.01 343.919 23.42 SlicedH2D 12.087 1.0 310.897 25.72 315.914 26.13 SlicedD2H 14.948 1.0 324.010 21.67 327.714 21.92 GeoMean 18.93 GeoMean 19.58 Table 3: EPCC da a mo emen esul s (in µsec), 1 MB da ase . Whi e cells highligh he bes esul s. no m. is he no malized esul using PGI as e e ence. Da a M mn PGI OpenUH accULL 10 eps, 1MB ime no m. ime no m. ime no m. Con igH2D 484.347 1.0 950.789 1.96 727.839 1.50 Con igD2H 461.936 1.0 632.691 1.37 792.761 1.72 SlicedH2D 17.094 1.0 267.982 15.68 274.462 16.06 SlicedD2H 36.335 1.0 254.702 7.01 285.685 7.86 GeoMean 4.14 GeoMean 4.24 Table 4: EPCC da a mo emen esul s (in µsec), 10 MB da ase . Whi e cells highligh he bes esul s. no m. is he no malized esul using PGI as e e ence. Da a M mn PGI OpenUH accULL 10 eps, 10MB ime no m. ime no m. ime no m. Con igH2D 4141.402 1.0 6887.984 1.66 3354.666 0.81 Con igD2H 5876.088 1.0 2043.747 0.35 4396.052 0.74 SlicedH2D 27.322 1.0 404.214 14.79 427.203 15.64 SlicedD2H 48.017 1.0 269.818 5.62 280.203 5.84 GeoMean 2.64 GeoMean 2.72 Table 5: EPCC da a mo emen esul s (in µsec), 1 GB da ase . Whi e cells highligh he bes esul s. no m. is he no malized esul using PGI as e e ence. Da a M mn PGI OpenUH accULL 10 eps, 1GB ime no m. ime no m. ime no m. Con igH2D 32310.009 1.0 788945.913 24.42 296340.991 9.17 Con igD2H 55179.119 1.0 553282.976 10.03 347280.359 6.29 SlicedH2D 400.066 1.0 535.011 1.34 533.943 1.34 SlicedD2H 158.071 1.0 2818.100 17.83 4294.407 27.17 GeoMean 8.75 GeoMean 6.76 9 In [5] i was s a ed ha PGI used pinned memo y and ha i was causing issues in smalle da ase s. I seems ha PGI has sol ed his issue since hen and, looking a he documen a ion, i is now possible o speci y he ype o memo y access we wan wi h a compila ion lag. When using la ge da ase s, OpenUH and accULL do no show he expec ed esul s acco ding o he e olu ion shown in ables 2, 3, and 4. We guess ha his is ela ed o he usage o pinned memo y by he PGI Compile , allowing i o ob ain be e esul s when da ase s a e big enough. Execu ion Pe o mance, EPCC Benchma ks In his sec ion we will ana- lyze he pe o mance o he code gene a ed by he PGI, OpenUH, and accULL compile s wi h he benchma ks con ained in he EPCC Le el 1 and Applica ion le el. We use h ee di e en da ase s: 1kB, 1MB, and 10MB. This choice is based on he ac ha bigge da ase s esul in an ou o memo y e o due o how he benchma ks y o alloca e memo y on he de ice. We suspec he memo y alloca ion is being done in each h ead inside he gene a ed ke nels, using mo e memo y han expec ed. In summa y, PGI code ob ains be e esul s in almos e e y benchma k. Howe e , he di e ences sho en when using bigge da ase s. Table 6: EPCC execu ion esul s (in µsec), 1 kB da ase . Whi e cells highligh he bes esul s. Exec. ime PGI OpenUH accULL 10 eps, 1kB ime no m. ime no m. ime no m. 2MM 99.087 1.0 522.304 5.27 2799.229 28.25 3MM 80.204 1.0 380.683 4.75 3799.048 47.37 ATAX 58.103 1.0 327.110 5.63 2564.702 44.14 BICG 72.408 1.0 350.380 4.84 2628.499 36.30 MVT 80.037 1.0 354.743 4.43 2665.299 33.30 SYRK 68.426 1.0 289.512 4.23 2394.803 35.00 COV 87.261 1.0 314.617 3.61 3795.372 43.49 COR 104.976 1.0 337.362 3.21 5208.668 49.62 SYR2K 73.290 1.0 317.574 4.33 2469.765 33.70 GESUMMV 65.613 1.0 312.996 4.77 1500.021 22.86 GEMM 49.710 1.0 323.725 6.51 1237.473 24.89 2DCONV 46.444 1.0 286.174 6.16 1207.528 26.00 3DCONV 45.514 1.0 285.792 6.28 1202.494 26.42 27S 335.884 1.0 432.801 1.29 3273.728 9.75 LE2D 6842374 1.0 * * * * HIMENO 547939 1.0 * * * * GeoMean 4.39 GeoMean 24.38 Fo da ase s o 1kB he esul s can be seen in Table 6. Benchma ks ha ail o execu e wi h a speci ic compile a e shown wi h an as e isk in he able. PGI code shows a e y good pe o mance, ollowed by he OpenUH code, which also