Parallel & Hybrid Programming

UNIVERSITY OF BURGOS A ea o Elec onic Technology Pa allel & Hyb id P og amming. José Ma ía Cáma a Neb eda, Césa Rep esa Pé ez, Ped o Luis Sánchez O ega Pa allel & Hyb id P og amming . 2015 A ea o Elec onic Technology Elec omechanical Enginee ing Depa men Uni e si y o Bu gos In oduc ion ............................................................................................................................................. 5 Ac i i y 1: MPI Ma ix Mul iplica ion ...................................................................................................... 7 OBJETIVES ............................................................................................................................................ 7 THEORETICAL CONCEPTS..................................................................................................................... 7 PRACTICAL EXERCISE ........................................................................................................................... 7 QUESTIONS .......................................................................................................................................... 8 Ac i i y 2: Pe o mance Assessmen ....................................................................................................... 9 OBJECTIVES .......................................................................................................................................... 9 THEORETICAL CONCEPTS..................................................................................................................... 9 PRACTICAL EXERCISE ......................................................................................................................... 12 QUESTIONS ........................................................................................................................................ 13 Ac i i y 3: In oduc ion o Hyb id P og amming ................................................................................... 14 OBJETIVES .......................................................................................................................................... 14 THEORETICAL CONCEPTS................................................................................................................... 14 PRACTICAL EXERCISE ......................................................................................................................... 17 QUESTIONS ........................................................................................................................................ 17 Ac i i y 4: Hyb id P og amming ............................................................................................................ 18 OBJECTIVES ........................................................................................................................................ 18 THEORETICAL CONCEPTS................................................................................................................... 18 PRACTICAL EXERCISE ......................................................................................................................... 19 Ac i i y 5: MPI s OpenMP .................................................................................................................... 20 OBJECTIVES ........................................................................................................................................ 20 THEORETICAL CONCEPTS................................................................................................................... 20 PRACTICAL EXERCISE ......................................................................................................................... 20 Ac i i y 6: Submi ing jobs o a clus e .................................................................................................. 21 OBJETIVES .......................................................................................................................................... 21 THEORETICAL CONCEPTS................................................................................................................... 21 PRACTICAL EXERCISE ......................................................................................................................... 26 Ac i i y 7: Job scheduling ...................................................................................................................... 27 OBJETIVES .......................................................................................................................................... 27 THEORETICAL CONCEPTS................................................................................................................... 27 PRACTICAL EXERCISE ......................................................................................................................... 30 Ac i i y 8: Job scheduling II ................................................................................................................... 31 OBJETIVES .......................................................................................................................................... 31 THEORETICAL CONCEPTS................................................................................................................... 31 PRACTICAL EXERCISE ......................................................................................................................... 31 Ac i i y 9: Job scheduling III .................................................................................................................. 32 OBJETIVES .......................................................................................................................................... 32 THEORETICAL CONCEPTS................................................................................................................... 32 PRACTICAL EXERCISE ......................................................................................................................... 32 Ac i i y 10: Pe o mance compe i ion. ................................................................................................. 33 OBJETIVES .......................................................................................................................................... 33 THEORETICAL CONCEPTS................................................................................................................... 33 PRACTICAL EXERCISE ......................................................................................................................... 33 Appendix A: Ins alling DeinoMPI ........................................................................................................... 34 Ins alla ion ......................................................................................................................................... 34 Con igu a ion ..................................................................................................................................... 34 Launching Jobs .................................................................................................................................. 34 G aphic En i onmen ........................................................................................................................ 34 Deino MPI manual. A ailable a : h p://mpi.deino.ne /manual.h m ................................................... 38 Appendix B: P ojec Con igu a ion in Visual S udio 2010 ..................................................................... 39 Appendix C: Con igu a ion o MS-MPI. ................................................................................................. 44 LABORATORY GUIDE In oduc ion 5 In oduc ion Ou in e es will be ocused on pa allel p og aming o mul icompu e MIMD machines. Ou applica ion p og ams will spli in o se e al p ocesses and each one will ha e he po en ial capabili y o be execu ed on a di e en node o ou clus e . The p ocesses c ea ed by he use will coope a e o achie e a common compu a ional objec i e. The collabo a ion will be possible due o communica ion and synch oniza ion ools p o ided by he p og amming en i onmen . Communica ion is implemen ed in he o m o message exchanging. Mos o he scena ios p oposed admi a numbe o di e en pa allel solu ions. We should y o come up wi h he mos ad an ageous in e ms o sys em pe o mance. To do so we mus ake in o accoun : • We will y o inc ease pe o mance (execu ion ime). To do so, we will y o squish he applica ion’s po en ial locali y, ha is, i s capabili y o wo k wi h local da a a oiding he need o much in o ma ion exchange be ween p ocesses. • Ano he impo an poin is “scalabili y”. In a ha dwa e en i onmen , whe e he amoun o a ailable esou ces is unknown a p og amming ime, he applica ion mus scale o make he mos o he a ailable esou ces a any ime. Pa allel p og aming is no an easy job. The heo y a ound he de elopmen o concu en and pa allel so wa e is beyond he scope o his cou se bu , we will p o ide some hin s. Pa allel p og amming, as well as sequen ial p og amming is a c ea i e ask; wha is abou o be exposed is no hing mo e han a se ies o s eps we ecommend o ollow when acing a pa alleliza ion. Le ’s spli up he p ocess in 4 s eps: • F agmen a ion: his ini ial s ep is mean o ind po en ial pa allel s uc u es wi hin he p oblem o be sol ed. As a i s app oach, we may y o decompose he job in as many small pa allel asks as possible. Two c i e ia can be ollowed o ca y ou his decomposi ion: o The unc ional way: seeks o possible di isions in he job o be ca ied ou by paying a en ion o i s na u e. o The da a way: pays a en ion o he na u e o he da a o be p ocessed ying o decompose hem in o he smalles chunks. • Communica ion: once iden i ied po en ial pa allel asks, communica ion needs be ween hem mus be analyzed. • Binding: gi en ha he cos o communica ions is high in e ms o global execu ion ime, he o me ly iden i ied asks ha e o me ge pa ially in o de o balance compu a ion and communica ion. • Mapping: once he p og am’s s uc u e is se led, he ecen ly gene a ed p ocesses ha e o be sp ead ac oss he compu e s a ailable. The s a egy o be adop ed di e s acco ding o In oduc ion LABORATORY GUIDE 6 he agmen a ion way. As a ule o humb, he e should be a leas as many p ocesses as compu e s a e a ailable in o de o p e en anyone being unused. I all compu e s a e equal, i would be ecommendable o make as c ea e as many p ocesses as compu e s. I no , he mos powe ul compu e s can hos a highe numbe o p ocesses. I is also possible o assign p ocesses o nodes on he go, hus balancing p ocesso s’ load dynamically. Depending on se e al aspec s, being he ype o compu e one o he mos ele an , pa allel p og amming admi s di e en app oaches: • Message passing: especially indica ed o dis ibu ed memo y compu e s, can be used on any ha dwa e pla o m. • Sha ed memo y: sui able only o sha ed memo y en i onmen s. • Hyb id p og amming: a combina ion o he wo p e ious. I is mean o op imize pe o mance when bo h sha ed and dis ibu ed memo y schemes a e p esen . This scena io is e y common in ecen days. Mode n clus e s and MPPs a e in eg a ed by mul ico e memo y sha ing nodes. In his cou se we will assume ha he s uden is amilia enough wi h message passing p og amming. Mo e p ecisely, he concep s gi en in he Bachelo Deg ee on Compu e Science abou MPI p og amming a e conside ed as known. O he wise i is highly ecommended o he s uden o go h ough he MPI P og amming Fundamen als cou se. A leas om ac i i y 0 o ac i i y 5. LABORATORY GUIDE Ac i i y 1: MPI Ma ix Mul iplica ion 7 Ac i i y 1: MPI Ma ix Mul iplica ion OBJETIVES  Apply p e iously acqui ed knowledge o de elop a bi mo e complex p og am in ended o be used as a benchma k o measu e sys em pe o mance. THEORETICAL CONCEPTS No new concep s will be in oduced in his chap e since i is mean o exploi hose al eady lea ned. As ob ious, no all aspec s o MPI de elopmen en i onmen ha e been exposed and no ou applica ion p og am is expec ed o ind he mos op imal solu ion bu qui e a good job is possible hough. Howe e , i may be help ul o in oduce some addi ional in o ma ion abou he unc ions we al eady know. Func ion MPI_Rec e u ns a MPI_S a us ype pa ame e ha we ha en’ used so a . I is a s uc u e in eg a ed by 3 elemen s: MPI_SOURCE, MPI_TAG & MPI_ERROR. The i s one con ains he Rank o he sende p ocess. I he message was ecei ed unde MPI_ANY_SOURCE i can be necessa y o ind ou who sen i la e on in he p og am. The second one e u ns he message’s ag. I i was ecei ed unde MPI_ANY_TAG, i could be in e es ing o ge o know he ag’s alue as well. The hi d one e u ns an e o code. We won’ deal wi h e o codes in his exe cise. PRACTICAL EXERCISE We will p og am a pa allel ma ix mul iply. I is he s uden ’s decision how o sca e calcula ions among all he p ocesses. The size o he ma ices (squa e) mus be con igu able. Dynamic memo y alloca ion is s ongly ecommended so no limi s o he size o he ma ices a e imposed. P ocess 0 will ini ialize he ope and ma ices wi h any alue ( andom, loop, e c). Da a ype will be loa . In a i s s age, mul iplica ion esul s will be displayed o check co ec ness. Once he p og am has been alida ed, esul p in ing mus be emo ed o allow ma ix size o g ow. Execu ion ime has o be displayed in all cases. REMARK: To combine double indexing wi h dynamic memo y alloca ion o ma ices, we mus use double poin e s. Each poin e wi hin an a ay will gi e access o a ow in a ma ix: // Decla e a double poi e o he ma ix // This will le us e e o he elemen s in a [ ow][column] manne loa **Ma ix; // Ini ialize he double poi e o s o e poi e s o each and e e y ow in he ma ix. Ma ix = ( loa **) malloc(ROWS*sizeo ( loa *)); // We ini ialize each poi e o he s a ing poi o each ow o (i=0; i< ROWS; i++) { Ma iX[i] = ( loa *) malloc(COLUMNS*sizeo ( loa )); } // Now we can us [ ow][column] o ma o ou ma ix: Ac i i y 1: MPI Ma ix Mul iplica ion LABORATORY GUIDE 8 o (in i=0; i<ROWS; i++) { o (in j=0; j<COLUMNS; j++) { Ma ix[i][j] = 0.0; } } Howe e , his dynamic alloca ion p ocedu e does no gua an ee ha ows in he ma ix a e con iguous in memo y. This can be necessa y o sending unc ions in ou p og am. We should send da a ow by ow in ha scena io. I we wan o keep double indexing while adding con igui y, we will ha e o p oceed as ollows: // Decla e a double poi e o he ma ix // This will le us e e o he elemen s in a [ ow][column] manne loa **Ma ix; // Ini ialize he double poi e o s o e poi e s o each and e e y ow in he ma ix. Ma iX = ( loa **) malloc(ROWS*sizeo ( loa *)); // Decla e a new poin e o alloca e memo y space o he whole ma ix. loa *M ; // Ini ialize he poin e ha will gua an ee consecu i e loca ion o all ows M = ( loa *) malloc(ROWS*COLUMNS*sizeo ( loa )); // We ini ialize each poi e o he s a ing poi o each ow. o (i=0; i< ROWS; i++) { Ma iX[i] = M + i* COLUMNS; } // Now we can us [ ow][column] o ma o ou ma ix: o (in i=0; i<ROWS; i++) { o (in j=0; j<COLUMNS; j++) { Ma ix[i][j] = 0.0; } } I is now impo an o no ice ha his al e na i e leads o he use o Ma ix[0] as he s a ing add ess o he da a s o ed in he ma ix. QUESTIONS • In o de o mul iply A×B ma ix A can be deli e ed o all p ocesses whils ma ix B se can be dis ibu ed in columns. Think o a di e en op ion. • Would i be possible o a ail o he powe o Ca esian opology o acili a e he esolu ion o his exe cise? • The need o b oadcas one o he ma ices slows p og am execu ion. Think o a di e en solu ion o a oid deli e ing so much in o ma ion. T y o guess wha he pe o mance o his new op ion would be compa ed wi h he cu en p og am. LABORATORY GUIDE Ac i i y 2: Pe o mance assessmen 9 Ac i i y 2: Pe o mance Assessmen OBJECTIVES  To measu e sys em’s pe o mance in a ious ci cums ances.  To lea n how o es ima e sys em’s powe and how o exploi i . A comp omise be ween lea ning e o and code op imiza ion mus be ob ained. THEORETICAL CONCEPTS In his chap e some common pe o mance ela ed concep s a e p esen ed: • Deg ee o pa allelism (DOP): Numbe o p ocesso s used o un a p og am in a p ecise momen on ime. The cu e, DOP = P( ), ep esen ing he deg ee o pa allelism as a unc ion o ime is called pa allelism p o ile o he p og am. I doesn’ need o ma ch he numbe o p ocesso s a ailable (n). Fo he ollowing de ini ions we will assume ha he e a e mo e p ocesso s han necessa y o each he maximum deg ee o pa allelism admi ed by a p og am: máx{P( )} = m < n. • To al amoun o wo k: Being ∆ he compu a ion capaci y o a single p ocesso , gi en ei he in MIPS o MFLOPS, and assuming all p ocesso s o be equal, i is possible o measu e he amoun o wo k ca ied ou be ween ime ins an A and B om he a ea unde he pa allelism p o ile as: ∫⋅⋅ ∆ = B A d PW )( . Usually he pa allelism p o ile is a disc e e g aph ( igu e 3), so he o al amoun o wo k can be compu ed as: ∑ = ⋅⋅∆= m ii iW 1 . Whe e i is he ime span when he deg ee o pa allelism is i, being m he maximum deg ee o pa allelism all o e he p og am’s execu ion ime. Acco ding o his, he sum o he di e en ime in e als is equal o he p og am’s execu ion ime: AB m ii −= ∑ =1 . • A e age pa allelism: Is he a i hme ic mean o he deg ee o pa allelism along ime: Ac i i y 3: In oduc ion o Hyb id P og amming LABORATORY GUIDE 16 o (i=0;i<n;i++){ Ope a ions o be pe o med } } The “n” ope a ions o be pe o med will be sca e ed among he N h eads. Tha will hope ully esul in a educ ion o execu ion ime in case o mul ico e/mul i h eaded p ocesso s. This is a sha ed memo y en i onmen bu , whe e a e he sha ed a iables? Va iables decla ed ou side he pa allel egion a e sha ed. Va iables decla ed inside he pa allel egion a e p i a e o each h ead. S ill i is possible o u n a sha ed a iable in o a p i a e one: #p agma omp pa allel num_ h eads (N) p i a e (j) { #p agma omp o o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } In his case, each h ead will ha e i s own copy o “j” e en hough i was decla ed ou side he egion bu , wha would be j’s alue on each h ead? In he p e ious piece o code “j” is no ini ialized ega dless he alue i migh ha e be o e he egion. I we wan use i s p e ious alue o ini ialize each h ead’s copy: #p agma omp pa allel num_ h eads (N) i s p i a e (j) { #p agma omp o o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } Likewise, we may need he mas e h ead o be awa e o he changes su e ed by “j” inside he egion once i inishes. We can o ce he alue o “j” o be he las one aken inside he egion: #p agma omp pa allel num_ h eads (N) i s p i a e (j) las p i a e (j) { #p agma omp o o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } To end up his b ie in oduc ion, we will ha e a look a an addi ional capabili y o OpenMP. I won’ be ha d o unde s and since he e is an equi alen one in MPI we ha e al eady used. This is he educ ion ope a ion. I applies o a si ua ion whe e a sha ed a iable is being modi ied in o di e en alues by di e en h eads. Some imes he inal alue o his a iable LABORATORY GUIDE Ac i i y 3: In oduc ion o Hyb id P og amming 17 has o be ob ained om a combina ion o he alues gene a ed by he di e en h eads. Le ’s ha e a look a he example: #p agma omp pa allel num_ h eads (N) { #p agma omp o educ ion(+:sum) o (i=0;i<n;i++){ sum=sum+(a[i]); } } I is ob ious ha we in end o ob ain a inal alue o “sum” which should be he esul o he “n” sums pe o med on i . The educ ion clause will ake he las alue gene a ed by each h ead and hen pe o m a inal sum on all o hem. To make his possible, a p i a e copy o he sha ed a iable is gene a ed on each h ead. PRACTICAL EXERCISE Take again he ma ix mul iply p og am and conduc he ollowing expe imen s: 1. Pe o m he mul iplica ion on 5000x5000 ma ices. Launch wo p ocesses. 2. Do he same wi h as many p ocesses as p ocesso co es a ailable. 3. Do i again wi h one mo e p ocess han co es. 4. Now adap you p og am o he hyb id p og aming pa adigm launching wo p ocesses and spli ing he wo king one in as many h eads as co es minus one. 5. Run he same hyb id p og am wi h as many h eads as co es. Compa e he ime spen by he di e en expe imen s and answe he ollowing QUESTIONS • Which p og amming pa adigm p o ides de highes pe o mance? • Is i mo e op imal o un only one p ocess/ h ead on each co e o i u ns ou ha p ocess 0 mus sha e co e wi h ano he p ocess/ h ead? • A e hese esul s wha we could expec ? Why? Ac i i y 4: Hyb id P og amming LABORATORY GUIDE 18 Ac i i y 4: Hyb id P og amming OBJECTIVES  Unde s and some wo k scheduling op ions in o de o op imize execu ion ime. THEORETICAL CONCEPTS Synch oniza ion. The de aul synch oniza ion p ocedu e in oduces a ba ie a he end o he pa allel egion so execu ion does no con inue un il all h eads each ha poin . This is a sensible hing o do bu , in ce ain cases, i may be use ul o a oid ha cons ain . This can be done by means o he “nowai ” clause. #p agma omp pa allel num_ h eads (N) { #p agma omp o nowai o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } In his pa icula example i doesn’ make any di e ence bu , in case we had ano he pa allel loop igh a e , i would sa e ime i some h eads could en e i as soon as possible. Scheduling. So a we ha e assumed ha he amoun o wo k o be done is deli e ed o he di e en h eads in a ai manne . Tha ’s igh bu , e en in his case he e could be di e en possibili ies ha esul in signi ican pe o mance a ia ions. The de aul scheduling policy di ides he numbe o i e a ions by he numbe o h eads hus gi ing each h ead he same amoun o wo k i possible. This wo k is assigned p io execu ion and no changes a e made a un ime. I is possible o speci y di e en wo k “chunks”. In his case each pa icula implemen a ion decides how o alloca e chunks on h eads. #p agma omp pa allel num_ h eads (N) { #p agma omp o schedule(s a ic,10) o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } LABORATORY GUIDE Ac i i y 4: Hyb id P og amming 19 In his example chunks o 10 i e a ions a e deli e ed. The las chunks a e made smalle when necessa y. S a ic policies do no allow o dynamically assigning pieces o wo k o h eads as hey inish hei p e iously assigned one. This esul s in a loss o e iciency ha should be a oided. Dynamic policies can be applied o do so. #p agma omp pa allel num_ h eads (N) { #p agma omp o schedule(dynamic,10) o (i=0;i<n;i++){ Ope a ions o be pe o med on a iable j } } In his example, h eads ge new chunks as soon as hey inish hei cu en calcula ion. PRACTICAL EXERCISE We will con inue he expe imen s done on he p e ious exe cise. We al eady ha e he esul s ob ained om he de aul s a ic scheduling. Now we will add hese new ones: • T y again he s a ic scheduling bu speci ying a chunk size o 10. • Then y chunk size 100. • Now shi o dynamic scheduling wi h chunk size 10. • T y again dynamic wi h chunk size 100. Compa e all esul s o see which he bes policy is and y o explain why. Wo k wi h he numbe o h eads ha p o ed o be he bes op ion in he p e ious exe cise. REFERENCES: OPENMP APPLICATION PROGRAM INTERFACE. A ailable a : h p://www.openmp.o g/mp-documen s/spec30.pd Ac i i y 5: MPI s OpenMP LABORATORY GUIDE 20 Ac i i y 5: MPI s OpenMP OBJECTIVES  In hyb id p og amming many di e en numbe o h eads and p ocesses may be launched. We will y o ind ou which is he bes combina ion.  Message passing and sha ed memo y in ol e di e en p og amming echniques and a dis inc use o ha dwa e esou ces. We need o know which one is mo e e icien and hen mo e con enien . THEORETICAL CONCEPTS No addi ional heo e ical discussion will be in oduced o his exe cise. PRACTICAL EXERCISE We will launch a ba e y o es mean o ul ill he i s o he objec i es al eady s a ed: • Repea he ma ix mul iplica ion on wo 5000x5000 ma ices wi h wo MPI p ocesses in he local machine. • Launch as many MPI p ocesses as co es a e a ailable. • Launch as many MPI p ocesses as co es a e a ailable plus one. • Back o wo MPI p ocesses spli he wo king one ( ank 1) in o as many h eads as co es a ailable minus one. • Spli ank 1 in o as many h eads as co es a e a ailable so one o i s h eads will sha e a co e wi h ank 0 p ocess. Compa e all esul s o see which he bes policy is and y o explain why. Now we will add ess he second objec i e. Use he 5000 x 5000 case again: • Launch as many p ocesses as p ocesso s a ailable plus one and spli he wo king p ocesses in o as many h eads as co es a ailable. • Launch as many p ocesses as co es a ailable plus one (no sha ed memo y his ime). Compa e he esul s and y o explain hem. See e e ences o ind answe s. REFERENCES: Compa ing he OpenMP, MPI, and Hyb id P og amming Pa adigms on an SMP Clus e Gab iele Jos and Haoqiang Jin and Die e An Mey and Fe ha F. Ha ay NAS Technical Repo NAS-03-019, No embe 2003. LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e . 21 Ac i i y 6: Submi ing jobs o a clus e OBJETIVES  Ge o know how Jobs a e submi ed o a compu a ion clus e .  Unde s and he di e ences be ween a local wo king en i onmen and a clus e a chi ec u e THEORETICAL CONCEPTS The Jobs we a e abou o submi o he clus e a e no di e en om hose we ha e been wo king wi h so a . They will be MPI p og ams mainly de i ed om he ma ix mul iply applica ion we a e using as a benchma k. We will wo k in Windows 8.1 using he use oles p e iously gene a ed wi hin he ARAVAN wo kg oup and also wi hin he HPC (High Pe o mance Compu ing) clus e . The use will be allowed o launch jobs o he clus e . F om now we a e going o use he Mic oso MPI implemen a ion: MS-MPI. The ool used o submi hese Jobs is he “Job Manage ” and i is pa o he clien ools ins alled by he HPC PACK 2012 R2. Be o e we can send jobs o execu ion he e a e a ew issues we ha e o deal wi h: 1. We won’ ha e a GUI. Ini ializa ion in o ma ion will be pa sed o he applica ions om he command line. O he in o ma ion needed a un ime has o be p o ided wi hin a ile. The e o e i will be necessa y o adap ou p og ams o hese si ua ions in ce ain cases. Conce ning he ma ix mul iply p og am we ha e de eloped, ma ix size will be in oduced as an ini ializa ion pa ame e om command line. A code line simlia o: “size = a oi (a g [1]);” will p o ide he nume ical alue o his pa ame e so i can be used wi hin he p og am. 2. P og am’s ou pu will be edi ec ed o a ex ile we will ha e o open once he p og am has inalized o see he esul s. 3. The job manage will conside he p og am’s execu ion unsuccess ul unless i e u ns a ze o code. We can w i e “exi (0)” a he end o he p og am o do so. I is de ined wi hin <s dlib.h>. Local job gene a ion wi h Job Mange . A job is in eg a ed by a numbe o asks. Tasks a e use applica ions mean o be execu ed by he sys em. We mean o launch jobs comp ising one single ask: ou MPI applica ion. In his case we can use he op ion “Single Task Job” o make he p ocess simple. Ac i i y 6: Submi ing Jobs o he clus e . LABORATORY GUIDE 22 Figu e 6.1. Single ask job con igu a ion. We ha e o selec he wo king di ec o y. In his case we in oduce he olde whe e he inpu and ou pu ex iles a e o be placed. We also in oduce he names o hese iles. I no inpu da a is equi ed he “S anda d inpu ” ield may be le blank. On he command line we desc ibe he ask o be pe o med, a pa allel MPI applica ion in his case: “mpiexec –n 4 c: mpiapps MPIapp1.exe”. I doesn’ need o be loca ed in he wo king di ec o y. The “-n 4” pa ame e ells he sys em o launch 4 p ocesses. Pa ame ic sweep jobs. In many eal si ua ions, asks a e no pe o med indi idually bu a he in a combined manne so esul s can be analyzed and compa ed. As a ma e o ac , we usually launch many execu ions o ou ma ix mul iply p og am o see how di e en con igu a ions and sizes a ec execu ion ime. I is possible o launch a job o each case bu i would be mo e e icien o launch hem all oge he . This is wha he “Pa ame ic sweep job” op ion makes possible. LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e . 23 Figu e 6.2. Pa ame ic sweep job con igu a ion. In his example we ha e se he pa ame e o a y om 1 o 5 inc emen ing one by one. As a esul , 5 asks will be conduc ed, one o each o i s alues. The as e isk used o place he pa ame e in he command line is also placed wi hin he names o he ex iles so each ask is linked o i s own ou pu ile. In his example we ha e used he pa ame e o modi y he command line a gumen pa sed o he p og am bu i can connec ed wi h any o he aspec o he in o ma ion p o ided in he command line. Fo ins ance, we could a y he numbe o p ocesses o be launched ins ead: “mpiexec –n * c: u a mul ima iz 5000”. We could p o ide mo e han one as e isk in he same command line bu i is e y unlikely ha he same alues make sense in di e en posi ions. Ne ing pa ame e s wi hin he same ask is no pe mi ed. Job gene a ion wi h Job Mange o he clus e . Gene a ing jobs o he local node o o he clus e is concep ually he same, since he o me is jus a sec ion o he la e . Ne e heless is impo an o ema k in his sec ion some se ings o be made: • Folde and sub olde sha ing. • Wo king di ec o y con igu a ion. • Node selec ion. Ac i i y 6: Submi ing Jobs o he clus e . LABORATORY GUIDE 24 Fo he applica ions o be execu ed by emo e nodes, he wo king di ec o y mus be sha ed. We can use Windows Explo e o edi he p ope ies o he olde con aining he wo king di ec o y and hen sha e i . Figu e 6.3. Sha ing he wo king di ec o y. Use s mean o execu e he applica ion mus ha e he app op ia e igh s. I he job is o be execu ed by o he nodes, i s pa h mus be known unde a common o ma . UNC (h ps://msdn.mic oso .com/en-us/lib a y/gg465305.aspx) is he one accep ed o his pu pose. I is use o decla e he pa h o he wo king di ec o y. The es o pa hs: inpu and ou pu iles and he applica ion i sel a e e e ed o he wo king di ec o y as a ela i e pa h. Figu e 6.4 shows how o make hese se ings. LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e . 25 Figu e 6.4. Con igu a iopn o he sha ed wo king di ec o y. In his pa icula case he applica ion’s whole pa h would be: TE-C-24 c: clus e p og amas Commandexample7 x64 Release Commandexample764.exe, whe e “150” is a command line a gumen o he applica ion. When con igu ing a new job, he “Resou ce Selec ion” op ion will display he a ailable nodes on he clus e so we can selec he desi ed ones. Ac i i y 9: Job scheduling III. LABORATORY GUIDE 32 Ac i i y 9: Job scheduling III OBJETIVES  Unde s anding p eemp ion.  Checking he in luence o p eemp ion on sys em pe o mance. THEORETICAL CONCEPTS P eemp ion allows highe p io i y jobs o in e up lowe p io i y ones. As shown be o e, his can be done in di e en ways. Since ou goal emains sys em pe o mance, highe p io i y should be gi en so he o e all execu ion ime is minimized. PRACTICAL EXERCISE G oup he asks launched in p e ious scheduling ac i i ies in wo jobs. On one job he asks comp ising less p ocesses will be placed and his job will be gi en he highes p io i y. The o he job, wi h he lowes p io i y will en ail he es o he asks. Unde bo h queued and balanced scheduling policies, epea he usual expe imen s ying he di e en p eemp ion op ions a ailable. Build up again he ables and compa e esul s. Decide wha p eemp ion policy is he mos ad isable o his ype o wo kload. Compa e he esul s ob ained unde queued scheduling policy wi h and wi hou he clicks on he “Adjus esou ces au oma ically” op ions. Compa e he esul s ob ained unde balanced scheduling policy using he di e en biasing op ions a ailable. LABORATORY GUIDE Ac i i y 10: Pe o mance compe i ion. 33 Ac i i y 10: Pe o mance compe i ion. OBJETIVES  Making he bes scheduling decisions. THEORETICAL CONCEPTS No heo e ical concep s a e in oduced in his ac i i y. PRACTICAL EXERCISE Fo a gi en ma ix mul iplica ion applica ion ( he same o all pa icipan s), each one will make wha a e expec ed o be he bes scheduling decisions. This will include using jobs, asks o bo h. Once hey a e made, he usual expe imen s will be conduc ed and he o e all execu ion imes compa ed in o de o ind ou wha we e ac ually he bes scheduling op ions. In you epo include you decisions, you esul s and compa e hem wi h he bes pe o me . Explain why you hink you decisions we e no he bes . I you a e he bes pe o me , cong a ula ions, you will sa e some wo k. Appendix A: Ins alling DeinoMPI LABORATORY GUIDE 34 Appendix A: Ins alling DeinoMPI DeinoMPI in an implemen a ion o he s anda d MPI-2 o Mic oso Windows de i ed om A gonne Nacional Labo a o y’s MPICH2. Sys em equi emen s: • Windows 2000/XP/Se e 2003/Windows 7 • .NET F amewo k 2.0 Ins alla ion DeinoMPI has o be downloaded and hen ins alled in all compu e s in he clus e . The ins alla ion p ocess is he same in all nodes. I equi es adminis a o p i ileges o ins alla ion bu all use s can execu e i a e wa ds. Once i is ins alled olde bin has o be added o he pa h. No e: make su e Deino’s e sion ma ches he ope a ing sys ems equi emen s (32 o 64 bi s). Con igu a ion Once he so wa e has been ins alled, each use will need o c ea e a “C eden ial S o e”. I is used o launch ou ines in a secu e manne . Mpiexec will no execu e any o hem wi hou his “C eden ial S o e”. The g aphic en i onmen will show he use his op ion in he i s execu ion. Launching Jobs Once again, bo h he g aphic en i onmen and he command line a e alid. G aphic En i onmen This ool can be used o launch MPI p ocesses, manage he “C eden ial S o e”, sea ch o compu e s wi hin he local ne wo k ha ha e MPI ins alled, e i y mpiexec en ies o diagnose common p oblems, and go o he DeinoMPI web si e o look o help and documen a ion. Mpiexec ab I is he main page and is used o launch and manage MPI p ocesses. LABORATORY GUIDE Appendix A: Ins alling DeinoMPI 35 Figu e A1. Mpiexec ab. These a e he main elemen s o his ab: • Applica ion: o The MPI applica ion’s pa h is in oduced he e. The same pa h will be aken by de aul in all nodes wi hin he clus e so i is ecommendable o copy he .exe ile in he same olde in all o hem. o I a ne wo k olde is speci ied, i is necessa y o ha e su icien p i ileges in he se e . o The “applica ion” bu on can be used o loca e he .exe ile. • Execu e: he p og am selec ed in he applica ion dialog is launched when his bu on is p essed.. • B eak: abo s p og am execu ion. • Numbe o p ocesses: Se s he numbe o p ocesses o be launched. • C eden ial S o e Accoun : Se s he ac i e use o he C eden ial S o e. • Check box “mo e op ions”: I expands/con ac s he op ions a ea. • Hos s: In oduce he e he lis o hos s whe e you wan he p ocesses o un. Hos names a e sepa a ed by blanks. To execu e he p og am in he local machine only, keep he de aul op ion “localonly” ac i e o w i e down i s name on his lis Appendix A: Ins alling DeinoMPI LABORATORY GUIDE 36 C eden ial S o e Tab. This ab is used o manage use ’s c eden ial s o e. I no c eden ial s o e has been c ea ed so a , selec “enable c ea e s o e op ions” check box o make emaining op ions a ailable. They a e hidden by de aul since hey a e only used he i s ime Deino is ini ia ed. Figu e A2. C eden ial S o e ab including all op ions. In o de o c ea e a c eden ial s o e, he “enable c ea e s o e op ions” check box mus be selec ed. Th ee possibili ies a ise: • “Passwo d”: o I his op ion is selec ed, he c eden ial s o e will be p o ec ed om access by a passwo d. I is he mos secu e op ion bu o ces he use o in oduce he passwo d any ime a job has o be launched. o I “No passwo d” is selec ed, he use o MPI is easie bu mo e ulne able. Wi hou a passwo d any p og am launched by he use can access he c eden ial s o e which is no eally a p oblem p o ided no malicious so wa e is being used. o E en wi h his “No passwo d” op ion ac i e, he c eden ial s o e is no a ailable o o he use s i he enc yp ion op ion is selec ed. • “Enc yp ion”: LABORATORY GUIDE Appendix A: Ins alling DeinoMPI 37 o “Windows P o ec Da a API” allows enc yp ion o he c eden ial s o e using he enc yp ion scheme used by Windows o he cu en use . This ensu es he c eden ial s o e will only be a ailable when he use is alida ed. o I a passwo d is selec ed he “symme ic key” enc yp ion o ma can be chosen. This enc yp ion is no speci ic o he use so o he use knowing he passwo d could access he s o e. o The “no enc yp ion” op ion is no ecommended since i s o es he c eden ial s o e in a plain ex ile accessible o all use s. • “Loca ion”: o Take he “Remo able media” op ion o sa e he s o e in an ex e nal de ice such as a memo y s ick. In his case, jobs can only be launched when he de ice is a ached o he compu e . This can be he sa es op ion since he use can decide when he c eden ial s o e is p esen . Combined wi h he use o a passwo d and i s enc yp ion i can be p o ec ed e en agains loss o obbe y. o The “Regis y” op ion mo es he “C eden ial S o e” o he Windows egis y. o Finally, i can be s o ed in he “Ha d d i e” which u ns ou o be he mos common decision. Clus e ab In his ab, he compu e s in he clus e a e displayed and he DeinoMPI e sion ins alled in each o hem. Figu e A3. Clus e ab – Big icons iew. Appendix A: Ins alling DeinoMPI LABORATORY GUIDE 38 Mo e hos s can be added w i ing down hei name o can be ound au oma ically wi hin he selec ed domain. Deino MPI manual. A ailable a : h p://mpi.deino.ne /manual.h m LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010 39 Appendix B: P ojec Con igu a ion in Visual S udio 2010 In his sec ion we will desc ibe he same con igu a ion p ocess bu o he 2010 e sion o Mic oso Visual S udio. Con igu a ion in mo e ecen e sions o Visual S udio is analogous. • Gene a e a new p ojec and solu ion. They may ha e bo h he same name: • Se i as emp y p ojec : Appendix B: P ojec Con igu a ion in Visual S udio 2010 LABORATORY GUIDE 40 • Once c ea ed bo h he P ojec and solu ion, add a code ile as new i em: LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010 41 • Now, and ne e be o e, he P ojec se ings a e en e ed (“P ope ies”): 1. In he C/C++ sec ion we mus en e he ou e o he olde whe e he heade MPI iles a e loca ed (“Addi ional Include Di ec o ies”). By de aul he A chi os de P og ama (x86) DeinoMPI include is assumed: 2. In he Linke sec ion we mus en e he ou e o he olde whe e he MPI lib a ies a e loca ed (“Addi ional Lib a y Di ec o ies”). By de aul A chi os de P og ama (x86) DeinoMPI lib is assumed: Appendix C: Con igu a ion o MS - MPI LABORATORY GUIDE 48 3. Se also he new lib a y ile. 4. When all hese pa s ha e been con igu ed he solu ion can be buil as usual. In o de o execu e he p og am, he .exe ile and MPI’s launche mus be in he same olde o ei he he pa h con igu ed acco dingly. The launche is mpiexec.exe and is placed in P og am Files > Mic oso MPI > bin. W i e down mpiexec –n np p og am.exe, whe e np is he numbe o p ocesses o be launched. LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010 49