UNIVERSITY OF BURGOS
A ea o Elec onic Technology
Pa allel & Hyb id P og amming.
José Ma ía Cáma a Neb eda, Césa Rep esa Pé ez, Ped o Luis Sánchez O ega
Pa allel & Hyb id P og amming
. 2015
A ea o Elec onic Technology
Elec
omechanical Enginee ing Depa men
Uni e si y o Bu gos
In oduc ion ............................................................................................................................................. 5
Ac i i y 1: MPI Ma ix Mul iplica ion ...................................................................................................... 7
OBJETIVES ............................................................................................................................................ 7
THEORETICAL CONCEPTS..................................................................................................................... 7
PRACTICAL EXERCISE ........................................................................................................................... 7
QUESTIONS .......................................................................................................................................... 8
Ac i i y 2: Pe o mance Assessmen ....................................................................................................... 9
OBJECTIVES .......................................................................................................................................... 9
THEORETICAL CONCEPTS..................................................................................................................... 9
PRACTICAL EXERCISE ......................................................................................................................... 12
QUESTIONS ........................................................................................................................................ 13
Ac i i y 3: In oduc ion o Hyb id P og amming ................................................................................... 14
OBJETIVES .......................................................................................................................................... 14
THEORETICAL CONCEPTS................................................................................................................... 14
PRACTICAL EXERCISE ......................................................................................................................... 17
QUESTIONS ........................................................................................................................................ 17
Ac i i y 4: Hyb id P og amming ............................................................................................................ 18
OBJECTIVES ........................................................................................................................................ 18
THEORETICAL CONCEPTS................................................................................................................... 18
PRACTICAL EXERCISE ......................................................................................................................... 19
Ac i i y 5: MPI s OpenMP .................................................................................................................... 20
OBJECTIVES ........................................................................................................................................ 20
THEORETICAL CONCEPTS................................................................................................................... 20
PRACTICAL EXERCISE ......................................................................................................................... 20
Ac i i y 6: Submi ing jobs o a clus e .................................................................................................. 21
OBJETIVES .......................................................................................................................................... 21
THEORETICAL CONCEPTS................................................................................................................... 21
PRACTICAL EXERCISE ......................................................................................................................... 26
Ac i i y 7: Job scheduling ...................................................................................................................... 27
OBJETIVES .......................................................................................................................................... 27
THEORETICAL CONCEPTS................................................................................................................... 27
PRACTICAL EXERCISE ......................................................................................................................... 30
Ac i i y 8: Job scheduling II ................................................................................................................... 31
OBJETIVES .......................................................................................................................................... 31
THEORETICAL CONCEPTS................................................................................................................... 31
PRACTICAL EXERCISE ......................................................................................................................... 31
Ac i i y 9: Job scheduling III .................................................................................................................. 32
OBJETIVES .......................................................................................................................................... 32
THEORETICAL CONCEPTS................................................................................................................... 32
PRACTICAL EXERCISE ......................................................................................................................... 32
Ac i i y 10: Pe o mance compe i ion. ................................................................................................. 33
OBJETIVES .......................................................................................................................................... 33
THEORETICAL CONCEPTS................................................................................................................... 33
PRACTICAL EXERCISE ......................................................................................................................... 33
Appendix A: Ins alling DeinoMPI ........................................................................................................... 34
Ins alla ion ......................................................................................................................................... 34
Con igu a ion ..................................................................................................................................... 34
Launching Jobs .................................................................................................................................. 34
G aphic En i onmen ........................................................................................................................ 34
Deino MPI manual. A ailable a : h p://mpi.deino.ne /manual.h m ................................................... 38
Appendix B: P ojec Con igu a ion in Visual S udio 2010 ..................................................................... 39
Appendix C: Con igu a ion o MS-MPI. ................................................................................................. 44
LABORATORY GUIDE In oduc ion
5
In oduc ion
Ou in e es will be ocused on pa allel p og aming o mul icompu e MIMD machines. Ou
applica ion p og ams will spli in o se e al p ocesses and each one will ha e he po en ial capabili y
o be execu ed on a di e en node o ou clus e .
The p ocesses c ea ed by he use will coope a e o achie e a common compu a ional objec i e. The
collabo a ion will be possible due o communica ion and synch oniza ion ools p o ided by he
p og amming en i onmen . Communica ion is implemen ed in he o m o message exchanging.
Mos o he scena ios p oposed admi a numbe o di e en pa allel solu ions. We should y o
come up wi h he mos ad an ageous in e ms o sys em pe o mance. To do so we mus ake in o
accoun :
• We will y o inc ease pe o mance (execu ion ime). To do so, we will y o squish he
applica ion’s po en ial locali y, ha is, i s capabili y o wo k wi h local da a a oiding he need
o much in o ma ion exchange be ween p ocesses.
• Ano he impo an poin is “scalabili y”. In a ha dwa e en i onmen , whe e he amoun o
a ailable esou ces is unknown a p og amming ime, he applica ion mus scale o make he
mos o he a ailable esou ces a any ime.
Pa allel p og aming is no an easy job. The heo y a ound he de elopmen o concu en and
pa allel so wa e is beyond he scope o his cou se bu , we will p o ide some hin s. Pa allel
p og amming, as well as sequen ial p og amming is a c ea i e ask; wha is abou o be exposed is
no hing mo e han a se ies o s eps we ecommend o ollow when acing a pa alleliza ion. Le ’s spli
up he p ocess in 4 s eps:
• F agmen a ion: his ini ial s ep is mean o ind po en ial pa allel s uc u es wi hin he
p oblem o be sol ed. As a i s app oach, we may y o decompose he job in as many small
pa allel asks as possible. Two c i e ia can be ollowed o ca y ou his decomposi ion:
o The unc ional way: seeks o possible di isions in he job o be ca ied ou by paying
a en ion o i s na u e.
o The da a way: pays a en ion o he na u e o he da a o be p ocessed ying o
decompose hem in o he smalles chunks.
• Communica ion: once iden i ied po en ial pa allel asks, communica ion needs be ween
hem mus be analyzed.
• Binding: gi en ha he cos o communica ions is high in e ms o global execu ion ime, he
o me ly iden i ied asks ha e o me ge pa ially in o de o balance compu a ion and
communica ion.
• Mapping: once he p og am’s s uc u e is se led, he ecen ly gene a ed p ocesses ha e o
be sp ead ac oss he compu e s a ailable. The s a egy o be adop ed di e s acco ding o

In oduc ion LABORATORY GUIDE
6
he agmen a ion way. As a ule o humb, he e should be a leas as many p ocesses as
compu e s a e a ailable in o de o p e en anyone being unused. I all compu e s a e equal,
i would be ecommendable o make as c ea e as many p ocesses as compu e s. I no , he
mos powe ul compu e s can hos a highe numbe o p ocesses. I is also possible o assign
p ocesses o nodes on he go, hus balancing p ocesso s’ load dynamically.
Depending on se e al aspec s, being he ype o compu e one o he mos ele an , pa allel
p og amming admi s di e en app oaches:
• Message passing: especially indica ed o dis ibu ed memo y compu e s, can be used on any
ha dwa e pla o m.
• Sha ed memo y: sui able only o sha ed memo y en i onmen s.
• Hyb id p og amming: a combina ion o he wo p e ious. I is mean o op imize
pe o mance when bo h sha ed and dis ibu ed memo y schemes a e p esen . This scena io
is e y common in ecen days. Mode n clus e s and MPPs a e in eg a ed by mul ico e
memo y sha ing nodes.
In his cou se we will assume ha he s uden is amilia enough wi h message passing p og amming.
Mo e p ecisely, he concep s gi en in he Bachelo Deg ee on Compu e Science abou MPI
p og amming a e conside ed as known. O he wise i is highly ecommended o he s uden o go
h ough he MPI P og amming Fundamen als cou se. A leas om ac i i y 0 o ac i i y 5.
LABORATORY GUIDE Ac i i y 1: MPI Ma ix Mul iplica ion
7
Ac i i y 1: MPI Ma ix Mul iplica ion
OBJETIVES
 Apply p e iously acqui ed knowledge o de elop a bi mo e complex p og am in ended o be
used as a benchma k o measu e sys em pe o mance.
THEORETICAL CONCEPTS
No new concep s will be in oduced in his chap e since i is mean o exploi hose al eady lea ned.
As ob ious, no all aspec s o MPI de elopmen en i onmen ha e been exposed and no ou
applica ion p og am is expec ed o ind he mos op imal solu ion bu qui e a good job is possible
hough.
Howe e , i may be help ul o in oduce some addi ional in o ma ion abou he unc ions we al eady
know. Func ion MPI_Rec e u ns a MPI_S a us ype pa ame e ha we ha en’ used so a . I is a
s uc u e in eg a ed by 3 elemen s: MPI_SOURCE, MPI_TAG & MPI_ERROR. The i s one con ains
he Rank o he sende p ocess. I he message was ecei ed unde MPI_ANY_SOURCE i can be
necessa y o ind ou who sen i la e on in he p og am. The second one e u ns he message’s ag.
I i was ecei ed unde MPI_ANY_TAG, i could be in e es ing o ge o know he ag’s alue as well.
The hi d one e u ns an e o code. We won’ deal wi h e o codes in his exe cise.
PRACTICAL EXERCISE
We will p og am a pa allel ma ix mul iply. I is he s uden ’s decision how o sca e calcula ions
among all he p ocesses. The size o he ma ices (squa e) mus be con igu able. Dynamic memo y
alloca ion is s ongly ecommended so no limi s o he size o he ma ices a e imposed.
P ocess 0 will ini ialize he ope and ma ices wi h any alue ( andom, loop, e c). Da a ype will be
loa . In a i s s age, mul iplica ion esul s will be displayed o check co ec ness. Once he p og am
has been alida ed, esul p in ing mus be emo ed o allow ma ix size o g ow. Execu ion ime has
o be displayed in all cases.
REMARK:
To combine double indexing wi h dynamic memo y alloca ion o ma ices, we mus use double
poin e s. Each poin e wi hin an a ay will gi e access o a ow in a ma ix:
// Decla e a double poi e o he ma ix
// This will le us e e o he elemen s in a [ ow][column] manne
loa **Ma ix;
// Ini ialize he double poi e o s o e poi e s o each and e e y ow in he ma ix.
Ma ix = ( loa **) malloc(ROWS*sizeo ( loa *));
// We ini ialize each poi e o he s a ing poi o each ow
o (i=0; i< ROWS; i++)
{
Ma iX[i] = ( loa *) malloc(COLUMNS*sizeo ( loa ));
}
// Now we can us [ ow][column] o ma o ou ma ix:
Ac i i y 1: MPI Ma ix Mul iplica ion LABORATORY GUIDE
8
o (in i=0; i<ROWS; i++)
{
o (in j=0; j<COLUMNS; j++)
{
Ma ix[i][j] = 0.0;
}
}
Howe e , his dynamic alloca ion p ocedu e does no gua an ee ha ows in he ma ix a e
con iguous in memo y. This can be necessa y o sending unc ions in ou p og am. We should send
da a ow by ow in ha scena io. I we wan o keep double indexing while adding con igui y, we will
ha e o p oceed as ollows:
// Decla e a double poi e o he ma ix
// This will le us e e o he elemen s in a [ ow][column] manne
loa **Ma ix;
// Ini ialize he double poi e o s o e poi e s o each and e e y ow in he ma ix.
Ma iX = ( loa **) malloc(ROWS*sizeo ( loa *));
// Decla e a new poin e o alloca e memo y space o he whole ma ix.
loa *M ;
// Ini ialize he poin e ha will gua an ee consecu i e loca ion o all ows
M = ( loa *) malloc(ROWS*COLUMNS*sizeo ( loa ));
// We ini ialize each poi e o he s a ing poi o each ow.
o (i=0; i< ROWS; i++)
{
Ma iX[i] = M + i* COLUMNS;
}
// Now we can us [ ow][column] o ma o ou ma ix:
o (in i=0; i<ROWS; i++)
{
o (in j=0; j<COLUMNS; j++)
{
Ma ix[i][j] = 0.0;
}
}
I is now impo an o no ice ha his al e na i e leads o he use o Ma ix[0] as he s a ing
add ess o he da a s o ed in he ma ix.
QUESTIONS
• In o de o mul iply A×B ma ix A can be deli e ed o all p ocesses whils ma ix B se can be
dis ibu ed in columns. Think o a di e en op ion.
• Would i be possible o a ail o he powe o Ca esian opology o acili a e he esolu ion o
his exe cise?
• The need o b oadcas one o he ma ices slows p og am execu ion. Think o a di e en
solu ion o a oid deli e ing so much in o ma ion. T y o guess wha he pe o mance o his
new op ion would be compa ed wi h he cu en p og am.
LABORATORY GUIDE Ac i i y 2: Pe o mance assessmen
9
Ac i i y 2: Pe o mance Assessmen
OBJECTIVES
 To measu e sys em’s pe o mance in a ious ci cums ances.
 To lea n how o es ima e sys em’s powe and how o exploi i . A comp omise be ween
lea ning e o and code op imiza ion mus be ob ained.
THEORETICAL CONCEPTS
In his chap e some common pe o mance ela ed concep s a e p esen ed:
• Deg ee o pa allelism (DOP): Numbe o p ocesso s used o un a p og am in a p ecise
momen on ime. The cu e, DOP = P( ), ep esen ing he deg ee o pa allelism as a unc ion
o ime is called pa allelism p o ile o he p og am. I doesn’ need o ma ch he numbe o
p ocesso s a ailable (n). Fo he ollowing de ini ions we will assume ha he e a e mo e
p ocesso s han necessa y o each he maximum deg ee o pa allelism admi ed by a
p og am: máx{P( )} = m < n.
• To al amoun o wo k: Being ∆ he compu a ion capaci y o a single p ocesso , gi en ei he
in MIPS o MFLOPS, and assuming all p ocesso s o be equal, i is possible o measu e he
amoun o wo k ca ied ou be ween ime ins an A and B om he a ea unde he
pa allelism p o ile as:
∫⋅⋅
∆
=
B
A
d PW )(
.
Usually he pa allelism p o ile is a disc e e g aph ( igu e 3), so he o al amoun o wo k can
be compu ed as:
∑
=
⋅⋅∆= m
ii
iW
1
.
Whe e i is he ime span when he deg ee o pa allelism is i, being m he maximum deg ee
o pa allelism all o e he p og am’s execu ion ime.
Acco ding o his, he sum o he di e en ime in e als is equal o he p og am’s execu ion
ime:
AB
m
ii
−=
∑
=1
.
• A e age pa allelism: Is he a i hme ic mean o he deg ee o pa allelism along ime:
Ac i i y 3: In oduc ion o Hyb id P og amming LABORATORY GUIDE
16
o (i=0;i<n;i++){
Ope a ions o be pe o med
}
}
The “n” ope a ions o be pe o med will be sca e ed among he N h eads. Tha will
hope ully esul in a educ ion o execu ion ime in case o mul ico e/mul i h eaded
p ocesso s.
This is a sha ed memo y en i onmen bu , whe e a e he sha ed a iables? Va iables decla ed
ou side he pa allel egion a e sha ed. Va iables decla ed inside he pa allel egion a e p i a e
o each h ead. S ill i is possible o u n a sha ed a iable in o a p i a e one:
#p agma omp pa allel num_ h eads (N) p i a e (j)
{
#p agma omp o
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
In his case, each h ead will ha e i s own copy o “j” e en hough i was decla ed ou side he
egion bu , wha would be j’s alue on each h ead? In he p e ious piece o code “j” is no
ini ialized ega dless he alue i migh ha e be o e he egion. I we wan use i s p e ious
alue o ini ialize each h ead’s copy:
#p agma omp pa allel num_ h eads (N) i s p i a e (j)
{
#p agma omp o
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
Likewise, we may need he mas e h ead o be awa e o he changes su e ed by “j” inside he
egion once i inishes. We can o ce he alue o “j” o be he las one aken inside he egion:
#p agma omp pa allel num_ h eads (N) i s p i a e (j) las p i a e (j)
{
#p agma omp o
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
To end up his b ie in oduc ion, we will ha e a look a an addi ional capabili y o OpenMP.
I won’ be ha d o unde s and since he e is an equi alen one in MPI we ha e al eady used.
This is he educ ion ope a ion. I applies o a si ua ion whe e a sha ed a iable is being
modi ied in o di e en alues by di e en h eads. Some imes he inal alue o his a iable

LABORATORY GUIDE Ac i i y 3: In oduc ion o Hyb id P og amming
17
has o be ob ained om a combina ion o he alues gene a ed by he di e en h eads. Le ’s
ha e a look a he example:
#p agma omp pa allel num_ h eads (N)
{
#p agma omp o educ ion(+:sum)
o (i=0;i<n;i++){
sum=sum+(a[i]);
}
}
I is ob ious ha we in end o ob ain a inal alue o “sum” which should be he esul o he
“n” sums pe o med on i . The educ ion clause will ake he las alue gene a ed by each
h ead and hen pe o m a inal sum on all o hem. To make his possible, a p i a e copy o
he sha ed a iable is gene a ed on each h ead.
PRACTICAL EXERCISE
Take again he ma ix mul iply p og am and conduc he ollowing expe imen s:
1. Pe o m he mul iplica ion on 5000x5000 ma ices. Launch wo p ocesses.
2. Do he same wi h as many p ocesses as p ocesso co es a ailable.
3. Do i again wi h one mo e p ocess han co es.
4. Now adap you p og am o he hyb id p og aming pa adigm launching wo p ocesses and
spli ing he wo king one in as many h eads as co es minus one.
5. Run he same hyb id p og am wi h as many h eads as co es.
Compa e he ime spen by he di e en expe imen s and answe he ollowing
QUESTIONS
• Which p og amming pa adigm p o ides de highes pe o mance?
• Is i mo e op imal o un only one p ocess/ h ead on each co e o i u ns ou ha p ocess 0
mus sha e co e wi h ano he p ocess/ h ead?
• A e hese esul s wha we could expec ? Why?
Ac i i y 4: Hyb id P og amming LABORATORY GUIDE
18
Ac i i y 4: Hyb id P og amming
OBJECTIVES
 Unde s and some wo k scheduling op ions in o de o op imize execu ion ime.
THEORETICAL CONCEPTS
Synch oniza ion.
The de aul synch oniza ion p ocedu e in oduces a ba ie a he end o he pa allel egion so
execu ion does no con inue un il all h eads each ha poin . This is a sensible hing o do
bu , in ce ain cases, i may be use ul o a oid ha cons ain . This can be done by means o
he “nowai ” clause.
#p agma omp pa allel num_ h eads (N)
{
#p agma omp o nowai
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
In his pa icula example i doesn’ make any di e ence bu , in case we had ano he pa allel
loop igh a e , i would sa e ime i some h eads could en e i as soon as possible.
Scheduling.
So a we ha e assumed ha he amoun o wo k o be done is deli e ed o he di e en
h eads in a ai manne . Tha ’s igh bu , e en in his case he e could be di e en
possibili ies ha esul in signi ican pe o mance a ia ions.
The de aul scheduling policy di ides he numbe o i e a ions by he numbe o h eads hus
gi ing each h ead he same amoun o wo k i possible. This wo k is assigned p io execu ion
and no changes a e made a un ime. I is possible o speci y di e en wo k “chunks”. In his
case each pa icula implemen a ion decides how o alloca e chunks on h eads.
#p agma omp pa allel num_ h eads (N)
{
#p agma omp o schedule(s a ic,10)
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
LABORATORY GUIDE Ac i i y 4: Hyb id P og amming
19
In his example chunks o 10 i e a ions a e deli e ed. The las chunks a e made smalle when
necessa y.
S a ic policies do no allow o dynamically assigning pieces o wo k o h eads as hey inish
hei p e iously assigned one. This esul s in a loss o e iciency ha should be a oided.
Dynamic policies can be applied o do so.
#p agma omp pa allel num_ h eads (N)
{
#p agma omp o schedule(dynamic,10)
o (i=0;i<n;i++){
Ope a ions o be pe o med on a iable j
}
}
In his example, h eads ge new chunks as soon as hey inish hei cu en calcula ion.
PRACTICAL EXERCISE
We will con inue he expe imen s done on he p e ious exe cise. We al eady ha e he esul s
ob ained om he de aul s a ic scheduling. Now we will add hese new ones:
• T y again he s a ic scheduling bu speci ying a chunk size o 10.
• Then y chunk size 100.
• Now shi o dynamic scheduling wi h chunk size 10.
• T y again dynamic wi h chunk size 100.
Compa e all esul s o see which he bes policy is and y o explain why.
Wo k wi h he numbe o h eads ha p o ed o be he bes op ion in he p e ious exe cise.
REFERENCES:
OPENMP APPLICATION PROGRAM INTERFACE. A ailable a :
h p://www.openmp.o g/mp-documen s/spec30.pd
Ac i i y 5: MPI s OpenMP LABORATORY GUIDE
20
Ac i i y 5: MPI s OpenMP
OBJECTIVES
 In hyb id p og amming many di e en numbe o h eads and p ocesses may be launched. We
will y o ind ou which is he bes combina ion.
 Message passing and sha ed memo y in ol e di e en p og amming echniques and a dis inc
use o ha dwa e esou ces. We need o know which one is mo e e icien and hen mo e
con enien .
THEORETICAL CONCEPTS
No addi ional heo e ical discussion will be in oduced o his exe cise.
PRACTICAL EXERCISE
We will launch a ba e y o es mean o ul ill he i s o he objec i es al eady s a ed:
• Repea he ma ix mul iplica ion on wo 5000x5000 ma ices wi h wo MPI p ocesses in he
local machine.
• Launch as many MPI p ocesses as co es a e a ailable.
• Launch as many MPI p ocesses as co es a e a ailable plus one.
• Back o wo MPI p ocesses spli he wo king one ( ank 1) in o as many h eads as co es
a ailable minus one.
• Spli ank 1 in o as many h eads as co es a e a ailable so one o i s h eads will sha e a co e
wi h ank 0 p ocess.
Compa e all esul s o see which he bes policy is and y o explain why.
Now we will add ess he second objec i e. Use he 5000 x 5000 case again:
• Launch as many p ocesses as p ocesso s a ailable plus one and spli he wo king p ocesses
in o as many h eads as co es a ailable.
• Launch as many p ocesses as co es a ailable plus one (no sha ed memo y his ime).
Compa e he esul s and y o explain hem. See e e ences o ind answe s.
REFERENCES:
Compa ing he OpenMP, MPI, and Hyb id P og amming Pa adigms on an SMP Clus e
Gab iele Jos and Haoqiang Jin and Die e An Mey and Fe ha F. Ha ay
NAS Technical Repo NAS-03-019, No embe 2003.
LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e .
21
Ac i i y 6: Submi ing jobs o a clus e
OBJETIVES
 Ge o know how Jobs a e submi ed o a compu a ion clus e .
 Unde s and he di e ences be ween a local wo king en i onmen and a clus e a chi ec u e
THEORETICAL CONCEPTS
The Jobs we a e abou o submi o he clus e a e no di e en om hose we ha e been wo king
wi h so a . They will be MPI p og ams mainly de i ed om he ma ix mul iply applica ion we a e
using as a benchma k. We will wo k in Windows 8.1 using he use oles p e iously gene a ed wi hin
he ARAVAN wo kg oup and also wi hin he HPC (High Pe o mance Compu ing) clus e . The use will
be allowed o launch jobs o he clus e . F om now we a e going o use he Mic oso MPI
implemen a ion: MS-MPI.
The ool used o submi hese Jobs is he “Job Manage ” and i is pa o he clien ools ins alled by
he HPC PACK 2012 R2. Be o e we can send jobs o execu ion he e a e a ew issues we ha e o deal
wi h:
1. We won’ ha e a GUI. Ini ializa ion in o ma ion will be pa sed o he applica ions om he
command line. O he in o ma ion needed a un ime has o be p o ided wi hin a ile.
The e o e i will be necessa y o adap ou p og ams o hese si ua ions in ce ain cases.
Conce ning he ma ix mul iply p og am we ha e de eloped, ma ix size will be in oduced as
an ini ializa ion pa ame e om command line. A code line simlia o: “size = a oi (a g [1]);”
will p o ide he nume ical alue o his pa ame e so i can be used wi hin he p og am.
2. P og am’s ou pu will be edi ec ed o a ex ile we will ha e o open once he p og am has
inalized o see he esul s.
3. The job manage will conside he p og am’s execu ion unsuccess ul unless i e u ns a ze o
code. We can w i e “exi (0)” a he end o he p og am o do so. I is de ined wi hin
<s dlib.h>.
Local job gene a ion wi h Job Mange .
A job is in eg a ed by a numbe o asks. Tasks a e use applica ions mean o be execu ed by he
sys em. We mean o launch jobs comp ising one single ask: ou MPI applica ion. In his case we can
use he op ion “Single Task Job” o make he p ocess simple.

Ac i i y 6: Submi ing Jobs o he clus e . LABORATORY GUIDE
22
Figu e 6.1. Single ask job con igu a ion.
We ha e o selec he wo king di ec o y. In his case we in oduce he olde whe e he inpu and
ou pu ex iles a e o be placed. We also in oduce he names o hese iles. I no inpu da a is
equi ed he “S anda d inpu ” ield may be le blank.
On he command line we desc ibe he ask o be pe o med, a pa allel MPI applica ion in his case:
“mpiexec –n 4 c: mpiapps MPIapp1.exe”. I doesn’ need o be loca ed in he wo king di ec o y. The
“-n 4” pa ame e ells he sys em o launch 4 p ocesses.
Pa ame ic sweep jobs.
In many eal si ua ions, asks a e no pe o med indi idually bu a he in a combined manne so
esul s can be analyzed and compa ed. As a ma e o ac , we usually launch many execu ions o ou
ma ix mul iply p og am o see how di e en con igu a ions and sizes a ec execu ion ime. I is
possible o launch a job o each case bu i would be mo e e icien o launch hem all oge he . This
is wha he “Pa ame ic sweep job” op ion makes possible.
LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e .
23
Figu e 6.2. Pa ame ic sweep job con igu a ion.
In his example we ha e se he pa ame e o a y om 1 o 5 inc emen ing one by one. As a esul ,
5 asks will be conduc ed, one o each o i s alues. The as e isk used o place he pa ame e in he
command line is also placed wi hin he names o he ex iles so each ask is linked o i s own ou pu
ile.
In his example we ha e used he pa ame e o modi y he command line a gumen pa sed o he
p og am bu i can connec ed wi h any o he aspec o he in o ma ion p o ided in he command
line. Fo ins ance, we could a y he numbe o p ocesses o be launched ins ead: “mpiexec –n *
c: u a mul ima iz 5000”. We could p o ide mo e han one as e isk in he same command line bu i
is e y unlikely ha he same alues make sense in di e en posi ions. Ne ing pa ame e s wi hin he
same ask is no pe mi ed.
Job gene a ion wi h Job Mange o he clus e .
Gene a ing jobs o he local node o o he clus e is concep ually he same, since he o me is jus
a sec ion o he la e . Ne e heless is impo an o ema k in his sec ion some se ings o be made:
• Folde and sub olde sha ing.
• Wo king di ec o y con igu a ion.
• Node selec ion.
Ac i i y 6: Submi ing Jobs o he clus e . LABORATORY GUIDE
24
Fo he applica ions o be execu ed by emo e nodes, he wo king di ec o y mus be sha ed. We can
use Windows Explo e o edi he p ope ies o he olde con aining he wo king di ec o y and hen
sha e i .
Figu e 6.3. Sha ing he wo king di ec o y.
Use s mean o execu e he applica ion mus ha e he app op ia e igh s.
I he job is o be execu ed by o he nodes, i s pa h mus be known unde a common o ma . UNC
(h ps://msdn.mic oso .com/en-us/lib a y/gg465305.aspx) is he one accep ed o his pu pose. I is
use o decla e he pa h o he wo king di ec o y. The es o pa hs: inpu and ou pu iles and he
applica ion i sel a e e e ed o he wo king di ec o y as a ela i e pa h. Figu e 6.4 shows how o
make hese se ings.
LABORATORY GUIDE Ac i i y 6: Submi ing Jobs o he clus e .
25
Figu e 6.4. Con igu a iopn o he sha ed wo king di ec o y.
In his pa icula case he applica ion’s whole pa h would be:
TE-C-24 c: clus e p og amas Commandexample7 x64 Release Commandexample764.exe, whe e
“150” is a command line a gumen o he applica ion.
When con igu ing a new job, he “Resou ce Selec ion” op ion will display he a ailable nodes on he
clus e so we can selec he desi ed ones.
Ac i i y 9: Job scheduling III. LABORATORY GUIDE
32
Ac i i y 9: Job scheduling III
OBJETIVES
 Unde s anding p eemp ion.
 Checking he in luence o p eemp ion on sys em pe o mance.
THEORETICAL CONCEPTS
P eemp ion allows highe p io i y jobs o in e up lowe p io i y ones. As shown be o e, his can be
done in di e en ways. Since ou goal emains sys em pe o mance, highe p io i y should be gi en
so he o e all execu ion ime is minimized.
PRACTICAL EXERCISE
G oup he asks launched in p e ious scheduling ac i i ies in wo jobs. On one job he asks
comp ising less p ocesses will be placed and his job will be gi en he highes p io i y. The o he job,
wi h he lowes p io i y will en ail he es o he asks.
Unde bo h queued and balanced scheduling policies, epea he usual expe imen s ying he
di e en p eemp ion op ions a ailable.
Build up again he ables and compa e esul s.
Decide wha p eemp ion policy is he mos ad isable o his ype o wo kload.
Compa e he esul s ob ained unde queued scheduling policy wi h and wi hou he clicks on he
“Adjus esou ces au oma ically” op ions.
Compa e he esul s ob ained unde balanced scheduling policy using he di e en biasing op ions
a ailable.

LABORATORY GUIDE Ac i i y 10: Pe o mance compe i ion.
33
Ac i i y 10: Pe o mance compe i ion.
OBJETIVES
 Making he bes scheduling decisions.
THEORETICAL CONCEPTS
No heo e ical concep s a e in oduced in his ac i i y.
PRACTICAL EXERCISE
Fo a gi en ma ix mul iplica ion applica ion ( he same o all pa icipan s), each one will make wha
a e expec ed o be he bes scheduling decisions. This will include using jobs, asks o bo h. Once
hey a e made, he usual expe imen s will be conduc ed and he o e all execu ion imes compa ed in
o de o ind ou wha we e ac ually he bes scheduling op ions.
In you epo include you decisions, you esul s and compa e hem wi h he bes pe o me .
Explain why you hink you decisions we e no he bes . I you a e he bes pe o me ,
cong a ula ions, you will sa e some wo k.
Appendix A: Ins alling DeinoMPI LABORATORY GUIDE
34
Appendix A: Ins alling DeinoMPI
DeinoMPI in an implemen a ion o he s anda d MPI-2 o Mic oso Windows de i ed om A gonne
Nacional Labo a o y’s MPICH2.
Sys em equi emen s:
• Windows 2000/XP/Se e 2003/Windows 7
• .NET F amewo k 2.0
Ins alla ion
DeinoMPI has o be downloaded and hen ins alled in all compu e s in he clus e . The ins alla ion
p ocess is he same in all nodes. I equi es adminis a o p i ileges o ins alla ion bu all use s can
execu e i a e wa ds.
Once i is ins alled olde bin has o be added o he pa h.
No e: make su e Deino’s e sion ma ches he ope a ing sys ems equi emen s (32 o 64 bi s).
Con igu a ion
Once he so wa e has been ins alled, each use will need o c ea e a “C eden ial S o e”. I is used o
launch ou ines in a secu e manne . Mpiexec will no execu e any o hem wi hou his “C eden ial
S o e”. The g aphic en i onmen will show he use his op ion in he i s execu ion.
Launching Jobs
Once again, bo h he g aphic en i onmen and he command line a e alid.
G aphic En i onmen
This ool can be used o launch MPI p ocesses, manage he “C eden ial S o e”, sea ch o compu e s
wi hin he local ne wo k ha ha e MPI ins alled, e i y mpiexec en ies o diagnose common
p oblems, and go o he DeinoMPI web si e o look o help and documen a ion.
Mpiexec ab
I is he main page and is used o launch and manage MPI p ocesses.
LABORATORY GUIDE Appendix A: Ins alling DeinoMPI
35
Figu e A1. Mpiexec ab.
These a e he main elemen s o his ab:
• Applica ion:
o The MPI applica ion’s pa h is in oduced he e. The same pa h will be aken by de aul in
all nodes wi hin he clus e so i is ecommendable o copy he .exe ile in he same
olde in all o hem.
o I a ne wo k olde is speci ied, i is necessa y o ha e su icien p i ileges in he se e .
o The “applica ion” bu on can be used o loca e he .exe ile.
• Execu e: he p og am selec ed in he applica ion dialog is launched when his bu on is
p essed..
• B eak: abo s p og am execu ion.
• Numbe o p ocesses: Se s he numbe o p ocesses o be launched.
• C eden ial S o e Accoun : Se s he ac i e use o he C eden ial S o e.
• Check box “mo e op ions”: I expands/con ac s he op ions a ea.
• Hos s: In oduce he e he lis o hos s whe e you wan he p ocesses o un. Hos names a e
sepa a ed by blanks. To execu e he p og am in he local machine only, keep he de aul
op ion “localonly” ac i e o w i e down i s name on his lis
Appendix A: Ins alling DeinoMPI LABORATORY GUIDE
36
C eden ial S o e Tab.
This ab is used o manage use ’s c eden ial s o e. I no c eden ial s o e has been c ea ed so a ,
selec “enable c ea e s o e op ions” check box o make emaining op ions a ailable. They a e hidden
by de aul since hey a e only used he i s ime Deino is ini ia ed.
Figu e A2. C eden ial S o e ab including all op ions.
In o de o c ea e a c eden ial s o e, he “enable c ea e s o e op ions” check box mus be selec ed.
Th ee possibili ies a ise:
• “Passwo d”:
o I his op ion is selec ed, he c eden ial s o e will be p o ec ed om access by a
passwo d. I is he mos secu e op ion bu o ces he use o in oduce he passwo d any
ime a job has o be launched.
o I “No passwo d” is selec ed, he use o MPI is easie bu mo e ulne able. Wi hou a
passwo d any p og am launched by he use can access he c eden ial s o e which is no
eally a p oblem p o ided no malicious so wa e is being used.
o E en wi h his “No passwo d” op ion ac i e, he c eden ial s o e is no a ailable o o he
use s i he enc yp ion op ion is selec ed.
• “Enc yp ion”:
LABORATORY GUIDE Appendix A: Ins alling DeinoMPI
37
o “Windows P o ec Da a API” allows enc yp ion o he c eden ial s o e using he
enc yp ion scheme used by Windows o he cu en use . This ensu es he c eden ial
s o e will only be a ailable when he use is alida ed.
o I a passwo d is selec ed he “symme ic key” enc yp ion o ma can be chosen. This
enc yp ion is no speci ic o he use so o he use knowing he passwo d could access
he s o e.
o The “no enc yp ion” op ion is no ecommended since i s o es he c eden ial s o e in a
plain ex ile accessible o all use s.
• “Loca ion”:
o Take he “Remo able media” op ion o sa e he s o e in an ex e nal de ice such as a
memo y s ick. In his case, jobs can only be launched when he de ice is a ached o he
compu e . This can be he sa es op ion since he use can decide when he c eden ial
s o e is p esen . Combined wi h he use o a passwo d and i s enc yp ion i can be
p o ec ed e en agains loss o obbe y.
o The “Regis y” op ion mo es he “C eden ial S o e” o he Windows egis y.
o Finally, i can be s o ed in he “Ha d d i e” which u ns ou o be he mos common
decision.
Clus e ab
In his ab, he compu e s in he clus e a e displayed and he DeinoMPI e sion ins alled in each o
hem.
Figu e A3. Clus e ab – Big icons iew.

Appendix A: Ins alling DeinoMPI LABORATORY GUIDE
38
Mo e hos s can be added w i ing down hei name o can be ound au oma ically wi hin he selec ed
domain.
Deino MPI manual. A ailable a : h p://mpi.deino.ne /manual.h m
LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010
39
Appendix B: P ojec Con igu a ion in
Visual S udio 2010
In his sec ion we will desc ibe he same con igu a ion p ocess bu o he 2010 e sion o Mic oso
Visual S udio. Con igu a ion in mo e ecen e sions o Visual S udio is analogous.
• Gene a e a new p ojec and solu ion. They may ha e bo h he same name:
• Se i as emp y p ojec :
Appendix B: P ojec Con igu a ion in Visual S udio 2010 LABORATORY GUIDE
40
• Once c ea ed bo h he P ojec and solu ion, add a code ile as new i em:
LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010
41
• Now, and ne e be o e, he P ojec se ings a e en e ed (“P ope ies”):
1. In he C/C++ sec ion we mus en e he ou e o he olde whe e he heade MPI iles
a e loca ed (“Addi ional Include Di ec o ies”). By de aul he A chi os de P og ama
(x86) DeinoMPI include is assumed:
2. In he Linke sec ion we mus en e he ou e o he olde whe e he MPI lib a ies a e
loca ed (“Addi ional Lib a y Di ec o ies”). By de aul A chi os de P og ama
(x86) DeinoMPI lib is assumed:
Appendix C: Con igu a ion o MS - MPI LABORATORY GUIDE
48
3. Se also he new lib a y ile.
4. When all hese pa s ha e been con igu ed he solu ion can be buil as usual. In o de
o execu e he p og am, he .exe ile and MPI’s launche mus be in he same olde
o ei he he pa h con igu ed acco dingly. The launche is mpiexec.exe and is placed
in P og am Files > Mic oso MPI > bin. W i e down mpiexec –n np p og am.exe,
whe e np is he numbe o p ocesses o be launched.

LABORATORY GUIDE Appendix B: P ojec Con igu a ion in Visual S udio 2010
49