Dynamic load balance o chemical sou ce e m e alua ion in
high- ideli y combus ion simula ions
Guillem Rami ez-Mi anda, Daniel Mi a∗, Edua do J. P´
e ez-S´
anchez, Anu ag Su apaneni,
Rica d Bo ell, Guillaume Houzeaux, Ma a Ga cia-Gasulla
Ba celona Supe compu ing Cen e (BSC), Plaza Eusebi G¨uell 1-3, 08034, Ba celona Spain
Abs ac
This pape p esen s a load balancing s a egy o eac ion a e e alua ion and chemis y
in eg a ion in eac ing low simula ions. The la ge dispa i y in scales du ing combus ion
in oduces s i ness in he nume ical in eg a ion o he PDEs and gene a es load imbal-
ance du ing he pa allel execu ion. The s a egy is based on he use o he DLB lib a y
o edis ibu e he compu ing esou ces a node le el, lending addi ional CPU-co es o
highe loaded MPI p ocesses. This app oach does no equi e explici da a ans e and
is ac i a ed au oma ically a un ime. Two chemis y desc ip ions, de ailed and educed,
a e e alua ed on wo di e en con igu a ions: lamina coun e low lame and a u bulen
swi l-s abilized lame. Fo single-node calcula ions, speedups o 2.3x and 7x a e ob ained
o he de ailed and educed chemis y, espec i ely. Resul s on mul i-node uns also show
ha DLB imp o es he pe o mance o he pu e-MPI code simila o single node uns. I
is shown DLB can ge pe o mance imp o emen s in bo h de ailed and educed chemis y
calcula ions.
Keywo ds: dynamic load balancing, combus ion, High-Pe o mance Compu ing,
compu a ional luid dynamics, DLB lib a y
1. In oduc ion
Regula ions applied o he powe and anspo a ion sec o s ha e p opelled esea ch
on op imiza ion o he mal engines and o he combus ion sys ems, o simul aneously e-
duce uel consump ion and pollu an emissions. The ex ensi e use o Compu a ional Fluid
Dynamics (CFD) o his pu pose, as a con en ional and indispensable ool, has made
∗Co esponding au ho
Email add ess: [email p o ec ed] (Daniel Mi a)
P ep in submi ed o Compu e &Fluids Oc obe 17, 2022
a Xi :2210.07364 1 [physics. lu-dyn] 13 Oc 2022
This is he AAM: The inal e sion o his pape can be ound a :
h ps://doi.o g/10.1016/j.comp luid.2022.105723.
(h ps://www.sciencedi ec .com/science/a icle/pii/S0045793022003164
Copy igh © 2024 Else ie B.V. CC BY-NC-ND 4.0
impe a i e he sea ch o e icien algo i hms ha can sol e he equa ions o chemically
eac ing lows a educed compu a ional cos s. The chemical sou ce e ms a e inse ed in
he anspo equa ions, oge he wi h he e alua ion o he con ec ion and di usion e ms,
ha cha ac e ize he eac ing low.
Mo eo e , he choice o he eac ion mechanism is a c i ical aspec when pe o ming
high- ideli y combus ion simula ions. The cos o he e alua ion o he chemical sou ce
e m depends on he size and s i ness o he eac ion mechanism, and can esul in he mos
demanding pa o he calcula ion. While he wo kload o e alua e he anspo e ms can
be pa allelized using s anda d domain decomposi ion s a egies [24], i is ha de o de ine
a compu a ionally balanced dis ibu ion o he chemis y wo kload and his speci ic ask
can end up aking 90% o he compu a ional ime [29, 35]. Such load imbalance can be
explained by he na u e o he combus ion p ocess, whe e he chemical eac ions occu in
speci ic egions o he domain and usually along hin laye s, esul ing, in consequence,
in high dispa i y o compu a ional load be ween p ocesso s, as no all he subdomains
equi e he e alua ion o he chemical eac ion a es. Mo eo e , he s i ness o he ODEs
(o dina y di e en ial equa ions) sys em, caused by he non-linea i y o chemis y and he
wide ange o ime scales o adicals (a ound 10−8s) and majo species (in he o de o
en o mic oseconds), inc eases he compu a ional cos o he chemis y in eg a ion and
accen ua es e en mo e he load imbalance.
Howe e , di e en om he anspo e ms o he low a iables, which depend on
he s a e a he icini y a each poin , chemical sou ce e ms only depend on he punc-
ual he mochemical s a e o he mix u e and, hence, i is suscep ible o a high deg ee o
pa alleliza ion.
Based on hese conside a ions, di e en s a egies ha e been used o achie e load bal-
ance in combus ion calcula ions. In he ea ly wo ks by The enin e al. [31], he e was a
ans e o poin s be ween neighbo ing p ocesso s, so ha he nodes equi ing mo e com-
pu ing ime han he a e age send a ew g id poin s o hei neighbo s. A simila s a egy
was p oposed by An onelly and D’Amb a (2011) [2], whe e a cell dis ibu ion based on a
dynamic load balancing ha p ese es con igui y o he compu a ional g id cells was used.
Ano he impo an aspec is he s i ness o sys em, which can be no iceably di e -
en be ween cells. Chemical mechanisms may ha e be ween 50 and 1000 species, which
include species ea u ing a wide ange o ime scales depending on he local condi ions.
Fo ins ance, he low empe a u e egion de e mines he au oigni ion wi h high deg ee o
s i ness, while he high empe a u e egion has low s i ness as species wi h la ge mass
ac ions can each equilib ium ela i ely as [2]. These obse a ions can be used o e-
duce he compu a ional cos . Muela e al. [25] p oposed a measu e o he s i ness o
sepa a e be ween explici and implici algo i hms. Koda asal e al. (2016) [24] p oposed
a “s i ness-based” algo i hm o load balancing chemical kine ics using in o ma ion om
2
p e ious ime-s eps. A simila s a egy has been used by Teckg¨
ul e al. (2021) [30], which
also includes a zonal e e ence cell mapping me hod o a oid he e alua ion o he ki-
ne ic a es in ambien egions wi h low eac i i y. In he con ex o High Pe o mance
Compu ing (HPC) wi h massi e use o CPUs, his s a egy can be la gely bene i ed by he
hyb id use o CPUs and GPUs p o ided ha explici algo i hms, which a e easily pa al-
lelized and consume less memo y, a e applied o GPUs [29]. Zi wes e al. (2018) [34]
p oposed a con e sion o eac ion mechanisms in o sou ce codes o es uc u e he da a
o he kine ic mechanisms o e icien compu a ion enabling compile op imiza ions. De-
spi e hese s a egies ha e esul ed in no iceable compu a ional cos educ ions, ensu ing
a load balance s a egy o gene al applica ions in p emixed and non-p emixed combus-
ion will be o high alue. All he p e ious s a egies a e based on a e-dis ibu ion o he
load a sys em le el, whe e he eac ion a e e alua ion and chemis y in eg a ion is e-
dis ibu ed acco ding o he cu en load and an es ima ion o he s i ness using Message
Passing In e ace (MPI) s anda d. Howe e , one o he undamen al issues associa ed o
hose s a egies is he e alua ion o he chemical s i ness, which is di icul o p edic and
can esul in load imbalance. Mo eo e , when applying hese me hods i is necessa y o
synch onize he di e en p ocesses and exchange da a h ough MPI communica ion.
This pape is de o ed o implemen and analyze new load balancing mechanisms o
chemis y in eg a ion. In pa icula , we p opose u ilizing he Dynamic Load Balancing
(DLB) lib a y 1[15, 16], which allows eusing CPU-co es associa ed wi h idle MPI p o-
cesses by o he p ocesses unning on he same node. I is a load balancing mechanism
based on ans e ing idle esou ces a he node le el a he han ans e ing wo kload
subse s h ough message passing. DLB ac s as an au oma ic un ime mechanism anspa -
en o he use and equi es minimum changes in he sou ce code ( wo lines in he p esen
s udy). In ac , DLB can be combined wi h he wo kload ans e ing s a egies men ioned
abo e. DLB has al eady been success ully applied o inc ease he load balance o he
assembly o he igh -hand side e ms in he Na ie -S okes equa ions [17], he pa icle
anspo [22], coupled codes [9] and has been also used in di e en a chi ec u es [18].
He e, i is ex ended o op imize he chemis y pa in eac ing low simula ions. The p o-
posed solu ion wi h DLB, does no need o add ex a da a mo emen , because e e y hing
is done h ough he sha ed memo y o he node. As i is a dynamic mechanism ha eac s
o he load imbalance, i does no need o p edic he s i ness no he compu a ion load
associa ed. And las bu no leas , i does no equi e a hea y implemen a ion e o in he
applica ion.
The emaining o he pape is o ganized as ollows. Sec ion 2 desc ibes he compu-
a ional amewo k ha is used o conduc ing he nume ical simula ions including he
1h ps://pm.bsc.es/dlb
3
modelling and nume ical desc ip ions used om he code Alya [32]. Sec ion 3 desc ibes
he compu a ional en i onmen in which hese simula ions a e conduc ed and Sec ion 4
de ails he assessmen o his dynamic load balance s a egy on ep esen a i e p oblems
in combus ion science. Finally, he conclusions and di ec ions o u u e wo k a e gi en in
Sec ion 5.
2. Modelling amewo k
2.1. Go e ning anspo equa ions
The simula ion o eac ing lows includes go e ning equa ions o chemical species
along wi h ene gy, momen um and con inui y. A low Mach numbe app oxima ion o he
Na ie -S okes is conside ed in his s udy o which he conse a ion o con inui y and
momen um ead:
∂ρ
∂ +∇·(ρu)=0,(1)
∂(ρu)
∂ +∇·(ρuu)=−∇p+∇·(µ∇u),(2)
whe e s anda d no a ion is used o all he quan i ies and ρ,u,pand µ ep esen he densi y,
eloci y ec o , p essu e and dynamic iscosi y.
Rega ding he e olu ion o he mul icomponen gas, i can be exp essed in e ms o he
anspo equa ions o he indi idual species Ykgi en by:
∂(ρYk)
∂ +∇·(ρuYk)=∇·(ρD∇Yk)+˙ωYkk=1,...,Ns.(3)
In his equa ion, Dis he di usion mass coe icien , o which a uni y Lewis assump ion
has been adop ed, while ˙ωYkdeno es he chemical sou ce e m o species Yk.Nsis he
numbe o species conside ed in he chemical mechanism. Finally, he o al en halpy h
equa ion, in which hea ing due o iscous o ces is neglec ed, eads as:
∂(ρh)
∂ +∇·(ρuh)=∇·(ρD∇h).(4)
2.2. Chemical in eg a ion
The chemical in eg a ion is one o he mos compu a ionally demanding pa s in he
in eg a ion o he go e ning equa ions due o he high non-linea i y o he A henius- ype
eac ion kine ics. I is, he e o e, clea ha he in eg a ion me hod o chemis y may play
4
an impo an ole in he o al ime o he simula ion, especially when de ailed chemis y
models a e conside ed.
In his wo k, o educe he s i ness o he in eg a ion o he species go e ning equa-
ions 3, a spli ing algo i hm is used o sepa a e he anspo om he chemis y [27]. The
solu ion o he chemis y p oblem is achie ed by he in eg a ion o he open sou ce Can-
e a [19] so wa e as an ex e nal lib a y in he mul iphysics code Alya [32]. A Fo an o
C++ w appe was c ea ed o in eg a e Can e a in o Fo an o Alya, so in e nal unc ions
om Can e a could be used in un ime. The eac ion a es a e ob ained om in-buil in-
e nal unc ions om Can e a, and he chemical in eg a ion is ob ained using he CVODE
algo i hm [11]. A lis ing o he in eg a ion loop o he code is gi en in Lis ing 1.
CVODE is a package w i en in C o sol e IVPs (Ini ial Value P oblems) de ined by
s i and non-s i ODEs in he o m:
˙y= ( ,y),(5)
wi h he ini ial condi ions gi en by y( = 0)=y0. In pa icula , he equa ions o
chemis y in eg a ion a e simila o hose o sys em 5 bu wi hou he dependence on ime:
dY
d = (T,Y),(6)
whe e Tis empe a u e, Y=(Y1,...,YN)Tis he ec o o mass ac ion o species. The
ini ial condi ions co espond o T( 0)=T0and Y( 0)=Y0.
CVODE is in u n based on he ODE sol e packages VODE and VODPK [7] and
sol es p e ious IVP by he applica ion o an implici empo al scheme based on ei he
Adams-Moul on o mula o backwa d-di e encing o mula (BDF) me hods, wi h he sub-
sequen esolu ion o he non-linea equa ion by New on’s me hod. Depending on he
o m o he Jacobian ma ix, dedica ed unc ions o dense o banded ma ices can be used
allowing, mo eo e , he p econdi ioning o he linea sys em.
The code used in Alya o chemical in eg a ion loop wi h no pa alleliza ion is ga he ed
in Lis ing 1.
1do ipoin=1,npoin
2i ( eac ion) call c ode_in eg a ion
3end do
4...
5call MPI_All educe (...)
Lis ing 1: Chemical in eg a ion loop in Alya code.
5

2.3. Compu a ional pla o m: Alya
All he compu a ional s a egies p esen ed in his pape ha e been implemen ed in
Alya, he high pe o mance compu a ional mul i-physics code de eloped a he Ba celona
Supe compu ing Cen e . Alya is de eloped using a modula a chi ec u e ha includes a
module coupling o ackle complex mul i-physics p oblems such as combus ion simula-
ions. Alya is w i en in Fo an and is designed o massi ely pa allel supe compu e s;
pa icula ly i is one o he wel e simula ion codes o he Uni ied Eu opean Applica ions
Benchma k Sui e (UEABS) [8], being egula ly es ed on he Eu opean Tie -0 supe com-
pu e s.
The pa alleliza ion implemen ed in Alya comp ises h ee le els: dis ibu ed memo y
o in e - and in a- node pa allelism, sha ed memo y o in a-node pa allelism, and SIMD
(Single Ins uc ion Mul iple Da a) and SIMT (Single Ins uc ion Mul iple Th ead) o
CPU ec o iza ion and GPU compu ing, espec i ely. The implemen a ion o such s a -
egy combines a ious p og amming models: MPI o in e -p ocess message passing and
synch oniza ion, di ec i es-based app oaches (OpenMP, OmpSs, and OpenACC) o loops
and ask-based pa allelism and GPU o loading, as well as CUDA o low-le el op imized
implemen a ion o speci ic ke nels.
The p ima y op ion o mesh pa i ioning is an in-house SFC-based pa i ione [4]. On-
line edis ibu ion is also pe o med o adjus he pa i ion o un- ime measu emen s. This
las op ion has been exploi ed o co-execu ion on he e ogeneous sys ems, whe e he pa i-
ion is adjus ed o a balanced execu ion using CPU and GPU de ices simul aneously [5].
Fo ask-based sha ed-memo y pa allelisms, a second-le el decomposi ion is pe o med
in which he size o he esul ing subse s is o he o de o 102elemen s. Finally, a da a e-
s uc u ing is pe o med o ec o iza ion: subse s o 8 o 32 elemen s a e packed oge he
o be execu ed in a SIMD model. The same s a egy is used o GPU compu ing being, in
his case, each pack, o he o de o 105elemen s, launched o he GPU whe e he SIMT
pa allel model is exploi ed.
3. Desc ip ion o he es cases
Two ep esen a i e combus ion p oblems a e used he e o e alua e he pe o mance o
DLB o ensu e a load balancing s a egy o he chemical in eg a ion.
The i s case co esponds o a lamina coun e low di usion lame a a mosphe ic
p essu e and 298K ai and uel empe a u e. The lame in his con igu a ion is cha ac e -
ized by a wide egion whe e uel and oxidize a e mixed and a eac ion laye o med a
he icini y o he s oichiome ic mix u e ac ion. Mo eo e , he lame is de e mined by
he le el o s ain, which is gi en by he eloci ies o he wo s eams and he dis ance
be ween he nozzles. A ep esen a ion o he lame is shown in Fig. 1. A wo-dimensional
6
domain wi h wo di e en mesh sizes was used o e alua e he load balancing s a egy
using a single compu ing node (coa se mesh) and a mul i-node calcula ion ( ine mesh). In
addi ion, o analyse he e ec o load imbalance due o chemical in eg a ion, wo eac ion
mechanisms wi h e y di e en sizes we e chosen o assess he pe o mance o DLB when
using de ailed and educed chemis y. The i s eac ion mechanism is a de ailed chemical
scheme o ke osene comp ising 189 species and 1327 eac ions [1], while he second is a
2-s ep educed mechanism con aining 6 species [14].
Figu e 1: Coun e low ke osene/ai lame using he de ailed eac ion mechanism: empe a u e ( op) and hea
elease a e (bo om).
The second es case is a mo e ealis ic p oblem and ea u es a u bulen p emixed
lame in a swi l-s abilized bu ne also known in he li e a u e as he PRECCINSTA bu ne [20,
6]. The ope a ing poin a equi alence a io φ=0.67 [6] is aken as e e ence bu changing
he uel o ke osene o allow o his compa ison. Fo his case, he go e ning equa ions
a e il e ed in space and he compu a ional amewo k is adap ed o un la ge-eddy sim-
ula ions (LES). De ails o he modelling and nume ical app oach a e gi en in p e ious
wo k [6] and a e omi ed he e o b e i y. Fo his es case, a ini e a e model conside ing
he il e ed equa ions o LES a e employed wi hou accoun ing o Tu bulence-Chemis y
In e ac ions (TCI). This app oach has been chosen due o i s simplici y and wi h he aim
o e alua e he load imbalance in LES and DNS applica ions. No wi hs anding, he con-
clusions d awn in he ollowing ela ed o he imp o emen s o hyb idiza ion and DLB a e
expec ed o be alid o o he u bulen combus ion models ha do chemical in eg a ion in
si u such as he Condi ional Momen Closu e (CMC) [23], Eddy Dissipa ion Concep [13]
o he T anspo ed P obabili y Densi y Func ions (TPDFs) models [21], g ea ly expanding
he po en ial o DLB o ad anced combus ion simula ions. The compu a ional domain
includes he plenum, swi le and combus ion chambe , and is composed by a hyb id mesh
7
including p isms, e ahed ons and py amids. This is shown in Fig. 2 along wi h a sample
snapsho o he empe a u e ield. The same p e ious eac ion mechanisms a e also es ed
in his con igu a ion. A summa y o he compu a ional cases and de ails o he mesh size
and esolu ion is gi en in Table 1.
Figu e 2: Swi l-s abilized u bulen p emixed lame using he de ailed eac ion mechanism: mesh esolu ion
( op) and empe a u e (bo om).
Iden i ie Con igu a ion Mechanism Species Reac ions Mesh (cells)
CF1 Coun e low De ailed 189 1327 6k
CF2 Coun e low Reduced 6 2 6k
CF3 Coun e low De ailed 189 1327 52k
CF4 Coun e low Reduced 6 2 52k
SB1 Swi l bu ne De ailed 189 1327 24M
SB2 Swi l bu ne Reduced 6 2 24M
Table 1: Desc ip ion o he compu a ional cases.
Fo bo h o he cases, wo me ics we e ob ained, namely, he ime elapsed o he
chemical in eg a ion in a Runge-Ku a sub-s ep and he o al ime equi ed o he in eg a-
8
ion o he whole ime s ep. The analysis includes single-node and mul i-node es s on he
coun e low con igu a ion aiming o e alua e he e ec o he hyb idiza ion, g ain size and
he op imiza ion wi h DLB. A e he co ec iden i ica ion o he op imal DLB pa ame e s
o he coun e low lame, he same se ings a e applied o he u bulen swi l bu ne case
in o de o e alua e his me hodology on p oduc ion uns. De ails o he analysis a e gi en
in he nex subsec ions.
4. Compu a ional backg ound
4.1. Pe o mance analysis
We s a he s udy wi h a pe o mance analysis o he execu ion which is ca ied ou
ollowing he POP2me hodology [33, 3]. We use he BSC pe o mance analysis ools3:
EXTRAE [28] o gene a e execu ion aces and Pa a e [26] o isualize hem.
In Figu e 3, we can see wo imelines om Pa a e . Pa a e imelines show in he
x axis he ime and each ow co esponds o one MPI p ocess. The MPI p ocesses a e
o de ed going om MPI ank 0 in he op o he iew o he highes ank in he bo om o
he iew. The colo can iden i y di e en me ics depending on he iew ha is selec ed.
On he op ace, we can see in a colo ba he du a ion o he use ul compu a ion.
We conside use ul compu a ion when he p ocess o h ead is doing compu a ion and no
wai ing in an MPI call. On he bo om ace, we show he MPI call, in his case black
colo co esponds o use ul compu a ion, because i is ou side an MPI call. Bo h aces a e
depic ed using he same imescale.
Fo illus a i e pu poses, in his iew we show wo ime s eps o one o he cases
simula ed in his wo k, namely, he de ailed chemis y simula ion o a coun e low lame.
In his case he execu ion is done using 192 MPI anks. We ha e ma ked wi h a yellow
squa e one o he chemical in eg a ion loops.
We can obse e he impo an load imbalance p esen in his sec ion o he execu ion,
whe e some MPI anks ha e a highe load han o he ones. This p oduces an impo an
ime spen wai ing in he MPI Ba ie call by some MPI p ocesses.
In Figu e 4 we see he same analysis o he simula ion o a coun e low lame wi h
educed chemis y. We can see he 192 MPI anks in he di e en ows and wo ime s eps
in he ime scale (x axis). The op ace shows he du a ion o he use ul compu a ion,
while he bo om one shows he MPI calls being execu ed.
This use case also p esen s a high load imbalance in he chemical in eg a ion phases
being one o he in eg a ion loops again ma ked wi h a yellow squa e. We can obse e
2h ps://pop-coe.eu/
3h ps:// ools.bsc.es
9
Figu e 7: Zoom in a Pa a e ace o wo s eps o he de ailed chemis y showing he e ec o DLB.
16

pe o mance ne wo k in e connec and unning SuSE Linux En e p ise Se e as ope a -
ing sys em. Compu e nodes a e equipped wi h 2 socke s In el Xeon Pla inum 8160 CPU
wi h 24 co es each unning a 2.10GHz o a o al o 48 co es pe node and 96 GB o main
memo y (1.88 GB/co e).
We ha e used he In el compile 2017.4 and IMPI 2017.4 as MPI lib a y. Fo all he
expe imen s we use he mas e b anch o Alya in eg a ed wi h he Can e a lib a y e sion
2.1. Fo he op imiza ion using DLB we ha e used DLB almos 3.0 and OmpSs 19.06.
All he esul s ga he ed in his sec ion a e a e ages o 5 uns. As in all he cases, he
s anda d de ia ion be ween he di e en uns is below 5%, he e o ba s a e no shown in
he plo s.
5.2. Single-node es ing
The main aim o his sec ion is o de e mine he impac o he code hyb idiza ion
wi h OmpSs on he pe o mance as well as o be able o dis inguish in a second s age he
op imiza ions p o ided by DLB. Also, we s udy in de ail he impac o he g ain size when
pa allelizing wi h OmpSs o unde s and he impo ance o his ac o on he compu a ional
cos and o y o ind i s op imal alue o ange o alues.
The i s se o es s comp ises he analysis o he ime educ ions by he use o hy-
b idiza ion and DLB on he coun e low lame con igu a ions CF1 and CF2.
5.2.1. Hyb idiza ion and g ain size s udy
In his sec ion, we analyze he pe o mance o he hyb id code, ou lined in Lis ing 2,
e sus he o iginal MPI-only implemen a ion.
Figu e 8 p esen s a nume ical s udy o he coun e low lame conside ing he de ailed
chemis y CF1 case using a single node o he Ma eNos um IV supe compu e . In he X
axis we can see di e en con igu a ions o MPI p ocesses and OmpSs h eads, e.g. 12 ×4
co esponds o 12 MPI p ocesses and 4 OmpSs h eads each. No e ha he p oduc o
hese combina ions is always 48 as all he esul s in he same plo a e using he same
numbe o compu a ional esou ces (48 co es). In he Y axis we can see he speed up o
he in eg a ion loop wi h espec o he MPI only e sion wi h 48 MPI p ocesses ( he pu e
MPI only is depic ed as a ba in he plo as a e e ence). The di e en se ies ep esen ed
wi h lines co espond o he hyb id e sion wi h di e en alues o he g ain size.
We obse e ha he MPI-only execu ion akes he same ime as he 48 ×1 con ig-
u a ion; he e o e, he ac i a ion o OmpSs does no gene a e an app eciable o e head.
Mo eo e , he end is ha by inc easing he numbe o OmpSs h eads pe node, he
chemis y in eg a ion cos is educed. As he e a e no in e -p ocess communica ions, he
main eason o his accele a ion is he implici load-balancing ob ained om he sha ed
memo y aski ica ion: s i and non-s i asks a e assigned o OmpSs h eads as hey a e
17
0
0,2
0,4
0,6
0,8
1
1,2
1,4
1,6
1,8
Pu e
MPI
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
14816
32 64 128
Figu e 8: Compa ison o ime o de ailed chemis y in eg a ion (case CF1) be ween only MPI and hy-
b idiza ion wi h di e en numbe o h eads o di e en g ain size alues.
comple ed. Finally, we obse e ha esul s a e almos independen o he g anula i y as
he o e lapping o he di e en lines shows. This means ha he e is no ele an o e head
added by he hyb idiza ion o OmpSs and also ha he g anula i y o asks is small enough
o p o ide pa allelism and wo k o all he h eads. Only he g anula i y o one elemen
pe ask shows a e y sligh educ ion o he speedup o he con igu a ion 12 ×4, his is
because in his case 12 OmpSs h eads a e asking o wo k o he un ime sys em, as he
asks a e e y small we can s a o see some conges ion in he un ime when accessing he
queue o wo k. All in all, 6 ×8 is he bes con igu a ion, and he speedup ob ained e sus
he pu e MPI code is 1.6×.
In Figu e 9 we see he same esul s o he educed chemis y CF2 case o which
quali a i e di e ences a ise wi h ega ds o he de ailed chemis y case CF1. In his case,
a di ec in eg a ion o he chemis y can be achie ed wi h much ewe i e a ions in he
CVODE sol e , ob aining much lowe o e all compu ing imes compa ed o he de ailed
chemis y case CF1. On he one hand, he g ain size signi ican ly a ec s he pe o mance.
In his case, unlike o he de ailed chemis y, he cos o he in eg a ion o a single elemen
is e y low, so a couple o hem need o be ga he ed pe ask o coun e balance he OmpSs
o e head. On he o he hand, we obse e ha he speedup ises up o 3.4×and he bes
esul is ob ained wi h he con igu a ion 12×4. This highe speedup comes om an ini ially
highe imbalance as we ha e seen in he pe o mance analysis sec ion (Sec ion 4.1), whose
co ec ion p oduces mo e no iceable e ec s.
As he p oblem is en i ely go e ned by chemis y, he e alua ion o he global a es is
local and can s ongly bene i om pa alleliza ion. This s udy p oposes he use o hyb id
18
0
0,5
1
1,5
2
2,5
3
3,5
4
Pu e
MPI
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
1
4
8
16
32 64 128
Figu e 9: Compa ison o ime o educed chemis y in eg a ion (case CF2) be ween only MPI and hy-
b idiza ion wi h di e en numbe o h eads o di e en g ain size alues.
code ha uses he pa allel p og amming model OmpSs o imp o e he load balance and
educe he compu a ional cos . Mo eo e , he OmpSs pa alleliza ion does no add a sig-
ni ican o e head al hough, in cases wi h a e y low compu a ional load o he chemical
in eg a ion, a small g ain size can add o e head when he numbe o h eads is inc eased.
In hese cases, he g ain size migh ha e an impo an impac on he pe o mance and a
mid o high alue o he g ain size can app eciably mi iga e such o e head.
5.2.2. DLB e alua ion
In his sec ion, we s udy he bene i o using DLB as an addi ional load balancing
mechanism in a-node and how he g ain size a ec s in hese simula ions.
In Figu e 10, we show he speedup ob ained by he hyb id e sion wi h and wi hou
DLB wi h espec o he MPI-only e sion (shown as a blue ba in he plo ) when unning
in one node o Ma enos um4. In he X axis we can see he di e en con igu a ions o MPI
h eads and OmpSs h eads o ill one node o 48 co es. Fo all he hyb id execu ions we
use a g ain size o 32, because ha is he minimal size ha does no show any signi ican
o e head in his p oblem.
Adding DLB o he de ailed chemis y in eg a ion as shown o CF1, Figu e 10,
gene a es a speedup o up o 2.3× e sus he MPI-only implemen a ion, and up o 1.5×
e sus he bes hyb id con igu a ion. This indica es ha he use o DLB can imp o e he
pe o mance and add ess he imbalance u he han only he hyb idiza ion o he code.
I is ele an ha wi h he con igu a ion 48 ×1, we ob ain esul s ha a e no a om
op imal, a speed up o 2×. I means ha we can ge mos o he pe o mance wi hou
19
0
0,5
1
1,5
2
2,5
Pu e
MPI
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
Hyb id DLB
Hyb id No LB
Figu e 10: Compa ison o ime o de ailed chemis y in eg a ion (case CF1) be ween only MPI and hy-
b idiza ion wi h and wi hou DLB o di e en numbe o h eads wi h g ain size 32.
equi ing an o e all hyb idiza ion o he code; only one h ead pe MPI p ocess needs
o be ac i a ed in he zones whe e DLB is used. I is impo an o no ice ha he line
co esponding o he execu ions wi h DLB (o ange) is la e han he one co esponding o
he hyb id code. This means ha he use o DLB makes he pe o mance less dependen on
he hyb id con igu a ion, hus, i elie es he p essu e om he use o decide he op imal
con igu a ion o MPI p ocesses and h eads.
Fo he educed chemis y CF2, we can see he esul s ob ained wi h DLB in Figu e 11.
As in he p e ious plo , we show he speedup o e he MPI-only e sion (Y axis), when
using di e en con igu a ions o MPI p ocesses and OmpSs h eads (X axis). I is obse ed
ha he speedup e sus he MPI-only e sion ises up o 7×, which ep esen s an addi ional
2×accele a ion e sus he bes hyb id op ion.
The speedup ob ained by DLB in he educed chemis y in eg a ion is highe han
in he de ailed one, as he load imbalance is also highe in he educed chemis y case.
No e ha he speedup ha DLB can ob ain is ela ed o he exis ing load imbalance o he
applica ion. As DLB e-dis ibu es he asks by he idle ime om he p ocesso s, he e is
no need o p edic he s i ness om he chemical p oblem as his is handled by DLB.
A second impo an aspec o conside is he impac o he g ain size on he DLB
pe o mance in hese applica ions. Figu e 12 shows he speedup e sus he MPI-only
e sion o he code (y axis) o he de ailed chemis y case. We can see he di e en
con igu a ions o MPI p ocesses and OmpSs h eads o ill a node o 48 co es. The di e en
se ies ep esen he di e en g ain sizes used by he hyb id code and DLB. To unde s and
he esul s, a ade-o be ween wo aspec s needs o be conside ed: i) he imbalance can be
20
0
1
2
3
4
5
6
7
8
Pu e
MPI
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
Hyb id DLB
Hyb id No LB
Figu e 11: Compa ison o ime o educed chemis y in eg a ion (case CF2) be ween only MPI and hy-
b idiza ion wi h and wi hou DLB o di e en numbe o h eads wi h g ain size 32.
1
1,2
1,4
1,6
1,8
2
2,2
2,4
2,6
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
4
8
16
32
64
128
Figu e 12: Compa ison o g ain size impac when using DLB o de ailed chemis y in eg a ion (case CF1)
o di e en numbe o h eads.
21

be e educed wi h hinne g anula i y; ii) he o e head o OmpSs is in e sely p opo ional
o he ask size. We can see ha he op imal g ain size is always lowe han o equal
o 32. In he de ailed case, he imbalance domina es he ade-o because he a e age
cos o he chemis y in eg a ion pe elemen is la ge compa ed o he OmpSs o e heads.
Conside ing he op imal g ain size o each con igu a ion, he speedup ob ained anges
be ween 2×and 2.3×wi h he 48 ×1 con igu a ion, we achie e 86% o he maximum
speedup. Mo eo e , a low a iance be ween di e en con igu a ions is obse ed, which
ein o ces p e ious obse a ions abou he use o DLB o isola e he pe o mance om he
selec ed con igu a ion.
1
2
3
4
5
6
7
8
48x1 24x2 12x4 8x6 6x8 4x12
Speed up In eg a ion
Con ig (MPIs x OmpSs h eads)
4
8
16
32
64
128
Figu e 13: Compa ison o g ain size impac when using DLB o educed chemis y in eg a ion (case CF2)
o di e en numbe o h eads.
The same s udy was done o he educed chemis y case CF2 and i is shown in Fig-
u e 13 whe e we obse e mo e a iabili y wi h espec o he g ain size. No e ha , since
he a e age chemis y in eg a ion cos pe elemen is much lowe , he o e head becomes
signi ican o low g ain sizes. Consequen ly, o all con igu a ions he op imal g anula i y
is always equal o o g ea e han 32. Conside ing he op imal g ain size o each con igu-
a ion, he speedup ob ained anges be ween 6.4×and 7×achie ing 91% o he maximum
speedup o he 48 ×1 con igu a ion.
Rega ding he whole ime-s ep, he implemen a ion o Alya is no comple ely hyb id
wha makes mo e adequa e he con igu a ion o 48 ×1 o p oduc ion simula ions since
o he wise, o he pa s o he ime-s ep would be penalized because some CPU-co es e-
se ed o OmpSs h eads would no be used. Fo he de ailed chemis y model CF1, i s
in eg a ion ep esen s 59% o he o e all ime s ep, so he 2.4×accele a ion achie ed wi h
22
he 48 ×1 con igu a ion esul s in a 1.6×o e all accele a ion. In he educed chemis y
scena io CF2, i s sha e is 26% o he ime-s ep, he e o e i s 7×accele a ion esul s in a
1.4×o e all accele a ion.
In his sec ion, we ha e shown ha DLB can imp o e he pe o mance o he chemical
in eg a ion loop u he han he hyb idiza ion o he code. Mo eo e , i is con i med ha
he speedup achie ed by DLB depends on he imbalance p esen in he o iginal execu ion
achie ing a 7×speedup o highly imbalanced uns wi h he same numbe o esou ces.
I is seen ha he selec ion o he g ain size has an impac on he pe o mance o DLB.
Finally, la ge g ain sizes o e less lexibili y o DLB o load balance, so he bes ade-o
be ween lexibili y and o e head is a ound a g ain size o 32 o all he cases.
5.3. Mul i-node es ing
As explained in Sec ion 4.2, DLB can only load balance wi hin a compu a ional node
wi h sha ed memo y, so in his sec ion we demons a e how DLB can imp o e he pe o -
mance o mul i-node execu ions e en hough i is only ac ing locally inside he node.
When conside ing mul i-node simula ions i is impo an o no ice ha , by de aul ,
esou ce manage s spawn he MPI p ocesses con iguously in he di e en compu a ional
nodes. Wi h he con inuous binding, subdomains associa ed wi h p ocesses unning in he
same node end o be adjacen in he domain. Wi h he Round Robin (RR) binding, his
locali y is a oided on pu pose o make each node o ha e di e en pa s o he domain.
This s a egy is well sui ed o combus ion simula ions, since chemical eac ions usually
occu in speci ic loca ions o he domain and along hin laye s, so he MPI anks con aining
he eac ing laye s end o be close o each o he . In o de o dis ibu e he mos loaded
p ocesses among he di e en nodes and imp o e he pe o mance o he load balancing
mechanism o DLB, i is usually be e o use a Round Robin (RR) dis ibu ion o MPI
p ocesses.
Figu e 14: Con iguous s. Round Robin dis ibu ion o MPI anks.
23
In Figu e 14, we show an example o a combus ion domain (le hand side) pa i ioned
be ween 8 MPI anks. This pa i ion assigns he nodes o he eac ing laye , which is he
mos compu a ionally expensi e o MPI ank 6 and 7. When a con iguous dis ibu ion o
MPI anks among nodes is done, he si ua ion depic ed in he op igh hand side pa o he
igu e is ound. Whe e MPI anks 1, 2, 3 and 4 a e assigned o Node 1 and MPI, anks 5,
6, 7 and 8 a e assigned o Node 2. Wi h his dis ibu ion, he wo mo e loaded p ocess a e
assigned o Node 2, p oducing a load imbalance ac oss nodes ha can no be add essed by
DLB. Howe e , conside ing a Round Robin dis ibu ion ins ead, bo om igh hand side,
Node 1 ge s MPI anks 1, 3, 5 and 7, and Node 2 ge s MPI anks 2, 4, 6 and 8. A good
load balance ac oss nodes is ound wi h his dis ibu ion, so he load imbalance inside he
nodes can be add essed wi h DLB.
This aspec is in es iga ed he e by he analysis o a coun e low di usion lame, co e-
sponding o cases CF3 and CF4 wi h de ailed and educed chemis y, espec i ely. These
cases di e om he p e ious cases CF1 and CF2 by ha ing a ine mesh wi h abou 4
imes la ge compu a ional load and hen i will be ex ended o u bulen p emixed lames,
cases SB1 and SB2. Fo hese cases, we conside he ange om 1 node (48 CPU-co es)
up o 16 nodes (768 CPU-co es). No e ha he chemis y in eg a ion does no equi e any
MPI communica ion o synch oniza ion ope a ion and, he e o e, he mos limi ing ac o
o scalabili y is he imbalance. All he execu ions in his sec ion a e done using a con ig-
u a ion o 48 ×1 o MPI p ocesses and OmpSs h eads pe node, as we wan o mimic a
p oduc ion un o Alya and a g ain size o 32, because i is he op imum alue de e mined
in he p e ious sec ion.
0
2
4
6
8
10
12
14
48 96 192 384 768
Spped up
# MPI anks
pu e MPI con iguous
Hyb id + DLB Con iguous
pu e MPI Round Robin
Hyb id + DLB Round Robin
Figu e 15: Speedup-up o de ailed chemical in eg a ion (case CF3) up o 16 nodes using DLB and a ying
dis ibu ion o MPI anks among nodes, wi h a con igu a ion o 48 ×1 and g ain size 32.
24
In Figu e 15, we see he speedup (Y axis) o he chemis y in eg a ion s age no malized
by he MPI-only execu ion on 48 co es as unc ion o he numbe o MPI anks used in he
simula ion. In hese plo s, solid lines ep esen con iguous binding, o which MPI anks
a e placed con iguously in he nodes, while dashed lines a e used o ep esen he Round
Robin binding, whe e MPI anks a e spawned in a Round Robin mode among he compu e
nodes. MPI only execu ions a e ep esen ed wi h blue lines, while hyb id DLB execu ions
a e ep esen ed wi h o ange lines.
We can see ha he binding has low impac on he pe o mance o he pu e MPI imple-
men a ion (blue lines o e lap). Con a ily, when DLB is used, he RR binding is help ul o
b eak he subdomain’s locali y and a oid si ua ions whe e he subdomains associa ed wi h
p ocesses o a node co e egions wi h simila condi ioning. In his case, we can see ha
o 2 nodes (96 MPI anks) he e is almos no di e ence. Bu o highe numbe o nodes
DLB is able o ob ain a be e speedup when using a Round Robin dis ibu ion. Wi h 768
MPI anks, i ob ains a 12×speedup wi h Round Robin e sus a 10×speedup wi h con-
iguous dis ibu ion. Addi ionally, DLB imp o es he pe o mance o he simula ion by a
ac o o 2×wi h espec o he o iginal pu e MPI un when using 16 nodes.
0
5
10
15
20
25
48 96 192 384 768
Speed up
# MPI anks
pu e MPI con iguous
Hyb id + DLB Con iguous
pu e MPI Round Robin
Hyb id + DLB Round Robin
Figu e 16: Speedup-up o educed chemical in eg a ion (case CF4) up o 16 nodes using DLB and a ying
dis ibu ion o MPI anks among nodes, wi h a con igu a ion o 48 ×1 and g ain size 32.
In Figu e 16, we see he same speedup, bu o he educed chemis y case CF4. The
blue lines co espond o uns o he o iginal MPI-pu e code, while o ange lines co espond
o hyb id uns wi h DLB. I is obse ed ha he dis ibu ion o MPI p ocesses among
nodes does no ha e an impac on he pe o mance o he o iginal pu e MPI code (blue
lines o e lap), as i also occu s wi h he de ailed chemis y case. Howe e , we obse e
ha he impac o he ound Robin dis ibu ion in his case is e en highe han o he
25
[31] D. Th´
e enin, F. Beh end , U. Maas, B. P zywa a, and J. Wa na z. De elopmen o a
pa allel di ec simula ion code o in es iga e eac i e lows. Compu e s and Fluids,
25(5):485–496, 1996.
[32] M. Vazquez, G. Houzeaux, S. Ko ic, A. A igues, J. Aguado-Sie a, R. A is, D. Mi a,
H. Calme , F. Cucchie i, H. Owen, A. Taha, J. M. Cela, and M. Vale o. Mul iphysics
enginee ing simula ion owa d exascale. J. Compu . Sci., 14:15 – 27, 2016.
[33] M. Wagne , S. Moh , J. Gim´
enez, and J. Laba a. A s uc u ed app oach o pe o -
mance analysis. In In e na ional Wo kshop on Pa allel Tools o High Pe o mance
Compu ing, pages 1–15. Sp inge , 2017.
[34] T. Zi wes, F. Zhang, J. A. Dene , P. Habis eu he , and H. Bockho n. Au oma ed
code gene a ion o maximizing pe o mance o de ailed chemis y calcula ions in
open oam. In Wol gang E. Nagel, Die ma H. K ¨
one , and Michael M. Resch, edi-
o s, High Pe o mance Compu ing in Science and Enginee ing ’ 17, pages 189–204,
Cham, 2018. Sp inge In e na ional Publishing.
[35] T. Zi wes, F. Zhang, J.A. Dene , P. Habis eu he , and H. Bockho n. Au oma ed code
gene a ion o maximizing pe o mance o de ailed chemis y calcula ions in open-
oam. High Pe o mance Compu ing in Science and Enginee ing’17, pages 189–204,
2018.
32