Ad ancing Sol e Pe o mance in
La ge-Scale Compu a ional Fluid
Dynamics: Gene alizing he
Linele P econdi ione o he
P essu e Co ec ion Equa ion
Au ho : Rami o de Olazábal
Supe iso s:
D . O iol Lehmkuhl Ba ba
D . Rica d Bo ell
A Thesis submi ed o
Uni e si a Poli ècnica de Ca alunya
o he deg ee o
Doc o o Philosophy
Ma ch 2025
Acknowledgemen s
I wan o exp ess my since e g a i ude o my supe iso s, D . O iol Lehmkuhl
and D . Rica d Bo ell, o selec ing me o his Ph.D. p og am. I was no an
easy jou ney. Al hough he challenges, hei guidance, suppo , and eachings
ha e been in aluable, and I eel hono ed o ha e wo ked wi h hem.
I am deeply hank ul o D . He be Owen o his suppo h oughou his
p ocess. I am also hank ul o D . Ma ias A ila o his pa ience wi h my many
ques ions and D . Guillaume Houzeaux o smoo hing my ela ionship wi h
Alya. Special hanks go o D . Lucas Gaspa ino and D . Sa a h Radhak ishnan
o hei suppo bo h wi hin and ou side BSC, o Ca los A nedo o his
pa ience while eaching me o ace, and o Samuel Gomez and Ma c Sil es e i
Cla os o helping me wi h ANSA e e y ime I needed i . I since ely hank all
my colleagues and iends a BSC; his jou ney would ha e been much mo e
challenging wi hou hem.
I would like o acknowledge he Ba celona Supe compu ing Cen e o p o iding
he esou ces and acili ies necessa y o conduc ing his esea ch. I am also
g a e ul o he inancial suppo om Spain’s Minis e io de Ciencia e Inno ación
h ough he g an “Ayudas pa a con a os p edoc o ales pa a la o mación de
doc o es” (Re : PRE2018-086548), which allowed me o ad ance my s udies
and esea ch.
The e is no way o hank my amily enough. I am especially g a e ul o my
pa en s and sis e , who ha e always been by my side despi e he dis ance,
o e ing unwa e ing suppo and encou agemen .
I mus also exp ess my g a i ude o D . Fe nando Lomba do. I owe him he
oppo uni y o wo k in such an inc edible and in ellec ually challenging place
as BSC. You suppo and belie in me ha e been genuinely decisi e in my ca ee .
I am equally hank ul o he Echa a ía and Belmon e amilies o wa mly
welcoming me when I a i ed in Ba celona and making me eel a home. Thei
kindness has g ea ly eased he challenges o hese yea s.
I canno o ge my iends—bo h li elong companions and hose I made o e
hese yea s. Thank you all: o Da id Oks, o easing my ansi ion in o Ba -
i
celona; o San i, o he engaging con e sa ions and sha ed bee s; o C is óbal,
o he s imula ing alks and wines; o Pablo, B uno, and Da id, o he ips
and good imes; o Nico, o he sha ed momen s; and Emi and Juani, o
b inging a ouch o A gen ina’s wa m h.
I am deeply g a e ul o e e yone who con ibu ed o his hesis in one way o
ano he o hei suppo , guidance, and encou agemen .
ii
Abs ac
The e icien solu ion o he Poisson equa ion is a undamen al challenge in
Compu a ional Fluid Dynamics (CFD), pa icula ly in la ge-scale simula ions
in ol ing high-Reynolds-numbe lows. The s ong aniso opy in oduced
by hin bounda y laye s equi es highly s e ched elemen s, which, in u n,
signi ican ly impac he con e gence o i e a i e sol e s. In such cases, he
P econdi ioned Conjuga e G adien (PCG) sol e , commonly used o p essu e
co ec ion, exhibi s slow con e gence due o he de e io a ed condi ioning o
he sys em ma ix. To add ess his issue, he Linele P econdi ione has been
widely adop ed, as i exploi s he unde lying mesh aniso opy by cons uc ing
linele s—one-dimensional s uc u es aligned wi h he s onges couplings—and
applying specialized ope a ions along hese segmen s. Howe e , adi ional
implemen a ions o he Linele P econdi ione ope a e wi hin each domain
pa i ion independen ly, esul ing in deg aded pe o mance as he numbe o
pa i ions inc eases in massi ely pa allel simula ions.
This hesis p esen s he Global Linele P econdi ione (GLP), a p econdi-
ioning s a egy designed o o e come hese limi a ions and imp o e scalabili y
in ex eme-scale CFD applica ions. The me hod ex ends and gene alizes he
adi ional linele app oach by in oducing a communica ion s ep wi hin he
p econdi ioning ope a ion, allowing in e domain coupling and p ese ing con-
nec i i y ac oss pa i ion bounda ies. This modi ica ion signi ican ly enhances
con e gence a es in highly aniso opic meshes by ensu ing ha he s onges
couplings in he linea sys em a e ea ed e ec i ely, ega dless o domain
decomposi ion cons ain s.
A key con ibu ion o his wo k is he de elopmen o a pu ely algeb aic
linele cons uc ion algo i hm, which elimina es he need o geome ic in-
o ma ion when de ining linele s. While con en ional me hods ely on explici
mesh s uc u es o de e mine aniso opic di ec ions, he algeb aic app oach
cons uc s linele s based solely on ma ix p ope ies, allowing g ea e lexibili y
and applicabili y o gene al uns uc u ed meshes. Fu he mo e, he geome ic-
based cons uc ion was also explo ed and in eg a ed wi hin he amewo k,
demons a ing supe io pe o mance in s uc u ed meshes wi h well-de ined
aniso opic ea u es. The compa ison be ween he geome ic and algeb aic
app oaches e ealed ha while he o me achie es be e pe o mance when
clea di ec ional s i ness is p esen , he la e p o ides a obus al e na i e
when mesh opology is complex o una ailable.
iii
The e ec i eness o GLP was assessed h ough ex ensi e nume ical expe i-
men s, including benchma k p oblems and eal-wo ld CFD applica ions such
as he 30P30N high-li ai oil, he S an o d di use , and he D i Ae model.
Resul s demons a ed ha GLP signi ican ly imp o es sol e con e gence o e
exis ing p econdi ione s, including p e ious e sions o he linele p econdi-
ione , pa icula ly in cases whe e a high pe cen age o elemen s lie wi hin
he bounda y laye . Pe o mance analyses e ealed ha while GLP incu s
a highe p ep ocessing cos due o linele cons uc ion and communica ion,
hese o e heads a e ou weighed by he subs an ial educ ion in PCG i e a ions,
leading o o e all compu a ional sa ings in la ge-scale simula ions.
In addi ion o imp o ing con e gence, GLP in oduces a pa i ion-agnos ic
o mula ion, making i independen o he domain decomposi ion s a egy.
Unlike adi ional p econdi ione s, which a e sensi i e o mesh pa i ioning,
GLP main ains i s nume ical pe o mance ac oss a ying decomposi ion con ig-
u a ions, enabling mo e lexible and balanced pa allel execu ion. The pa allel
implemen a ion o he me hod, ailo ed o High Pe o mance Compu ing
(HPC) en i onmen s, ensu es scalabili y ac oss a wide ange o co e coun s, as
demons a ed by de ailed scalabili y analyses.
Keywo ds: Compu a ional Fluid Dynamics, Poisson Equa ion, P econdi-
ioned Conjuga e G adien , Linele P econdi ione , High-Pe o mance Compu -
ing, Aniso opic Meshes, Pa allel Sol e s, Algeb aic P econdi ione s, Domain
Decomposi ion.
i
Con en s
Acknowledgemen s i
Abs ac iii
Con en s
Lis o Figu es ii
Lis o Tables xii
Lis o ac onyms and abb e ia ions xi
1 In oduc ion and undamen als 1
1.1 Mo i a ion............................. 1
1.2 Objec i es and hesis s uc u e . . . . . . . . . . . . . . . . . . 4
1.3 Go e ning Equa ions . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.1 Re e ence sys ems . . . . . . . . . . . . . . . . . . . . . 6
1.3.2 Fluid mechanics . . . . . . . . . . . . . . . . . . . . . . 8
Comple e Se o Equa ions . . . . . . . . . . . . . . . . 8
1.4 Disc e iza ion o he Na ie -S okes equa ions . . . . . . . . . . 9
1.4.1 Space Disc e iza ion . . . . . . . . . . . . . . . . . . . 10
Weak Fo mula ion . . . . . . . . . . . . . . . . . . . . 10
Weak Fo mula ion o he Na ie -S okes Equa ions . . . 11
Time-dependen o mula ion: . . . . . . . . . . . 11
Fini e Elemen Me hod . . . . . . . . . . . . . . . . . . 12
Fini e Elemen Disc e iza ion o he Na ie -S okes
Equa ions.................... 14
1.4.2 Time Disc e iza ion . . . . . . . . . . . . . . . . . . . . 15
F ac ional S ep Schemes o he Na ie -S okes Equa ions
16
1.5 Me hods o Sol ing Linea Sys ems . . . . . . . . . . . . . . . 17
1.5.1 Di ec Me hods . . . . . . . . . . . . . . . . . . . . . . 18
1.5.2 Mul ig id Me hods . . . . . . . . . . . . . . . . . . . . 18
1.5.3 I e a i e Me hods . . . . . . . . . . . . . . . . . . . . . 18
K ylo Me hods...................... 19
Conjuga e G adien Me hod . . . . . . . . . . . . . . . 19
P econdi ioned Conjuga e G adien Me hod . . . . . . 20
1.5.4 Di e en ypes o p econdi ione s . . . . . . . . . . . . 22
Con en s
1.6 Linele p econdi ione . . . . . . . . . . . . . . . . . . . . . . . 23
1.6.1 Ca ac e iza ion o he linele p econdi ione . . . . . . 25
1.6.2
E ec o he in eg a ion ule on he linele p econdi ione
27
1.6.3 Pa alleliza ion o he linele p econdi ione . . . . . . . 29
1.7 Auxilia y me hods . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.7.1 TDMA: T idiagonal Ma ix Algo i hm . . . . . . . . . 33
1.7.2 The Schu -complemen me hod . . . . . . . . . . . . . 33
1.8 Conclusions ............................ 35
Re e ences................................ 35
2
E icien Pa alleliza ion o he T idiagonal Ma ix Algo i hm
39
2.1 Va ious app oaches . . . . . . . . . . . . . . . . . . . . . . . . 40
2.1.1 P e ious app oaches . . . . . . . . . . . . . . . . . . . 40
2.1.2 Ou app oach: agnos ic o he domain pa i ion . . . . 40
2.2
Pa alleliza ion o he TDMA using he Schu -Complemen
Me hod............................... 41
2.2.1
An e icien way o compu e each slice’s con ibu ion
o he .h.s. ........................ 41
Di ec compu a ion o he p oduc As,pA−1
p,p ...... 42
2.2.2 Pa allel app oach o sol e he linea sys em . . . . . . 45
2.3 Conclusions ............................ 49
Re e ences................................ 49
3 Assembling he p econdi ioning ma ix 51
3.1 Assembling he slices . . . . . . . . . . . . . . . . . . . . . . . 52
3.1.1 Algeb aic slice assembly wi hin each subdomain . . . . 52
Rema ks on choosing he nex node in he slice . . . . 53
3.1.2 Geome ic slice assembly wi hin each subdomain . . . 58
3.2 Halocleaning ........................... 59
3.2.1
Each p ocess has he global ID o i s nodes and he
nodes o which hey a e coupled ia he halo . . . . . . 61
3.2.2
Each p ocess does no ha e he global ID o i s nodes
and he nodes o which hey a e coupled ia he halo . 62
3.3 G owing he linele s: joining slices om di e en subdomains 66
3.3.1
Selec ing a speci ic se o slices o be he s a ing poin
o hei espec i e linele s . . . . . . . . . . . . . . . . 67
3.3.2 G owing he linele s wi hou speci ying a s a ing poin 70
Loop inding s a egies . . . . . . . . . . . . . . . . . . 76
Loop cleaning s a egy . . . . . . . . . . . . . . . . . . 78
G owing he linele s . . . . . . . . . . . . . . . . . . . . 79
3.4 Conclusions ............................ 79
Re e ences................................ 80
4 Implemen a ion and pe o mance 81
4.1 Implemen a ion .......................... 82
4.2 Wo k low.............................. 82
4.3 Compu a ional Resou ces . . . . . . . . . . . . . . . . . . . . . 82
4.4 Nume ical e i ica ion o he sol e . . . . . . . . . . . . . . . 83
4.5 Analysis o pe o mance . . . . . . . . . . . . . . . . . . . . . 98
4.5.1 Execu ion Analysis . . . . . . . . . . . . . . . . . . . . 98
i
4.5.2 Scalabili y Analysis and Pe o mance Me ics . . . . . 101
4.6 P ep ocessing s age . . . . . . . . . . . . . . . . . . . . . . . . 106
4.7 Conclusions ............................ 109
Re e ences................................ 113
5 Nume ical Applica ions – Sol ing Real-Wo ld Scena ios 114
5.1 30P30NAi oil........................... 115
5.1.1 Compu a ion and Communica ion Time Analysis . . . 116
5.1.2 Compu a ional Cos pe PCG I e a ion . . . . . . . . . 117
5.2 S an o d Di use . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.3 D i Ae Model........................... 127
5.4 Pe o mance Analysis . . . . . . . . . . . . . . . . . . . . . . . 130
5.4.1 P ep ocessing S age . . . . . . . . . . . . . . . . . . . . 130
5.4.2 P econdi ioning S ep . . . . . . . . . . . . . . . . . . . 132
5.5 Scalabili y Analysis and Pe o mance Me ics . . . . . . . . . 134
5.5.1 SpeedUp ......................... 134
5.5.2 Communica ion E iciency . . . . . . . . . . . . . . . . 135
5.5.3 Load Balance . . . . . . . . . . . . . . . . . . . . . . . 136
5.5.4 Pa allel E iciency . . . . . . . . . . . . . . . . . . . . . 137
5.6 Conclusions ............................ 139
Re e ences................................ 141
6 Conclusion 142
6.1 Limi a ions............................. 143
6.2 Fu u eWo k............................ 145
Re e ences................................ 145
Appendices 146
A Lis o publica ions and con ibu ions 147
A.1 Academic w i ings . . . . . . . . . . . . . . . . . . . . . . . . . 147
A.2 Con e ences and wo kshops . . . . . . . . . . . . . . . . . . . . 147
B De ailed desc ip ion on g owing he linele s 148
B.1 G owing he linele s om selec ed slices . . . . . . . . . . . . 148
B.2 G owing he linele s wi hou speci ying a s a ing poin . . . . 153
Lis o Figu es
1.1 Eule ian, Lag angian, and ALE e e ence sys ems. . . . . . . . . . 7
ii
Lis o Figu es
1.2
Linele s uc u e de i ed om he s onges couplings in he mesh
and he idiagonal ma ix ob ained by exclusi ely conside ing hese
dominan connec ions. . . . . . . . . . . . . . . . . . . . . . . . . . 24
1.3
Compa ison o he numbe o i e a ions equi ed o each a
ole ance o 10
−6
o he linele and diagonal p econdi ione in
he PCG Me hod on meshes wi h a ying maximum aspec a io
and bounda y laye co e age. . . . . . . . . . . . . . . . . . . . . . 26
1.4
Con e gence o he esidual o he linele p econdi ione o
di e en in eg a ion ules. . . . . . . . . . . . . . . . . . . . . . . . 28
1.5
Con e gence o he p econdi ioned esidual o he linele p econ-
di ione o di e en in eg a ion ules. . . . . . . . . . . . . . . . . 28
1.6
Same mesh as depic ed in Figu e 1.2a, bu dis ibu ed ac oss i e
dis inc subdomains. The con en ional me hod in ol es es ic ing
linele s o a single domain pa i ion, excluding in e -subdomain
couplings om he p econdi ioning ma ix. . . . . . . . . . . . . . 30
1.8
Con e gence o he PCG me hod o he case o a linele
p econdi ione applied o a pa i ioned mesh. . . . . . . . . . . . . 30
1.7
Example o a mesh o he nume ical simula ion o an ai plane
wing. On he igh , a close iew p o ides de ailed insigh in o he
p isma ic bounda y laye . Typically, a signi ican pe cen age o he
mesh elemen s a e loca ed he e. The p onounced aniso opy in his
egion in oduces s i ness o he disc e e Poisson’s equa ion. . . . 31
1.9
Numbe o i e a ions equi ed o he case shown in Figu e 1.7 o
each a ole ance o 10−3a each ime s ep. . . . . . . . . . . . . . 32
1.10
Numbe o i e a ions equi ed o he case shown in Figu e 1.8 o
each a ole ance o 10−6a each ime s ep. . . . . . . . . . . . . . 32
2.1
De ini ion o he e ms "slice" and "linele ": We in oduce he e m
"slice" o e e o he local s uc u e o nodes ha a e connec ed
wi hin a single subdomain. In con as , "linele " e e s o he global
s uc u e a ising om many slices’ connec ion. In e ace nodes a e
depic ed as emp y nodes, and inne nodes as illed ones. . . . . . . 41
2.2
Diag am showing he idiagonal s uc u e o a linele p econdi-
ioningma ix. ............................. 43
2.3
S uc u e o a eo de ed linele p econdi ioning ma ix acco ding
o he Schu algo i hm. . . . . . . . . . . . . . . . . . . . . . . . . 44
2.4
Schema ic ep esen a ion o he p oposed pa allel algo i hm o sol e
he idiagonal p econdi ioning ma ix. Each linele is assigned o
a speci ic p ocess esponsible o sol ing he linea sys em a he
in e ace. ................................ 48
3.1 I e a i e p ocess o build slices. . . . . . . . . . . . . . . . . . . . 54
3.2
Ske ch showing nodes and couplings o an example mesh. Nodes
h,iand jshould belong o he same slice. . . . . . . . . . . . . . . 55
iii
CHAPTER 1
In oduc ion and undamen als
1.1 Mo i a ion
Many physical laws a e exp essed h ough di e en ial equa ions, as hey p o ide
a undamen al amewo k o desc ibing he dynamics o physical sys ems.
Howe e , analy ical solu ions exis only o a limi ed se o simpli ied cases,
necessi a ing nume ical me hods o mo e ealis ic scena ios. Fo example, in
Compu a ional Fluid Dynamics (CFD), accu a e simula ions equi e sol ing
la ge linea sys ems de i ed om he Na ie -S okes equa ions, which go e n
luid mo ion h ough nonlinea , coupled pa ial di e en ial equa ions. Despi e
signi ican p og ess in nume ical me hods, sol ing hese equa ions emains
challenging. Mo eo e , p o ing he exis ence and smoo hness o solu ions o
he Na ie -S okes equa ions is ecognized as one o he millennium p oblems in
ma hema ics [5].
Nume ical solu ions p o ide insigh s in o c i ical aspec s o luid beha-
io , such as u bulence, hea ans e , and p essu e dis ibu ion, which a e
essen ial o op imizing designs, ensu ing sa e y, and ad ancing echnology.
Consequen ly, CFD has become an indispensable ool ac oss a ious indus ies,
including ae ospace, au omo i e, ene gy, medicine, and en i onmen al science.
Applica ions ange om designing mo e ae odynamically e icien ai c a and
educing ehicle emissions o op imizing wind u bines and modeling blood low
in medical esea ch.
While nume ical me hods o sol ing he Na ie -S okes sys em o equa ions
go e ning luid low we e p oposed as ea ly as he 19 h cen u y, he ad en o
digi al compu ing in he mid-20 h cen u y enabled mo e complex algo i hms
o be de eloped and applied o mo e ealis ic scena ios. One o he main
challenges in inding an accu a e solu ion o his se o equa ions is he ise
o u bulen beha io in he low, whe e he eloci y ield becomes chao ic
and unp edic able. This phenomenon is cha ac e ized by a wide ange o
in e ac ing scales in which he ene gy cascades om la ge, uns able eddies
o smalle ones un il i is dissipa ed in o hea . The ac ha he Na ie -
S okes equa ions gi e ise o his chao ic and mul iscale beha io makes he
di ec solu ion o hese equa ions in easible o mos u bulen lows. The
g ow h o massi ely pa allel compu ing a chi ec u es in ecen decades has
signi ican ly expanded he applicabili y o hese me hods o la ge-scale p oblems.
1
1.1. Mo i a ion
Fo incomp essible lows, he ac ional s ep p ojec ion me hod is commonly
used o decouple he eloci y and p essu e ields while ensu ing mass conse -
a ion [22]. This app oach equi es sol ing a Poisson equa ion o p essu e
co ec ion a each ime s ep, which is o en he mos compu a ionally expensi e
componen o he sol e and also one o he mos di icul o pa allelize. In ou
expe ience, we ha e obse ed ha his equa ion mus be sol ed wi h he highes
possible p ecision. The eason behind his s ingen accu acy c i e ion lies in
he necessi y o uphold mass conse a ion. When sol ed wi h low p ecision
h esholds, he conse a ion o mass ends o be comp omised, leading o
non-con e gence o he p oblem. Also, he ime aken o sol e his Poisson
equa ion g ows wi h he ime s ep employed, making e icien sol e s essen ial
o la ge-scale simula ions. Ou obse a ions show ha i consumes be ween
40% and 60% o he sol e ime. Mo eo e , o highly aniso opic meshes, such
aniso opy deg ades he condi ioning o he linea sys em associa ed wi h he
Poisson equa ion.
Nume ical me hods o sol ing la ge linea sys ems can be classi ied in o
di ec and i e a i e sol e s. While di ec me hods p o ide exac solu ions,
hey a e imp ac ical o la ge-scale p oblems due o hei
O
(
n2
)compu a ional
and memo y complexi y. I e a i e me hods, such as K ylo subspace sol e s
and mul ig id me hods, a e mo e e icien o la ge sys ems. Howe e , hei
con e gence a e is highly dependen on he condi ioning o he sys em ma ix
and he choice o p econdi ione s.
P econdi ione s play a c ucial ole in accele a ing he con e gence o i e -
a i e sol e s by imp o ing he nume ical p ope ies o he sys em ma ix. In
CFD, he p esence o highly aniso opic meshes—pa icula ly in bounda y lay-
e s—leads o poo ly condi ioned linea sys ems, which slow down con e gence.
Specialized p econdi ione s a e equi ed o mi iga e hese issues and ensu e
e icien simula ions.
A c i ical challenge in CFD is modeling u bulence in he low. Tu -
bulence is cha ac e ized by chao ic luid mo ion in which a wide ange o
spa ial and empo al scales in e ac by mixing, hea ans e , and momen um
anspo . In high-Reynolds-numbe lows, u bulen luc ua ions span om
la ge ene gy-con aining eddies o small dissipa i e scales, making hei ull
esolu ion compu a ionally p ohibi i e. As he ene gy cascades down h ough a
hie a chy o scales, i is e en ually dissipa ed by iscous o ces a he smalles
scales, —known as he Kolmogo o scales. This phenomenon is called he
Kolmogo o cascade, which desc ibes how ene gy is ans e ed om la ge o
p og essi ely smalle scales. The complexi y o his phenomenon has mo i a ed
he de elopmen o a ious modeling s a egies ha aim o cap u e he essen ial
ea u es o u bulence wi hou esol ing e e y scale.
In p ac ical CFD simula ions, accu a ely cap u ing he complex na u e
o u bulence equi es adop ing modeling s a egies, each wi h p os and cons
ega ding ideli y and compu a ional cos . Di ec Nume ical Simula ion (DNS)
esol es all u bulen scales bu is limi ed o low Reynolds numbe s due o i s
high compu a ional cos . La ge Eddy Simula ion (LES), on he o he hand,
esol es he la ge-scale eddies while modeling he e ec s o smalle scales,
2
1.1. Mo i a ion
balancing ideli y and e iciency. Reynolds-A e aged Na ie -S okes (RANS)
me hods u he educe he compu a ional cos by a e aging he u bulen
luc ua ions. Addi ionally, LES o en employs wall modeling echniques o
accu a ely cap u e nea -wall beha io wi hou equi ing p ohibi i ely ine
meshes in hese egions.
These modeling choices di ec ly a ec he nume ical p ope ies o he
disc e ized equa ions. Fo ins ance, employing Wall-modelled La ge Eddy Simu-
la ion (WMLES) in high-Reynolds-numbe lows can lead o aniso opic meshes
wi hin he bounda y laye , which can exace ba e he ill-condi ioning o he
linea sys ems. This scena io unde sco es he need o obus p econdi ioning
s a egies capable o handling he s i ness in oduced by s eep g adien s and
s e ched elemen s.
A ela ed and equally c i ical challenge in luid simula ions is he accu a e
esolu ion o he bounda y laye , a hin egion nea solid su aces whe e
eloci y g adien s a e s eep. The physical cha ac e is ics o he bounda y laye
a e in luenced by he Reynolds numbe , which indica es whe he he low is
lamina (smoo h) o u bulen (chao ic). In his egion, iscosi y e ec s a e
highly signi ican , and he esolu ion o his egion equi es highly aniso opic
g ids. The low in his egion emains p edominan ly lamina o low Reynolds
numbe s. In con as , i ansi ions om lamina o u bulen a a ce ain
dis ance om he leading edge a high Reynolds numbe s.
Al hough lamina bounda y laye s gene a e less ic ion, hey a e uns able
and can quickly become u bulen . P ope ly simula ing he ansi ion om
lamina o u bulen low is c ucial o accu a ely p edic ing ae odynamic
pe o mance and minimizing ene gy losses caused by low sepa a ion.
In high Reynolds numbe lows, bounda y laye s a e subjec o signi ican
aniso opy due o s eep eloci y g adien s nea su aces, necessi a ing ine g id
esolu ion o cap u e hese e ec s accu a ely. This esul s in highly s e ched
elemen s in he mesh, which con ibu e o he s i ness and ill-condi ioning o
he sys em ma ix.
Fo such p oblems, s anda d p econdi ione s like diagonal, ILU, and mul-
ig id me hods may no pe o m op imally. I has been obse ed ha while
mul ig id me hods can educe he numbe o PCG i e a ions, hei o al
compu a ional cos emains compa able o ha o simple p econdi ione s [30].
Fu he mo e, hey s uggle wi h highly s e ched elemen s, making hem less
e ec i e o bounda y-laye -domina ed p oblems.
To add ess he challenges posed by aniso opic g ids, he linele p econdi-
ione was in oduced as a specialized app oach ha app oxima es he s onges
couplings in he sys em h ough a se o decoupled 1D p oblems. P e ious
s udies [23,28,1,4,12,18,20,9,21] ha e demons a ed i s e ec i eness,
showing signi ican imp o emen s o e con en ional p econdi ione s such as
diagonal, ILU, and mul ig id [23,28]. In pa icula , [28] epo ed ha he
linele p econdi ione educes he numbe o PCG i e a ions by a leas 50% and
dec eases o al CPU ime by up o 40% compa ed o diagonal [24], mul ig id
3
1.2. Objec i es and hesis s uc u e
[32], and ILU [24] p econdi ione s.
Howe e , adi ional implemen a ions impose a c i ical cons ain : linele s
mus be con ined wi hin a single subdomain, limi ing scalabili y in pa allel
compu a ions. Exis ing app oaches a emp o mi iga e his by modi ying
ei he he linele s uc u e o he pa i ioning algo i hm o ensu e ha each
linele emains con ained wi hin a subdomain [23]. This es ic ion, howe e ,
educes lexibili y and hinde s he p econdi ione ’s e iciency in la ge-scale simu-
la ions. Despi e he widesp ead use o linele -based p econdi ione s, no exis ing
sol e allows linele s o span mul iple p ocesses in a dis ibu ed compu ing
en i onmen .
This wo k add esses hese limi a ions by de eloping a pu ely algeb aic
linele p econdi ione ha ope a es di ec ly on he sys em ma ix, independen
o mesh geome y. This new app oach enables linele s o ex end ac oss mul iple
p ocesses, imp o ing scalabili y and pe o mance in pa allel compu ing en i -
onmen s. Unlike geome ic-based me hods, his pu ely algeb aic o mula ion
p ese es he key ad an ages o linele p econdi ioning while elimina ing he
cons ain s imposed by domain pa i ioning.
Finally, he inc easing complexi y o nume ical simula ions in CFD aligns
wi h b oade ends in High-Pe o mance Compu ing (HPC). While adi ional
pe o mance imp o emen s ha e ollowed Moo e’s law, u u e ad ancemen s
will depend on inno a ions a highe le els o he compu ing s ack, including
ha dwa e a chi ec u e, so wa e op imiza ion, and algo i hmic de elopmen .
This ansi ion equi es e icien nume ical sol e s ha can ully exploi mode n
compu ing capabili ies.
De eloping scalable, high-pe o mance p econdi ione s is essen ial o ad-
ancing la ge-scale CFD simula ions. By imp o ing sol e e iciency, his
wo k con ibu es o educing compu a ional cos s and enabling mo e de ailed
and accu a e simula ions, ul ima ely suppo ing ad ancemen s in ae ospace
enginee ing, ene gy sys ems, en i onmen al modeling, and o he c i ical ields.
1.2 Objec i es and hesis s uc u e
The p ima y objec i e o his hesis is o de elop an e icien p econdi ioning
s a egy o la ge-scale CFD simula ions, wi h a pa icula ocus on o e coming
he challenges posed by highly aniso opic meshes. Simula ing high-Reynolds-
numbe u bulen lows is equen ly hinde ed by he slow con e gence o
i e a i e Poisson sol e s, mainly due o he ill-condi ioning caused by hin
bounda y laye s and se e e mesh s e ching. Building on he wo k o So o,
Löhne , and Camelli [28], we ex end he concep o he linele p econdi ione
in o a global o mula ion. This Global Linele P econdi ione (GLP) is designed
o p ese e he in e domain couplings in dis ibu ed-memo y en i onmen s,
he eby accele a ing con e gence while emaining obus in massi ely pa allel
simula ions. Ou app oach in eg a es bo h algeb aic and geome ic pe spec i es:
he algeb aic o mula ion allows o applica ion wi hou di ec eliance on
4
1.3. Go e ning Equa ions
geome ic in o ma ion, while he geome ic cons uc ion akes in o accoun
di ec ional ea u es o he mesh o be e cap u e aniso opy.
In essence, he co e objec i e o his hesis can be ansla ed in o wo
decep i ely simple ye challenging ques ions. Fi s , how can one cons uc a
idiagonal ma ix ha is an e ec i e p econdi ione o he Poisson equa ion
in a dis ibu ed-memo y en i onmen while emaining agnos ic o he domain
pa i ion? Second, how can we e icien ly pa allelize he T idiagonal Ma ix
Algo i hm (TDMA) algo i hm when he ma ix en ies a e sp ead o e many
p ocesses? The answe s o hese ques ions lie a he co e o ou wo k. In pa -
icula , he solu ion o he i s ques ion gene alizes he cons uc ion app oach
p esen ed in [28] by inco po a ing in e domain couplings in addi ion o he
local ones.
The hesis is o ganized o in oduce he eade o he heo e ical ounda-
ions and hen o he p ac ical implemen a ion and alida ion o ou me hod.
Chap e 1 es ablishes he undamen al p inciples o luid mechanics and
he nume ical ea men o he Na ie –S okes equa ions, emphasizing he
compu a ional challenges associa ed wi h sol ing hese equa ions on highly
aniso opic meshes. Chap e 2 add esses he e icien pa alleliza ion o he
TDMA by inco po a ing he Schu complemen me hod, he chosen app oach
o main aining in e domain coupling ac oss di e en p ocesses. In Chap e 3,
we de ail he assembly p ocess o he p econdi ioning ma ix: local linele slices
a e cons uc ed wi hin each subdomain and hen joined ac oss subdomain
bounda ies o o m a cohe en global s uc u e. Chap e 4 ocuses on he
implemen a ion wi hin he Alya simula ion so wa e and p esen s an ex ensi e
pe o mance e alua ion, including analyses o scalabili y, load balance, and
communica ion e iciency. Finally, Chap e 5 applies he GLP p econdi ione
o eal-wo ld CFD scena ios— he 30P30N high-li ai oil, he S an o d di use ,
and he D i Ae model—demons a ing i s e ec i eness in p ac ical, la ge-scale
simula ions. Finally, he conclusion no only summa izes he achie emen s and
key con ibu ions o he wo k bu also discusses i s limi a ions and ou lines
di ec ions o u u e esea ch.
1.3 Go e ning Equa ions
This sec ion p esen s some o he equa ions and nume ical me hods usually
employed in CFD. Fi s , we in oduce he Na ie -S okes equa ions, which
desc ibe luid mo ion. Then, he ini e elemen me hod is in oduced o
spa ial disc e iza ion, p o iding he amewo k o app oxima ing he go e ning
equa ions on complex geome ies. Then, he ac ional s ep p ojec ion me hod
is used o p opaga e he solu ion in ime. This me hod decouples he p essu e
and eloci y calcula ions while main aining he incomp essibili y condi ion.
This me hod in ol es sol ing a Poisson equa ion o p essu e co ec ion a
each ime s ep. The esolu ion o his Poisson equa ion plays a c i ical ole in
main aining mass conse a ion and ensu ing con e gence.
Sol ing he p essu e Poisson equa ion becomes inc easingly challenging on
5
1.3. Go e ning Equa ions
highly aniso opic meshes, as hese can lead o s i ness and poo condi ioning
in he sys em ma ix. To add ess his issue, p econdi ione s a e used o imp o e
he con e gence a e o i e a i e sol e s by educing he condi ion numbe o
he ma ix, making he i e a i e sol e s mo e e icien when handling la ge
linea sys ems. This ensu es compu a ional e iciency and enables he e ec i e
esolu ion o such sys ems, which is a cen al ocus o his s udy.
1.3.1 Re e ence sys ems
Le us in oduce he heo e ical amewo k by b ie ly discussing he e e ence
sys ems commonly used in luid mechanics. Indeed, be o e a emp ing o
desc ibe he physics o a gi en luid, i is necessa y o de ine he e e ence
sys em—o poin o iew— ha will be used o desc ibe i s dynamics. In his
ega d, h ee ypes o e e ence sys ems a e commonly used in luid mechanics.
Fi s , we ha e he Lag angian e e ence sys em (also known as he cu en o
ma e ial desc ip ion), which we will deno e as
RL
. In his sys em, he obse e
ollows a speci ic luid pa icle as i mo es h ough ime and space. Thus, he
posi ion and eloci y o a pa icle ha a a gi en ime
0
is loca ed a posi ion
x0will be gi en by
x=x( , x0)
U( , x0) = ∂x( , x0)
∂
(1.1)
The es o he ele an quan i ies a e desc ibed in he same way, such as he
olume di e en ial
δV
(
0,x0
)o he luid elemen ha a ime
0
is in he su -
oundings o
x0
, i s mass
δM
(
0,x0
), densi y
ρ
(
0,x0
) =
δM
(
0,x0
)
/δV
(
0,x0
),
empe a u e
T
(
0,x0
), e c. Thus, om his pe spec i e, desc ibing a pa icula
p ope y o a con inuous medium in ol es exp essing i as a unc ion o all
ini ial posi ions x0and all equi ed ime ins an s :A( 0,x0).
On he o he hand, he Eule ian desc ip ion (also known as spa ial con igu -
a ion) is ca ied ou in a e e ence sys em ixed o he labo a o y, which we
will deno e as
RE
. Unlike he Lag angian desc ip ion, whe e a luid pa icle
is ollowed in i s mo ion and i s luid p ope ies a e desc ibed in e ms o he
p ope y o he pa icle a ime
0
loca ed a posi ion
x0
, in he Eule ian
desc ip ion, we desc ibe he p ope y o he luid pa icle ha is loca ed a
posi ion
x
a ime
. This will, he e o e, be a ield desc ip ion o he luid. Thus,
he eloci y ield will be gi en by
U
(
, x
), he densi y by
ρ
(
, x
), he empe -
a u e by
T
(
, x
), and simila ly, o any o he p ope y de ined on he luid
a
(
, x
).
Finally, he e is a hi d op ion, ALE ( e e en ial con igu a ion), which
we will deno e as
RA
. In his sys em, he luid p ope ies a e desc ibed in
an independen e e ence sys em, ypically ixed o he compu a ional g id.
This is pa icula ly ele an when wo king wi h mesh dis o ions, as i allows
o p ese ing an accu a e desc ip ion o in e aces unde signi ican mesh
dis o ions.
6
1.3. Go e ning Equa ions
Since he h ee ep esen a ions a e di e en ways o isualizing he same
physical phenomenon, i is possible o ansi ion om one o ano he h ough
he app op ia e ans o ma ions. Thus, we can conside he amily o in e ible
mappings de ined by
φ:RE×[ 0, end)→RA×[ 0, end)
Ψ : RA×[ 0, end)→RL×[ 0, end)
Φ : RL×[ 0, end)→RE×[ 0, end)
(1.2)
Figu e 1.1: Eule ian, Lag angian, and ALE e e ence sys ems and hei
ans o ma ions.
Since he h ee desc ip ions a e equi alen , i , o example, he Lag angian
desc ip ion o a luid p ope y is known, he Eule ian desc ip ion will be gi en
by:
a( , x) = A( , x0( , x)),(1.3)
whe e x0( , x)is he equa ion ha esul s om sol ing o x0in he equa ion
x( , x0) = x,(1.4)
which ela es he o iginal posi ion
x0
ha a gi en pa icle had a ime
0
wi h
he posi ion i occupies a ime
. Due o pa icle iden i y, he p e ious ela ion
will always be bijec i e and, he e o e, in e ible.
Simila ly, gi en he Eule ian desc ip ion o a medium, he co esponding
Lag angian desc ip ion will be gi en by
A( , x0) = a( , x( , x0)),(1.5)
whe e he ollowing equa ion mus be sol ed be o ehand
x( , x0) = x0+Z
0
u(τ, x(τ, x0))dτ, (1.6)
which esul s om in eg a ing he equa ion
7
1.3. Go e ning Equa ions
∂x( , x0)
∂ =U( , x0) = u( , x0).(1.7)
Gi en ha we will no wo k wi h mesh mo emen in his hesis, he ALE
desc ip ion will coincide wi h he Eule ian one. We will use he la e h oughou
his wo k.
1.3.2 Fluid mechanics
Since his wo k aims o imp o e he e iciency o sol e s o he Poisson equa ion,
we conside i impo an o p esen a de ailed backg ound and he ole he
Poisson equa ion plays in CFD. Fo ha , we p esen he e he comple e se
o Na ie -S okes equa ions. This will allow us o unde s and no only he
disc e iza ion me hods used bu also he ole ha he Poisson equa ion plays
in hei con ex and he impo ance o ha ing as and e icien me hods o i s
solu ion. We will also conside he case o a New onian incomp essible luid.
This app oxima ion, as we will see, is alid o low Mach numbe lows, ha is,
lows in which he eloci y o he luid is much smalle han he speed o sound.
In his sec ion, we o mula e he Na ie -S okes equa ions in he Eule ian ame
o e e ence.
Comple e Se o Equa ions
The Na ie -S okes equa ions desc ibe he mo ion o luid subs ances unde he
in luence o a ious o ces. Fo an incomp essible luid mo ing in a domain Ω
bounded by Γ =
∂
Ωo e he ime in e al (
0,
), he go e ning equa ions a e
gi en by
ρ∂ ui+ρuj∂jui+∂ip−∂j(µ(∂jui+∂iuj)) = ion Ω×( 0, ),(1.8a)
∂juj= 0 on Ω×( 0, ).(1.8b)
o
i, j
= 1
, ..., d
, wi h
d
being he dimension o he domain. He e,
ui
is he
eloci y ield,
p
is he p essu e,
ρ
is he luid densi y,
µ
is he dynamic iscosi y,
and
i
is he momen um sou ce. The i s equa ion in
(1.8)
ep esen s he
conse a ion o momen um, while he second equa ion en o ces incomp essibili y.
To ully desc ibe he luid mo ion, he sys em in
(1.8)
mus be supplemen ed
wi h app op ia e bounda y condi ions o e Ω
×
(
0,
). De ining Γ =
∂
Ω, we
dis inguish be ween Di ichle condi ions, which p esc ibe eloci y, and Neumann
condi ions, which p esc ibe ac ion. Le Γ
D
and Γ
N
be he po ions o he
bounda y whe e hese condi ions apply, sa is ying Γ
D∪
Γ
N
= Γ and Γ
D∩
Γ
N
=
∅
.
The bounda y condi ions a e gi en by
ui=uD
ion ΓD×( 0, ),(1.9a)
σijnj= N
ion ΓN×( 0, ),(1.9b)
whe e
nj
is he uni no mal o he bounda y Γ
N
, and
σij
is he Cauchy s ess
enso :
8
1.4. Disc e iza ion o he Na ie -S okes equa ions
σij =−pδij +µ(∂iuj+∂jui).(1.10)
He e, N
i ep esen s he imposed ac ion a he Neumann bounda ies.
Since he equa ions in
(1.8)
con ain i s -o de empo al de i a i es o
eloci y, an ini ial condi ion mus also be imposed:
ui(x, 0) = u0
i(x)on Ω.(1.11)
I he luid densi y
ρ
and iscosi y coe icien
µ
a e assumed cons an ,
in oducing he kinema ic iscosi y
ν
=
µ/ρ
and ede ining
p
o be he p essu e
pe uni densi y (p→p/ρ), he sys em simpli ies o
∂ ui+uj∂jui+∂ip−ν∂j(∂jui+∂iuj) = ion Ω×( 0, ),(1.12a)
∂juj= 0 on Ω×( 0, ),(1.12b)
ui=uD
ion ΓD×( 0, ),(1.12c)
σijnj= N
ion ΓN×( 0, ),(1.12d)
ui(x, 0) = u0
i(x)on Ω× { 0}.(1.12e)
o i, j = 1, ..., d.
1.4 Disc e iza ion o he Na ie -S okes equa ions
This sec ion ocuses on he disc e iza ion o he Na ie -S okes equa ions, which
con e s hem om a con inuous ma hema ical model in o a o m ha can be
sol ed using compu a ional me hods. We begin by explo ing a ious spa ial
disc e iza ion echniques sui ed o di e en p oblem cha ac e is ics. Among he
mos common me hods, we chose he FEM, which is well-sui ed o complex
geome ies.
Nex , we in oduce he weak o mula ion o he Na ie -S okes equa ions,
a less es ic i e app oach han he s ong o m. Thus enabling solu ions in
b oade unc ion spaces. This ounda ion allows he applica ion o he FEM
o sol e he equa ions, including ime-dependen p oblems, by inco po a ing
sui able unc ion spaces o empo al solu ions. In FEM, he solu ion is
app oxima ed as a linea combina ion o shape unc ions de ined on he mesh
nodes, educing he p oblem o de e mining he nodal unknowns. We hen
mo e in o he ime disc e iza ion p ocess o he Na ie -S okes equa ions. The
disc e ized o m o he equa ions, inco po a ing bo h eloci y and p essu e, is
in oduced nex . The goal is o ind
Un+1
and
Pn+1
, which a e he nodal alues
o eloci y and p essu e a he subsequen ime s ep. The sys em o equa ions
is ou lined, se ing he s age o he ac ional s ep schemes. A his poin ,
ac ional s ep schemes a e p esen ed as an app oach o decouple eloci y and
p essu e in he ime disc e iza ion. These schemes simpli y he p oblem in o
sepa a e s eps, making he solu ion p ocess mo e e icien . The sys em is sol ed
in s ages, wi h he momen um ad anced explici ly and a Poisson equa ion sol ed
o p essu e s abiliza ion. The ac ional s ep me hod leads o a Poisson-like
equa ion o p essu e, which equi es he solu ion o a linea sys em o he o m
9
1.4. Disc e iza ion o he Na ie -S okes equa ions
L
∆
P
=
b
, whe e
L
app oxima es he Laplacian ope a o . This s ep is one o
he mos ime-consuming and challenging o pa allelize, and hus, achie ing a
solu ion is expensi e in ime and compu a ional esou ces: sol ing i e icien ly
is c ucial o he pe o mance o he nume ical me hod. Finally, we highligh
how elemen dimensions a ec he o e all ma ix Lcon ibu ions.
1.4.1 Space Disc e iza ion
The choice o disc e iza ion me hod depends on he complexi y o he p oblem,
he geome y o he domain, and he desi ed p ope ies o he solu ion. The
Fini e Elemen s Me hod (FEM) cons uc s local app oxima ions o he solu ion
using local da a, assembling hem in o a global app oxima ion. One o i s main
ad an ages is i s abili y o handle complex, i egula geome ies due o i s
use o lexible, uns uc u ed meshes ha con o m well o in ica e bounda ies.
Unlike he Fini e Di e ence Me hod (FDM), which elies on s uc u ed g ids
and is less sui ed o i egula domains, FEM allows o local e inemen and
adap i i y, making i pa icula ly use ul in egions wi h s eep g adien s o high
solu ion a iabili y.
On he o he hand, he Fini e Volumes Me hod (FVM) e alua es exac
exp essions o he a e age alue o he solu ion o e a con ol olume. I de ines
he solu ion as cell-a e aged alues, equi ing econs uc ion o in e pola ion
me hods o app oxima e alues a bounda ies, ypically using neighbo ing cell
da a. Unlike FEM, which suppo s highe -o de basis unc ions wi hou he
need o econs uc ion, FVM en o ces local lux conse a ion, ensu ing ha
he ne lux ac oss each con ol olume is balanced. In con as , FEM does no
gua an ee local conse a ion bu ensu es global conse a ion o luxes.
Gi en i s widesp ead adop ion in indus y and i s sui abili y o handling
mul iphysics p oblems, his wo k ocuses on he FEM o spa ial disc e iza ion.
Howe e , i is impo an o no e ha he me hods de eloped in his hesis can
be ex ended o o he disc e iza ion app oaches.
Weak Fo mula ion
Be o e applying he FEM o he Na ie -S okes equa ions, i is necessa y o
in oduce he weak o mula ion o a PDE. This o mula ion is he basis o he
Fini e Elemen Me hod. The main ad an age o his app oach is ha i is less
es ic i e and elaxes some o he o iginal PDE cons ain s.
Conside a di e en ial equa ion o a unc ion u(x)in a domain Ω:
L {u(x)}= (x),(1.13)
wi h he app op ia e bounda y condi ions. Equa ion
(1.13)
is known as he
s ong o m o he PDE.
We de ine he unc ion space
V
=
H1
(Ω) as he space o unc ions
such
ha
∈L2
(Ω) and
∂i ∈L2
(Ω), whe e
L2
(Ω) is he space o squa e-in eg able
unc ions, and
H1
(Ω) consis s o unc ions whose i s de i a i es also belong
o
L2
(Ω). Mul iplying equa ion
(1.13)
by a es unc ion
∈V
and in eg a ing
10
1.5. Me hods o Sol ing Linea Sys ems
he p ojec ion s ep consis s o sol ing a Poisson equa ion o he p essu e
s abiliza ion [7]. This means ha a his s ep, a linea sys em o he o m
L∆P=b, (1.36)
needs o be sol ed, whe e ma ix
L
is he app oxima ion o he Laplacian
ope a o , and ∆
P
is he inc emen in p essu e (which is he unknown o be
sol ed o ). No e also ha only he igh -hand side ( .h.s.)
b
is a ec ed by he
p ope ies o he low, whe eas
L
depends only on he p ope ies o he mesh.
This associa ed algeb aic sys em is he implici pa o he ime s ep and is one
o he mos ime-consuming and di icul o pa allelize. The e o e, choosing
a sui able nume ical me hod is c ucial o ob aining a solu ion wi hou being
expensi e in e ms o compu a ional esou ces. This ma ix plays a undamen al
ole in many applica ions, such as hea ans e , s uc u al mechanics, and
compu a ional luid dynamics. Hence, i is impo an o ha e an e icien
me hod o compu e and sol e i .
In he ollowing sec ions o his wo k, we will exp ess equa ion
(1.36)
in he
o m
Ax
=
b
, whe e
A
ep esen s he ma ix ha app oxima es he Laplacian
ope a o and
x
deno es he p essu e inc emen . This no a ion helps s eamline
ou p esen a ion and highligh s ha he p oposed me hod is applicable o
sol ing equa ion
(1.36)
and any linea sys em ha ollows he linele s uc u e,
which will be desc ibed in he subsequen sec ions.
1.5 Me hods o Sol ing Linea Sys ems
The solu ion o la ge linea sys ems can be add essed using ei he di ec
o i e a i e echniques. Di ec me hods, such as Gaussian elimina ion and
ac o iza ion (e.g., LU o Cholesky), heo e ically p o ide exac solu ions a e
a ini e numbe o ope a ions. Howe e , hey a e imp ac ical o la ge-scale
p oblems due o hei high compu a ional and memo y demands. Speci ically,
di ec sol e s equi e
O
(
n3
)a i hme ic ope a ions and
O
(
n2
)memo y s o age,
whe e
n
is he numbe o unknowns [11,17]. These equi emen s make di ec
me hods unsui able o la ge-scale applica ions. Addi ionally, loa ing-poin
e o s can accumula e, p e en ing exac solu ions in p ac ice.
On he o he hand, I e a i e me hods a e pa icula ly ad an ageous o
spa se sys ems, whe e he scalabili y issues o di ec me hods become p ohibi i e
as he p oblem size
n
g ows. I e a i e sol e s gene ally equi e only
O
(
n
)
a i hme ic ope a ions and
O
(
n
)memo y, making hem p e e able o la ge
p oblems. These me hods s a om an ini ial guess and e ine i i e a i ely
un il he esidual alls below a p ede ined h eshold, ensu ing su icien accu acy
while a oiding unnecessa y compu a ions. The con e gence o i e a i e me hods
depends on he condi ioning o he ma ix and he quali y o he ini ial guess,
and hey a e widely used in p ac ice due o hei adap abili y and e iciency.
Among i e a i e echniques, h ee main ca ego ies exis : s a iona y me hods,
mul ig id me hods [3,31], and K ylo subspace me hods (e.g., Conjuga e
G adien , GMRES) [27,24]. K ylo me hods app oxima e he solu ion wi hin
17
1.5. Me hods o Sol ing Linea Sys ems
a subspace gene a ed by he ma ix and he ini ial guess, while mul ig id
me hods le e age a hie a chy o disc e iza ions o add ess mul iple scales o
beha io . While mul ig id me hods can be highly e ec i e, hey o en equi e
applica ion-speci ic uning and can ace challenges when applied o highly
aniso opic bounda y laye s. On he o he hand, s a iona y me hods wo k by
inding he ixed poin o a p ede ined mapping.
1.5.1 Di ec Me hods
Di ec me hods sol e a linea sys em in a ini e numbe o s eps. Despi e hei
high compu a ional cos and memo y demands, hey can some imes ou pe o m
i e a i e sol e s o small p oblems [29]. Di ec me hods a e gene ally classi ied
in o elimina ion me hods, such as Gaussian elimina ion and ac o iza ion
me hods, including LU and Cholesky ac o iza ions.
LU decomposi ion ac o izes he linea sys em ma ix
A
in o he p oduc
o a lowe iangula ma ix
L
and an uppe iangula ma ix
U
, p o ided
his decomposi ion exis s. The sys em
Ax
=
b
can hen be sol ed in wo s eps:
Ly
=
b
ollowed by
Ux
=
y
. The Cholesky decomposi ion is a pa icula case
o LU decomposi ion o symme ic and posi i e-de ini e ma ices, ac o izing
A
as
A
=
LLT
, whe e
L
is a lowe iangula ma ix. Sol ing
Ax
=
b
wi h
Cholesky equi es sol ing
Ly
=
b
and hen
LTx
=
y
. Cholesky decomposi ion
is gene ally mo e e icien han LU decomposi ion, equi ing ewe a i hme ic
ope a ions and less memo y [29].
1.5.2 Mul ig id Me hods
Mul ig id me hods ope a e on a hie a chy o g ids, e ining e o s on di e en
scales o accele a e con e gence. On ine g ids, i e a i e sol e s like Gauss-
Seidel o Jacobi a e used o educe high- equency e o s (smoo hing), while
low- equency e o s a e add essed by ans e ing he p oblem o coa se g ids.
G id ans e ope a ions include es ic ion (coa se-g id ep esen a ion) and
p olonga ion ( ine-g id in e pola ion). These me hods ollow s uc u ed cycles,
such as he V-cycle o W-cycle, balancing compu a ional e iciency and e o
educ ion. Howe e , hei pe o mance can deg ade o highly aniso opic
p oblems, o en equi ing specialized smoo he s [3,31].
1.5.3 I e a i e Me hods
I e a i e me hods equi e less compu a ional esou ces and memo y han di ec
me hods bu can su e om slow con e gence o di e gence when dealing wi h
ill-condi ioned ma ices. P econdi ioning s a egies a e equen ly applied o
mi iga e hese issues and imp o e sol e e iciency [2,24].
Gi en a linea sys em Ax =b, i s esidual (x)is de ined as:
(x) := b−Ax.
18
1.5. Me hods o Sol ing Linea Sys ems
Mos i e a i e me hods a emp o app oach he exac solu ion
¯
x
by gene -
a ing a sequence o app oxima ions ha g adually educe
(x)
o ze o. Howe e ,
due o loa ing-poin e o s, eaching he exac solu ion is o en imp ac ical,
and a p ede ined ole ance le el is usually employed o de e mine when o s op
he i e a ions.
The e iciency o i e a i e sol e s depends on he p ope ies o he sys em
ma ix and he choice o p econdi ione s, which will be explo ed in he ollowing
sec ions.
K ylo Me hods
K ylo me hods cons uc a sequence o app oxima ions wi hin a K ylo
subspace:
Kn( 0) = span{ 0, A 0, A2 0, . . . , An−1 0},
whe e
0
=
Ax0−b
is he ini ial esidual. These me hods, including Con-
juga e G adien (CG) and GMRES, minimize he esidual o e his subspace,
o en achie ing accep able accu acy in a ewe i e a ions han he sys em size
N
. CG is one o he mos e ec i e sol e s o symme ic and posi i e-de ini e
ma ices, hough i s e iciency s ongly depends on p econdi ioning [2,24].
In compu a ional luid dynamics, highly aniso opic meshes induce s i ness,
leading o ill-condi ioned ma ices. This s i ness slows down con e gence
signi ican ly, necessi a ing obus p econdi ione s.
Conjuga e G adien Me hod
The CG me hod sol es
Ax
=
b
o symme ic posi i e-de ini e
A
by minimizing
he quad a ic unc ion:
(x) = 1
2xTAx −xTb, (1.37)
which has a global minimum a he exac solu ion. Ins ead o simple s eep-
es descen , CG uses mu ually independen (A-conjuga e) sea ch di ec ions,
educing he numbe o necessa y i e a ions.
The CG algo i hm p oceeds as ollows:
19
1.5. Me hods o Sol ing Linea Sys ems
Algo i hm 1 Conjuga e G adien Me hod
1: =b−Ax0
2: d=
3: δnew = T
4: δ0=δnew
5: while i<imax and || ||2> ϵ2do
6: q=Ad
7: α=δnew/(dTq)
8: x=x+αd
9: i imod 50 = 0 hen
10: =b−Ax
11: else
12: = −αq
13: δold =δnew
14: δnew = T
15: β=δnew/δold
16: d= +βd
17: i=i+ 1
This me hod e icien ly con e ges in a mos
N
i e a ions bu o en achie es a
sa is ac o y solu ion in signi ican ly ewe s eps. The ou pu c i e ion
|| ||2> ϵ2
is used o de e mine when o s op he i e a ions, whe e
ϵ
is a p ede ined
ole ance le el. Ano he common choice is o s op when he ela i e esidual is
below a ce ain h eshold, e.g., δnew/||b||< ϵ.
P econdi ioned Conjuga e G adien Me hod
When he sys em ma ix
A
is ill-condi ioned, as o en occu s in he disc e iz-
a ion o he Laplacian ope a o on highly aniso opic meshes, he nume ical
s i ness o he p oblem inc eases, signi ican ly deg ading he con e gence a e
o i e a i e sol e s. P econdi ioning echniques can be applied o mi iga e his
issue. A p econdi ione is a ma ix ha modi ies he o iginal linea sys em
o imp o e nume ical s abili y and con e gence speed. I achie es his by
app oxima ing he sys em ma ix in a way ha educes i s condi ion numbe
and imp o es eigen alue clus e ing, hus making i e a i e sol e s mo e e icien
[26,16].
The e a e se e al ypes o p econdi ione s, classi ied based on hei
cons uc ion and applica ion [2,24]:
•
Le p econdi ioning: This echnique modi ies he ope a o applied o
x
, sol ing he sys em
M−1Ax
=
M−1b
. This ans o ma ion imp o es
spec al p ope ies bu equi es ca e ul ea men o con e gence c i e ia.
The i ial choice
M
=
A
yields a condi ion numbe o one and con e gence
in a single i e a ion, bu sol ing
M−1Ax
would be as cos ly as sol ing
Ax =bdi ec ly.
20
1.5. Me hods o Sol ing Linea Sys ems
•
Righ p econdi ioning: This echnique ans o ms he solu ion ec o
di ec ly by sol ing
AM−1y
=
b
and hen eco e ing
x
ia
Mx
=
y
. This
app oach ensu es ha he ans o med sys em e ains he same eigen alues
as he o iginal.
•
Mixed p econdi ioning: This echnique in ol es a wo-sided ans o m-
a ion
M−1
LAM−1
Ry
=
M−1
Lb
, ollowed by sol ing
MRx
=
y
. I combines
elemen s o bo h le and igh p econdi ioning o achie e be e condi-
ioning.
The choice o he mos sui able p econdi ioning echnique is highly p oblem-
dependen . Fo non-symme ic ma ices, me hods like GMRES a e mo e
app op ia e [8,25,24], whe eas, o symme ic ma ices, he Conjuga e G adi-
en (CG) me hod is o en he bes choice.
P econdi ioning mus p ese e symme y o he CG me hod, which equi es
a symme ic sys em ma ix. The nai e app oach o applying
M−1A
does no
necessa ily main ain symme y, e en i
M
and
A
a e symme ic. Ins ead, we
in oduce a symme ic ac o iza ion. Since
M
is symme ic and posi i e-de ini e,
he e exis s a ma ix
E
such ha
M
=
EET
. Using his, we ans o m he
o iginal sys em:
E−1AE−Tˆx=E−1b, ˆx=ETx.
The ans o med sys em e ains symme y and posi i e de ini eness, allowing
i o be sol ed e icien ly wi h he CG me hod. No ably, explici compu a ion o
E
is unnecessa y; i su ices o e icien ly compu e he ac ion o
M−1
on a ec o .
The pe o mance o a p econdi ione
M
depends on how well i educes he
condi ion numbe o
M−1A
and clus e s i s eigen alues. The challenge is se-
lec ing a p econdi ione ha balances e ec i eness and compu a ional e iciency.
Thus, he P econdi ioned Conjuga e G adien (PCG) me hod in ol es he
ollowing i e a i e s eps:
21
1.5. Me hods o Sol ing Linea Sys ems
Algo i hm 2 P econdi ioned Conjuga e G adien (PCG) Algo i hm
1: i= 0
2: =b−Ax0
3: d=M−1
4: δnew = T
5: δ0=δnew
6: while i<imax and || ||2> ϵ2do
7: q=Ad
8: α=δnew/(dTq)
9: x=x+αd
10: i imod 50 = 0 hen
11: =b−Ax
12: else
13: = −αq
14: s=M−1
15: δold =δnew
16: δnew = Ts
17: β=δnew/δold
18: d=s+βd
19: i=i+ 1
1.5.4 Di e en ypes o p econdi ione s
The simples p econdi ione is he Jacobi p econdi ione , also known as diagonal
p econdi ioning. I uses only he diagonal en ies o he sys em ma ix
A
o
cons uc a diagonal ma ix
M
, which is hen used as he p econdi ione . This
me hod, which e ec i ely scales he quad a ic o m
(1.37)
along he coo dina e
axes, is compu a ionally inexpensi e and easy o implemen , as he in e sion
o a diagonal ma ix is i ial. Howe e , i s e ec i eness is o en limi ed o
well-condi ioned o diagonally dominan ma ices, making i less sui able o
sys ems wi h s ong aniso opies o la ge condi ion numbe s.
A mo e ad anced class o p econdi ione s elies on incomple e ac o iza ions,
such as Incomple e Cholesky (IC) and Incomple e LU (ILU). The Incomple e
Cholesky p econdi ione is used o symme ic posi i e-de ini e ma ices and
app oxima es
A
by ac o ing i in o
LLT
, whe e
L
is a lowe iangula ma ix.
Unlike ull Cholesky ac o iza ion, IC limi s ill-in o p ese e he spa si y
pa e n o
A
, educing s o age and compu a ional cos s. The p econdi ione
is applied indi ec ly by sol ing
LLTw
=
z
using back-subs i u ion. While IC
is o en e ec i e, i can su e om nume ical ins abili ies, pa icula ly o
ill-condi ioned o inde ini e ma ices.
Simila ly, LU decomposi ion and i s incomple e a ian , Incomple e LU
(ILU), a e widely used p econdi ione s, pa icula ly o non-symme ic ma ices.
LU decomposi ion ac o s
A
in o he p oduc o a lowe iangula ma ix
L
and an uppe iangula ma ix
U
. Howe e , di ec LU ac o iza ion in oduces
signi ican ill-in, inc easing memo y and compu a ional cos s. The ILU me hod
alle ia es his by allowing only a con olled amoun o ill-ins, p oducing an
app oxima ion
A≈LU
. While sol ing
LUx
=
b
is e icien , ILU does no
22
1.6. Linele p econdi ione
di ec ly yield he solu ion o
Ax
=
b
. Ins ead, he p econdi ione
M
=
LU
is
used wi hin an i e a i e sol e such as he conjuga e g adien me hod. Despi e
hei widesp ead use, IC and ILU may equi e uning pa ame e s, such as d op
ole ance, o con ol spa si y and main ain nume ical s abili y.
Mul ig id me hods, ano he p ominen class o p econdi ione s, add ess
e o componen s a di e en spa ial esolu ions by employing a hie a chy o
coa se g ids o e icien ly elimina e low- equency e o s, while ine g ids
e ine high- equency componen s. These highly scalable me hods can achie e
nea -op imal con e gence a es o ellip ic and pa abolic p oblems, making hem
a widely used app oach in CFD applica ions. Howe e , hei implemen a ion
can be pa icula ly challenging o uns uc u ed g ids o highly aniso opic
domains, whe e con e gence may be signi ican ly deg aded o e en lead o
sol e di e gence.
Algeb aic Mul ig id (AMG), a a ian o mul ig id ha builds coa se g ids
algeb aically a he han geome ically, is also widely used in CFD due o i s
adap abili y o complex geome ies and uns uc u ed meshes.
O he ad anced p econdi ione s include Schwa z me hods, which pa i ion
he domain in o o e lapping o non-o e lapping subdomains and sol e smalle
subp oblems locally o cons uc a global p econdi ione .
Spa se App oxima e In e se (SPAI) p econdi ione s a e a class o p econdi-
ione s ha app oxima e he in e se o he sys em ma ix
A
by cons uc ing
a spa se ma ix
M
, i.e.,
M≈A−1
, ha e ains he spa si y pa e n o
A
.
This allows o e icien compu a ion while imp o ing he con e gence a e o
i e a i e sol e s like Conjuga e G adien (CG). Howe e , SPAI p econdi ione s
may no always p o ide signi ican imp o emen s i he o iginal ma ix
A
is
ill-condi ioned. In such cases, addi ional echniques may be needed o imp o e
he s abili y and e iciency o he p econdi ione .
I is also wo h men ioning ha he choice o he p econdi ione is highly
p oblem-dependen . A p econdi ione ha pe o ms well in one case migh
pe o m poo ly in ano he , whe eas a di e en p econdi ione could be mo e
e ec i e. Also, a sui able p econdi ione mus balance compu a ional cos ,
memo y equi emen s, and con e gence pe o mance o ensu e compu a ional
e iciency and good scalabili y. The e o e, selec ing an app op ia e p econdi-
ione o i e a i e sol e s is as c i ical as choosing he sol e i sel .
1.6 Linele p econdi ione
Among he wide a ie y o such p econdi ione s, he p e e ed op ion o wo k
wi h in cases o high mesh aniso opy is he linele p econdi ione [13,18,19].
Ano he kind o linele p econdi ione is he s eamline linele p econdi ione ,
de eloped in p e ious wo ks [8]. Ne e heless, we ind he o me mo e sui able
o he p esen wo k. In his me hod, he p econdi ioning ma ix
M
is buil
by il e ing ou some o he coe icien s o ma ix
A
. B ie ly, he main idea
23
1.6. Linele p econdi ione
(a) Ske ch o di e en linele s buil in an
uns uc u ed mesh.
(b) T idiagonal s uc u e o he
linele p econdi ioning ma ix
o he case shown in Figu e
1.2a.
Figu e 1.2: Linele s uc u e de i ed om he s onges couplings in he mesh
and he idiagonal ma ix ob ained by exclusi ely conside ing hese dominan
connec ions.
consis s o building lines ollowing he di ec ion o he s onges couplings: he
so-called linele s. These s uc u es will only include nodes whose couplings a e
s onge han a ce ain ole ance. Also, a node can be pa o a mos one
linele , depending on he s eng h o i s couplings: i he couplings a e s ong
enough, he node will be included in a linele ; o he wise, i will no , bu he
same node will ne e be pa o wo o mo e linele s. Then, he p econdi ioning
ma ix is buil by assembling he diagonal en ies o he sys em ma ix (
Aii
)
and he non-diagonal en ies (
Aij
,
i=j
) o he edges belonging o hese lines
o high couplings, any o he coupling no included in any linele is disca ded.
This means ha o he mesh nodes ha do no belong o any linele , only he
diagonal en y is p ese ed in he p econdi ioning ma ix. In o he wo ds, he
diagonal p econdi ione is applied [14] o nodes no included in any linele .
Finally, i hese poin s a e enumbe ed ollowing he linele s, he p econdi ioning
ma ix associa ed is block- idiagonal. This me hod is pa icula ly use ul when
ill-condi ioning comes om a high aniso opy in he mesh.
Schema ically, he linele p econdi ioning ma ix is de ined as ollows:
Mii =Aii i.e., Mhas he same diagonal as A
Mij =Aij i=ji i, j belongs o a linele
Mij = 0 i=ji i, j does no belong o a linele .
(1.38)
A e a nodal enume a ion ollowing he linele s, he s uc u e o
M
consis s
o se s o decoupled idiagonal ma ices, each associa ed wi h a linele . As
s a ed abo e, only he diagonal en y emains o nodes no associa ed wi h
any linele . Figu e 1.2a shows an example g id whe e he s onges couplings
a e shown in ed, exposing he linele s uc u e o couplings. The s uc u e o
he co esponding p econdi ione Mis shown in Figu e 1.2b.
Gi en ha he se o p econdi ioning equa ions can be w i en as
s=M−1 , (1.39)
24
1.6. Linele p econdi ione
whe e
is he esidual and
M
he p econdi ioning ma ix whose idiagonal
s uc u e is shown in Figu e 1.2b, we need an e icien way o compu e
M−1
.
The se o decoupled idiagonal equa ions in
M
allows a as solu ion o he
p econdi ioning equa ions in PCG by a di ec me hod, whe e one idiagonal
sys em needs o be sol ed o each linele . Hence, he T idiagonal Ma ix Al-
go i hm (TDMA) (also known as he Thomas Algo i hm) is sui able o his s ep.
I is impo an o no e ha he linele p econdi ione can be cons uc ed
ei he algeb aically by di ec ly inspec ing he linea sys em ma ix
A
, as
p e iously men ioned, o geome ically. In he algeb aic app oach, linele s
a e o med by ollowing he di ec ion o he s onges couplings in ma ix
A
.
Con e sely, in he geome ic app oach, linele s a e cons uc ed by connec ing
he closes nodes. This is analogous o he p ocedu e desc ibed in
(1.38)
, bu
ins ead o analyzing he linea sys em ma ix
A
, i elies on inspec ing he
ma ix de ined as
Dij =(1
||xi−xj||2,i=j
0,i=j(1.40)
whe e x
i
and x
j
ep esen he coo dina es o nodes
i
and
j
, espec i ely. He e,
Dij
co esponds o he in e se o he dis ance be ween nodes
i
and
j
. Once he
linele s a e cons uc ed, he en ies used o o m he p econdi ioning ma ix
M
a e ex ac ed om he linea sys em ma ix A.
1.6.1 Ca ac e iza ion o he linele p econdi ione
In ou expe ience, he linele p econdi ione , designed o exploi he mesh
aniso opy in i s a o , has demons a ed imp o ed pe o mance wi h inc easing
aspec a ios. Speci ically, a highe aspec a io co ela ed wi h a educed
numbe o i e a ions equi ed o con e gence, indica ing as e con e gence
a es (we de ine he aspec a io as he geome ical aspec a io, ep esen ing
he a io o he maximum o minimum leng h o a gi en elemen ). Consequen ly,
he p esen ed esul s a e pa icula ly ad an ageous o scena ios cha ac e ized
by highe aspec a ios.
Be o e analyzing he con e gence o he p oposed me hod in de ail, le us
i s examine he e ec o aspec a io a ia ion on he con e gence o he linele
p econdi ione . We conside wo meshes: i s , a hyb id mesh wi h 0.49 million
elemen s, whe e 49% a e wi hin he bounda y laye , and second, a s uc u ed
mesh wi h 0.5 million elemen s, whe e 100% a e wi hin he bounda y laye .
In his con ex , we a ied he maximum aspec a io o he bounda y laye ,
main aining a p og ession o 1.01, and compa ed he numbe o i e a ions he
sol e equi es o each a ela i e esidual o 10
−6
. To isola e he e ec o he
maximum aspec a io on con e gence, we an he case sequen ially and wi hou
domain pa i ioning; in his case, ou new app oach o he linele p econdi ione ,
which we call Global Linele P econdi ione (GLP), is equi alen o p e ious
ones Local Linele P econdi ione (LLP), whe e no communica ion scheme
is implemen ed. Addi ionally, we compa ed i s beha io wi h he diagonal
p econdi ione .
25
1.6. Linele p econdi ione
(a) Case un on a mesh wi h 0.48M elemen s, 49% o hem
wi hin he bounda y laye . The numbe o i e a ions o he
linele p econdi ione anges om 310 o an aspec a io o
14 o 321 o an aspec a io o 1024.
(b) Case un on a mesh wi h 0.5M elemen s, all wi hin he
bounda y laye . The numbe o i e a ions o he linele
p econdi ione goes om 94 o an aspec a io o 14 o 11
o an aspec a io o 1024.
Figu e 1.3: Compa ison o he numbe o i e a ions equi ed o each a ole ance
o 10
−6
o he linele and diagonal p econdi ione in he PCG Me hod on meshes
wi h a ying maximum aspec a io and bounda y laye co e age.
26
1.7. Auxilia y me hods
1.7 Auxilia y me hods
This sec ion in oduces he TDMA and he Schu -complemen me hod, which
will be used in he ollowing sec ions o sol e he p econdi ioning equa ions.
In pa icula , he Schu -complemen me hod will be used o pa allelize he
TDMA.
1.7.1 TDMA: T idiagonal Ma ix Algo i hm
This is a simple, s aigh o wa d me hod used o ind a di ec solu ion o any
linea sys em whose ma ix is idiagonal. In his wo k, i will be used o sol e
equa ions (1.47) and (1.48). Suppose we need o sol e he ollowing p oblem:
b1c10 0 0 . . . 0
a2b2c20 0 . . . 0
0a3b3c30. . . 0
0 0 a4b4c4. . . 0
. . . . . . . . . . . . . . . . . . . . .
00000. . . bn
x1
x2
x3
x4
. . .
xn
=
b1
b2
b3
b4
. . .
bn
.
I he e is no need o p ese e he alues o
bi
and
di
, hen he ollowing
i e a ion sol es analy ically o xi:
Fo i=2,3,...,n do:
wi=ai
bi−1
bi=bi−wici−1
di=di−widi−1.
(1.41)
Followed by he back subs i u ion:
xn=dn
bn
xi=di−cixi+1
bi
,
(1.42)
whe e he las line in (1.42) is i e a ed o e i=n−1, n −2, ..., 1.
1.7.2 The Schu -complemen me hod
The Schu -complemen me hod simpli ies sol ing linea sys ems by pa i ioning
ma ices in o blocks and elimina ing a iables, making i especially e ec i e
o spa se ma ices wi h block s uc u es. The main idea behind his me hod
is o decompose he linea sys em in o smalle subsys ems in ol ing only a
subse o a iables. Once he educed sys em is sol ed, he emaining a iables
a e de e mined using back-subs i u ion. This me hod is pa icula ly use ul o
pa allel compu ing, as i allows each p ocesso o sol e i s subdomain plus an
33
1.7. Auxilia y me hods
in e ace p oblem wi h jus one global communica ion.
Suppose a linea sys em
Ax
=
b
, whe e
x
is dis ibu ed among
P
p ocesso s.
In e ace and inne nodes a e de ined so ha he inne nodes o a gi en
subdomain a e only coupled o inne nodes o he same subdomain and o
in e ace nodes. In e ace nodes, on he con a y, couple inne nodes o di e en
subdomains. No e ha inne nodes om di e en subdomains a e no coupled
wi hin his app oach.
We ake i apa o o m a new subse
xs
so ha
x
is pa i ioned in o
P
subse s plus an in e ace s. Thus, ec o xcan be eo de ed as
x= (x0, x1, ..., xP−1, xs) .(1.43)
I is impo an o no e ha by de ini ion, gi en wo subse s
xp1
and
xp2
,
he sys em ma ix does no link elemen s o hese subse s oge he , bu only
be ween
xpi
and he in e ace e ms
xs
, hence equa ion
Ax
=
b
can be ew i en
as:
A0,00. . . A0,s
0A1,1. . . A1,s
. . . . . . . . . . . .
0. . . AP−1,P −1AP−1,s
As,0As,1. . . As,s
x0
x1
. . .
xP−1
xs
=
b0
b1
. . .
bP−1
bs
(1.44)
By block Gaussian elimina ion, he solu ion o equa ion
(1.44)
is gi en by
sol ing he ollowing se o linea equa ions:
˜
bs=bs−
P−1
X
p=0
As,pA−1
p,pbp(1.45)
˜
As,s =As,s −
P−1
X
p=0
As,pA−1
p,pAp,s.(1.46)
Finally, he solu ion o xis ound as:
˜
As,sxs=˜
bs(1.47)
and
˜
Ap,pxp=bp−Ap,sxs.(1.48)
The e o e, once he linea sys em is eo de ed as
(1.44)
, in e ace nodes can
be sol ed by applying equa ions
(1.45)
,
(1.46)
and
(1.47)
. Using he solu ions
o hese in e ace nodes
xs
, equa ion
(1.48)
sol es he sys em o he emaining
nodes.
No e ha i ma ix
A
does no change du ing he execu ion, hen ma ices
As,s
,
Ap,p
, and
Ap,s
can be compu ed only once a he p ep ocessing s age.
Also, he p oduc
As,pA−1
p,p
can be compu ed only once. The way o do his will
34
1.8. Conclusions
be explained in de ail in sec ion 1.7.2.
1.8 Conclusions
This chap e has in oduced he mo i a ions and challenges unde lying he de el-
opmen o he linele p econdi ione , pa icula ly in he con ex o high- ideli y
CFD simula ions. One o he p ima y limi a ions o con en ional sol e s lies in
he ea men o highly aniso opic meshes, whe e adi ional p econdi ione s
s uggle o main ain e iciency and con e gence. The linele p econdi ione was
de eloped o add ess his issue by le e aging he dominan coupling di ec ions
in he sys em ma ix. Howe e , despi e i s e ec i eness in handling hese
p oblems, ex ending he me hod o pa allel compu ing en i onmen s wi hou
deg ading he sol e ’s con e gence p esen s new challenges, pa icula ly in
handling in e -subdomain couplings. I is common p ac ice o cons ain linele s
o a single subdomain, p e en ing hem om spanning mul iple pa i ions.
While his simpli ies he applica ion o he p econdi ione , i also esul s in he
exclusion o in e -subdomain couplings om he p econdi ioning ma ix, leading
o a deg ada ion in con e gence. This e ec becomes especially p onounced
in highly aniso opic meshes wi h many domain pa i ions. As shown in his
chap e , he loss o hese couplings can inc ease he numbe o sol e i e a ions
equi ed o achie e a gi en le el o accu acy.
To o e come hese limi a ions, we p opose a no el pa alleliza ion s a egy
o he linele p econdi ione , which allows linele s o ex end ac oss mul iple
subdomains wi hou deg ading he sol e ’s con e gence. This app oach is
based on he Schu complemen me hod, which enables e icien handling o
in e -subdomain dependencies while main aining he spa si y and s uc u e o
he p econdi ioning ma ix. By adap ing exis ing algo i hms o inco po a e
hese couplings, we ensu e ha he p econdi ione emains e ec i e ega dless
o he domain pa i ioning s a egy used. This mi iga es he deg ada ion in
con e gence obse ed in adi ional me hods and enhances sol e obus ness
and scalabili y in la ge-scale simula ions.
By allowing linele s o be dis ibu ed ac oss mul iple p ocesses wi hou
sac i icing accu acy, we enable mo e e icien simula ions o complex low
phenomena, pa icula ly in cases whe e high Reynolds numbe s and bounda y
laye e ec s play a c i ical ole. The subsequen chap e s will del e in o he
heo e ical o mula ion, implemen a ion de ails, and nume ical expe imen s
ha alida e he e ec i eness o his app oach.
Re e ences
[1]
Richa d M Beam and R.F Wa ming. ‘An implici ini e-di e ence
algo i hm o hype bolic sys ems in conse a ion-law o m’. In: Jou nal
o Compu a ional Physics 22.1 (1976), pp. 87–110.
35
1.8. Conclusions
[2]
Michele Benzi. ‘P econdi ioning Techniques o La ge Linea Sys ems: A
Su ey’. In: Jou nal o Compu a ional Physics 182.2 (2002), pp. 418–477.
issn: 0021-9991. doi:h ps://doi.o g/10.1006/jcph.2002.7176.u l:
h ps://www.sciencedi ec .com/science/a icle/pii/S0021999102971767.
[3]
William L. B iggs, Van Emden Henson and S e e F. McCo mick. A
Mul ig id Tu o ial (2nd Ed.) USA: Socie y o Indus ial and Applied
Ma hema ics, 2000. isbn: 0898714621.
[4]
W.R. B iley and H. McDonald. ‘Solu ion o he mul idimensional
comp essible Na ie -S okes equa ions by a gene alized implici me hod’.
In: Jou nal o Compu a ional Physics 24.4 (1977), pp. 372–397.
[5]
J.A. Ca lson, A. Ja e, A. Wiles, Clay Ma hema ics Ins i u e and Ame ican
Ma hema ical Socie y. The Millennium P ize P oblems. Ame ican
Ma hema ical Socie y, 2006, pp. 57–67. isbn: 9780821836798. u l:h ps:
//books.google.es/books?id=7wJIPJ80RdUC.
[6]
Ramon Codina. ‘P essu e S abili y in F ac ional S ep Fini e Elemen
Me hods o Incomp essible Flows’. In: Jou nal o Compu a ional Physics
170.1 (2001), pp. 112–140. issn: 0021-9991. doi:h ps://doi.o g/10.1006/
jcph.2001.6725.u l:h ps://www.sciencedi ec .com/science/a icle/pii/
S0021999101967257.
[7]
Ramon Codina. ‘P essu e S abili y in F ac ional S ep Fini e Elemen
Me hods o Incomp essible Flows’. In: Jou nal o Compu a ional Physics
170 (June 2001), pp. 112–140. doi:10.1006/jcph.2001.6725.
[8]
P. Có doba Pañella. ‘Local p econdi ioning o pa allel i e a i e sol e s.’
PhD hesis. UPC, Facul a de Ma emà iques i Es adís ica, 2022. u l:
h p://hdl.handle.ne /2117/370230.
[9]
D. J. D. Ma iplis. ‘On con e gence accele a ion echniques o uns uc-
u ed meshes’. In: 29 h AIAA, Fluid Dynamics Con e ence.doi:10.2514/
6.1998-2966. ep in : h ps://a c.aiaa.o g/doi/pd /10.2514/6.1998-2966.
u l:h ps://a c.aiaa.o g/doi/abs/10.2514/6.1998-2966.
[10]
V.Ragh end a D.Bahuguna and B.V.R.Kuma . Topics in sobole spaces
and applica ions. Na osa Publishing House, 2002, p. 28. isbn: 1842650947.
[11]
Timo hy A. Da is. Di ec Me hods o Spa se Linea Sys ems. USA:
Socie y o Indus ial and Applied Ma hema ics, 2006, p. 67. isbn:
0898716136.
[12]
O. Hassan, K. Mo gan and J. Pe ai e. ‘An implici ini e-elemen me hod
o high-speed lows’. In: In e na ional Jou nal o Nume ical Me hods in
Enginee ing 32.1 (1991), pp. 183–205.
[13]
Rainald Löhne ; Fe nando Mu ; Juan Raul Ceb al; Romain Aub y;
Guillaume Houzeaux. ‘De la ed p econdi ioned conjuga e g adien sol e s
o he p essu e-Poisson equa ion: Ex ensions and imp o emen s’. In:
In e na ional Jou nal o Nume ical Me hods in Enginee ing 2010-jun
22 ol. 87 iss. 1-5 87 (1-5 June 2010). doi:10.1002/nme.2932.u l:
libgen.li/ ile.php?md5=385da88d3a20687c5a 11860062 9112.
[14]
Roux F. Houzeaux G. Mogoules F. Pa allel scien i ic compu ing. Wiley
Compu e Enginee ing Se ies, 2015, pp. 59, 62, 65, 66, 86, 88, 98, 99, 100,
120. isbn: 9781118761687.
36
1.8. Conclusions
[15]
Thomas JR Hughes. The ini e elemen me hod: linea s a ic and dynamic
ini e elemen analysis. Cou ie Co po a ion, 2003, p. 118.
[16]
No iyuki Kushida. ‘Condi ion Numbe Es ima ion o P econdi ioned
Ma ices’. In: PloS one 10 (Ma . 2015), e0122331. doi:10.1371/jou nal.
pone.0122331.
[17]
Ul ich Lange and Ma in Neumülle . ‘Di ec and I e a i e Sol e s’. In:
Compu a ional Acous ics. Ed. by Man ed Kal enbache . Cham: Sp inge
In e na ional Publishing, 2018, pp. 66–67. isbn: 978-3-319-59038-7. doi:
10.1007/978-3-319-59038-7_5.u l:h ps://doi.o g/10.1007/978-3-319-
59038-7_5.
[18]
D. Ma in and R. Löhne . ‘An implici , linele -based sol e o incom-
p essible lows’. In: AIAA 1992-668. 30 h Ae ospace Sciences Mee ing and
Exhibi . Jan. 1992.
[19]
DOROTHEE MARTIN and RAINALD LOEHNER. ‘An implici , linele -
based sol e o incomp essible lows’. In: 30 h Ae ospace Sciences Mee ing
and Exhibi .doi:10.2514/6.1992-668. ep in : h ps://a c.aiaa.o g/doi/pd /
10.2514/6.1992-668.u l:h ps://a c.aiaa.o g/doi/abs/10.2514/6.1992-
668.
[20]
D. J. Ma iplis. ‘Di ec ional Agglome a ion Mul ig id Techniques o
High-Reynolds-Numbe Viscous Flows’. In: AIAA Jou nal 37.10 (1999),
pp. 1222–1230.
[21]
D. J. Ma iplis. ‘On he accu acy o he pseudocomp essibili y me hod
in sol ing he incomp essible Na ie -S okes equa ions’. In: Applied
Ma hema ical Modelling 11.1 (1987), pp. 35–44.
[22]
J. Pe o . ‘An Analysis o he F ac ional S ep Me hod’. In: Jou nal o
Compu a ional Physics 108 (Sep . 1993), pp. 51–58. doi:10.1006/jcph.
1993.1162.
[23]
Ra i Ramamu i and Rainald Löhne . ‘A pa allel implici incomp essible
low sol e using uns uc u ed meshes’. In: Compu e s and Fluids 25 (Feb.
1996). doi:10.1016/0045-7930(95)00032-1.
[24]
Y. Saad. I e a i e Me hods o Spa se Linea Sys ems. 2nd. USA: Socie y
o Indus ial and Applied Ma hema ics, 2003, pp. 4, 5, 57, 63, 69, 75, 82,
83, 90, 92, 100, 149. isbn: 0898715342.
[25]
Youce Saad and Ma in H. Schul z. ‘GMRES: A Gene alized Minimal
Residual Algo i hm o Sol ing Nonsymme ic Linea Sys ems’. In: SIAM
Jou nal on Scien i ic and S a is ical Compu ing 7.3 (1986), pp. 856–869.
doi:10.1137/0907058. ep in : h ps://doi.o g/10.1137/0907058.u l:
h ps://doi.o g/10.1137/0907058.
[26]
Youse Saad. I e a i e Me hods o Spa se Linea Sys ems. Second.
Socie y o Indus ial and Applied Ma hema ics, 2003. doi:10.1137/
1.9780898718003. ep in : h ps://epubs.siam.o g/doi/pd /10.1137/
1.9780898718003.u l:h ps://epubs.siam.o g/doi/abs/10.1137/1.
9780898718003.
[27]
Jona han R Shewchuk. An In oduc ion o he Conjuga e G adien Me hod
Wi hou he Agonizing Pain. Tech. ep. USA, 1994.
37
1.8. Conclusions
[28]
O lando So o, Rainald Lohne and Fe nando Camelli. ‘A Linele
p econdi ione o incomp essible low sol e s’. In: In e na ional Jou nal
o Nume ical Me hods o Hea & Fluid Flow - INT J NUMER
METHOD HEAT FL F 13 (Feb. 2003), pp. 133–147. doi:10.1108/
09615530310456796.
[29]
JM Va gas-Felix and S Bo ello-Rionda. ‘Pa allel di ec sol e s o ini e
elemen p oblems’. In: Comunicaciones del CIMAT, I-10-08 (CC) (2010).
[30]
Jacob Wal z. ‘Uns uc u ed mul ig id o ime-dependen incomp essible
luid low. PhD Thesis, School o Compu a ional Sciences o Geo ge Mason
Uni e si y’. In: (Jan. 2000).
[31] P. Wesseling. In oduc ion o Mul og id Me hods. Tech. ep. 1995.
[32]
P. Wesseling and C.W. Oos e lee. ‘Geome ic mul ig id wi h applica ions
o compu a ional luid dynamics’. In: Jou nal o Compu a ional and
Applied Ma hema ics 128.1 (2001). Nume ical Analysis 2000. Vol. VII:
Pa ial Di e en ial Equa ions, pp. 311–334.
38
CHAPTER 2
E icien Pa alleliza ion o he
T idiagonal Ma ix Algo i hm
Pa o he con en s o his chap e ha e been published as:
R. de Olazábal, R. Bo ell and O. Lehmkuhl. ‘An algeb aic global
linele p econdi ione o incomp essible low sol e s’. In: Jou nal o
Compu a ional Physics 514 (2024),
This me hod is p ima ily based on he p e ious wo k done by O. So o, R.
Löhne , and F. Camelli [2] on a linele p econdi ione o aniso opic meshes.
In a way, i se es as an ex ension o i and i s na u al con inua ion. One o
he main challenges o he linele p econdi ione is ha i equi es in e ing
a idiagonal ma ix whose en ies may be dis ibu ed ac oss many p ocesses.
The e o e, he usual app oach is o disca d in e domain couplings. In his wo k,
we p opose a me hod ha allows o hese couplings o be aken in o accoun ,
hus imp o ing he p econdi ione ’s pe o mance.
Two p ima y challenges mus be add essed o achie e his goal. The i s
o pe o m a T idiagonal Ma ix Algo i hm (TDMA) on he p econdi ioning
ma ix
M
when i s en ies a e dis ibu ed ac oss mul iple p ocesses. The second
is e icien ly assembling
M
while p ese ing he necessa y couplings be ween
subdomains.
This chap e ocuses on he i s challenge: pa allelizing he TDMA by
employing he Schu Complemen Me hod. This app oach enables he e icien
solu ion o he linea sys em
(1.39)
in pa allel while main aining he idiagonal
s uc u e o he p econdi ioning ma ix
M
. We also p esen a echnique o
e icien ly compu e each subdomain’s con ibu ion o he .h.s., minimizing
edundan compu a ions and communica ion o e head. By le e aging he
inhe en spa si y o he idiagonal sys em and op imizing in e p ocess commu-
nica ion, we demons a e how o signi ican ly educe compu a ional cos s while
main aining he accu acy and obus ness o he p econdi ione .
Finally, no e ha he me hod p esen ed in his chap e is no es ic ed o
he linele p econdi ione . I can be applied o any idiagonal ma ix whose
en ies a e dis ibu ed ac oss mul iple p ocesses. Howe e , h oughou his
39
2.1. Va ious app oaches
chap e , we ocus on linele s and he linele p econdi ione wi hou loss o
gene ali y, as he connec i i y g aph o any idiagonal ma ix o ms a 1D
s uc u e, which can be in e p e ed as a linele .
2.1 Va ious app oaches
Linele p econdi ione s ha e been widely used in he li e a u e. Since
pa allelizing hem when he en ies o he p econdi ioning ma ix a e sp ead
o e many p ocesses is no i ial, one o wo app oaches is aken when dealing
wi h hese ypes o p econdi ione s in pa allel sol e s.
2.1.1 P e ious app oaches
The i s app oach is o le each subdomain de ine i s linele s. In his app oach,
no p ocess sha es in o ma ion wi h ano he ega ding he linele s. In his
case, he domain pa i ion is independen o he linele s. The absence o
communica ions implies ha each subdomain may con ain sec ions o di e en
linele s, bu hese sec ions a e ea ed as sepa a ed linele s on hei own. This
app oach, as shown in igu es 1.8-1.10, can limi he nume ical pe o mance o
he p econdi ione .
The second one is o adap he mesh pa i ion o he size o he linele s wi hin
each subdomain. Since linele s can include mo e elemen s han in he p e ious
case, and hus mo e couplings a e aken in o accoun , a majo co espondence
be ween ma ices
A
and
M
is achie ed. Ne e heless, adap ing he mesh may
lead o a poo load balance o , due o he geome y o he p oblem, i may no
be possible o accommoda e e e y linele wi hin a single subdomain. In his
si ua ion, i will s ill be necessa y o cu some o he linele s, lea ing us in a
simila si ua ion o he p e ious case.
2.1.2 Ou app oach: agnos ic o he domain pa i ion
In his wo k, we y a di e en app oach: he domain pa i ion is de ined
independen ly o he linele geome y, bu a communica ion s ep is implemen ed
wi hin he p econdi ioning s ep. The e o e, he linele s uc u es, and he e o e
he p econdi ione , a e agnos ic o he domain pa i ion. This is done so ha
di e en sec ions o each linele can sha e in o ma ion. Hence, he numbe o
i e a ions will be educed compa ed o he case whe e hese communica ions
a e no allowed (since, algeb aically, i will be equi alen o sol ing he linea
sys em shown in equa ion (1.39) sequen ially, le ing he linele s be as long as
possible and achie ing a majo co espondence be ween ma ices
A
and
M
).
Ne e heless, his does no necessa ily imply a educ ion in he o al sol e ime
since a communica ion s ep needs o be included, and hus, i will gene a e an
o e head. The e o e, a ade-o needs o be conside ed be ween he educ ion
in he numbe o i e a ions and he inc ease in cos pe i e a ion.
Fo cla i ica ion pu poses, we will ede ine he e ms “linele ” and “slice
o a linele ” o be used in his wo k. As shown in Figu e 2.1, we will use he
e m “slice” o e e o he local s uc u e gi en by he nodes loca ed wi hin
40
2.2. Pa alleliza ion o he TDMA using he Schu -Complemen Me hod
Figu e 2.1: De ini ion o he e ms "slice" and "linele ": We in oduce he e m
"slice" o e e o he local s uc u e o nodes ha a e connec ed wi hin a single
subdomain. In con as , "linele " e e s o he global s uc u e a ising om
many slices’ connec ion. In e ace nodes a e depic ed as emp y nodes, and inne
nodes as illed ones.
one subdomain. “Linele ”, on he o he hand, will e e o he global s uc u e
gi en by coupling di e en slices. No e ha a slice can be composed o a single
node, and a linele can be composed o a single slice.
I a whole linele is loca ed wi hin a single subdomain, equa ion
(1.39)
could
be di ec ly sol ed ia he TDMA p e iously explained. Ne e heless, his wo k
aims o de elop a me hod able o sol e equa ion
(1.39)
e en i a linele is sp ead
o e many p ocesses, i.e., e en i en ies o he idiagonal ma ix
M
shown
in igu e 1.2b a e dis ibu ed o e many p ocesses. The e o e, we wan a way
o pe o m a TDMA e en i he en ies o he ma ix a e sp ead o e many
p ocesses. The Schu Complemen Me hod was ound o be a sui able solu ion
o dealing wi h his case, pa allelizing he TDMA.
2.2 Pa alleliza ion o he TDMA using he
Schu -Complemen Me hod
This sec ion add esses he pa alleliza ion o he TDMA using he Schu -
Complemen me hod. We s a his discussion by ocusing on e icien ly
compu ing each slice’s con ibu ion o he .h.s.. In pa icula , we a oid
unnecessa y compu a ions and communica ion cos s. By p o i ing om he
idiagonal s uc u e o he ma ix and ca e ul handling o communica ion
be ween subdomains, we demons a e how o calcula e he con ibu ions wi hou
equi ing a ull TDMA ope a ion a e e y i e a ion. Once he con ibu ions
a e compu ed, he nex s ep is o ga he hem in a cen al p ocess and sol e
he sys em. The idiagonal s uc u e o he ma ix allows us o apply he
Schu -Complemen me hod o sol e he sys em, b eaking he p oblem in o
smalle slices and dis ibu ing he wo kload ac oss di e en p ocesses.
2.2.1
An e icien way o compu e each slice’s con ibu ion o he
.h.s.
Recall om Figu es 1.2a and 1.2b ha each block in he p econdi ioning ma ix
M
co esponds o a linele , wi h couplings be ween nodes in he linele s limi ed
o i s neighbo s. By di iding a linele in o sec ions and co ec ly iden i ying
41
2.2. Pa alleliza ion o he TDMA using he Schu -Complemen Me hod
he in e ace nodes, we can apply he Schu Complemen Me hod o sol e
he linea sys em
(1.39)
. The slicing o linele s is de e mined by he domain
pa i ion, wi h he in e ace nodes chosen as he i s ones a e he in e domain
in e ace, as shown in Figu e 2.1. In his igu e, in e ace nodes a e ep esen ed
as emp y ci cles, while inne nodes a e depic ed as illed ci cles. Impo an ly,
inne nodes om di e en subdomains a e no coupled. This sec ion explains he
algo i hm used o compu e he con ibu ions o he linea sys em
(1.39)
when
Mis idiagonal, and i s en ies a e dis ibu ed ac oss mul iple subdomains.
F om equa ions
(1.45)
and
(1.46)
, i ollows ha each p ocess managing a
slice o a linele mus compu e i s con ibu ion o equa ion
(1.47)
. Speci ically,
each p ocess needs o e alua e
As,pA−1
p,pbp
and
As,pA−1
p,pAp,s
. Howe e , i he
sys em ma ix emains cons an du ing he sys em’s e olu ion,
As,pA−1
p,pAp,s
can be compu ed once du ing p ep ocessing. In con as ,
As,pA−1
p,pbp
mus be
upda ed a e e y i e a ion. A di ec app oach would in ol e using a TDMA
o compu e
A−1
p,pbp
and hen mul iplying i by
As,p
, bu his would equi e
pe o ming a TDMA compu a ion e e y ime
˜
bs
is needed. Ins ead, his sec ion
demons a es ha no TDMA is necessa y o compu e
ˆ
bs
om equa ion
(1.45)
.
A mos , only wo columns o
As,pA−1
p,p
need o be compu ed a he s a , a e
which wo do p oduc s wi h
bp
su ice o calcula e
As,pA−1
p,pbp
du ing each
i e a ion.
As highligh ed in he Schu algo i hm (equa ions
(1.45)
–
(1.48)
), when he
ma ix emains unchanged be ween i e a ions, he p oduc
As,pA−1
p,p
, along wi h
As,s
, can be compu ed once a he p ep ocessing s age. Taking in o accoun
he s uc u e o he Schu ma ices
Ap,p
,
As,p
, and he idiagonal o m o he
p econdi ione ma ix
M
(Figu e 1.2b), his p oduc esul s in a spa se ma ix.
A leas one 2
×
2diagonal block is nonze o. I can be explici ly compu ed,
such as by applying TDMA o ex ac he equi ed componen s o
As,pA−1
p,p
, as
e e enced in equa ions (1.45) and (1.46).
Di ec compu a ion o he p oduc As,pA−1
p,p
Conside he example o a linele sp ead o e h ee p ocesses as shown in igu e
2.2. No e ha we may need a p ope enume a ion o he nodes o achie e
a idiagonal s uc u e. Suppose nodes 1 o 5belong o p ocess 0, nodes 6
o 9belong o p ocess 1, and nodes 10 o 12 belong o p ocess 2. We di ide
his ma ix in o di e en blocks ollowing he linele pa i ion imposed by he
domain pa i ion, and we eo de hem as shown in equa ion
(1.44)
. The esul
can be seen in igu e 2.2. In his case, we ha e ou diagonal blocks,
A0,0
,
A1,1
,
A2,2
, and he in e aces block
As,s
, oge he wi h six non diagonal blocks,
As,0
,
As,1, and As,2(and hei espec i e ansposes).
Since ma ix
As,p
has a mos wo nonze o en ies, he ec o esul ing
om ope a ing
As,pA−1
p,pbp
, will also ha e a mos wo nonze o elemen s. Hence,
ins ead o compu ing all o hem, we can ake ad an age o he ac ha we
al eady know he s uc u e o
As,p
. Thus, we can compu e only he nonze o
en ies while sa ing compu a ional esou ces.
A s aigh o wa d app oach o compu e
As,pA−1
p,pbp
would be o apply a
TDMA o ob ain
A−1
p,pbp
, and hen mul iply i wi h
As,p
. Howe e , his would
42
2.3. Conclusions
2.3 Conclusions
This chap e add esses he p oposed me hod o pa allelize he TDMA when
applied o ma ices whose en ies a e dis ibu ed ac oss mul iple p ocesses.
We de eloped a s a egy o p ese e in e domain couplings while main aining
compu a ional e iciency. This app oach o e s a solu ion agnos ic o domain
pa i ioning, imp o ing he co espondence be ween he sys em ma ix
A
and he p econdi ione
M
, educing he numbe o i e a ions equi ed o
con e gence.
The wo k in oduces an algeb aic and pa allel amewo k o implemen ing a
Linele P econdi ione o sol e Poisson’s equa ion, gene alizing he con en ional
app oach by allowing linele s o span mul iple subdomains. This ensu es a mo e
balanced wo kload dis ibu ion ac oss compu a ional esou ces. The algeb aic
app oach o his me hod allows o b oade applica ion o ma ices wi h simila
p ope ies. The lexibili y o he algeb aic me hod also opens up oppo uni ies
o ex ending i s use beyond he Poisson equa ion o any ma ix exhibi ing
linele s uc u e, ega dless o geome y.
Addi ionally, he p econdi ioning ma ix can also be assembled based on
geome ic conside a ions, which emphasizes he capabili y o his app oach
o be adap ed o di e en p oblems, wi h he added bene i ha he domain
decomposi ion can be independen ly op imized o load balancing, he eby
minimizing compu a ional o e head.
In o de o inc ease e iciency, we de ised a s a egy o compu e each
subdomain’s con ibu ion o he .h.s. o he sys em while aking ad an age
o he spa si y o he idiagonal ma ix o minimize edundan compu a ions
and communica ion o e head. We hen in oduced a pa allel scheme o sol ing
he esul ing sys em, mapping linele s o p ocesses in a way ha balances
compu a ional load while minimizing communica ion cos s. I is wo h men-
ioning ha his echnique is no limi ed o linele -based p econdi ione s, as
i can be applied o any idiagonal sys em dis ibu ed ac oss mul iple p ocesses.
In he nex chap e , we will ex end his wo k by add essing he e icien as-
sembly o he p econdi ioning ma ix
M
, ensu ing ha he necessa y couplings
be ween subdomains a e p ese ed while main aining scalabili y in la ge-scale
pa allel simula ions.
Re e ences
[1]
Youse Saad. I e a i e Me hods o Spa se Linea Sys ems. Second.
Socie y o Indus ial and Applied Ma hema ics, 2003. doi:10.1137/
1.9780898718003. ep in : h ps://epubs.siam.o g/doi/pd /10.1137/
1.9780898718003.u l:h ps://epubs.siam.o g/doi/abs/10.1137/1.
9780898718003.
49
2.3. Conclusions
[2]
O lando So o, Rainald Lohne and Fe nando Camelli. ‘A Linele
p econdi ione o incomp essible low sol e s’. In: In e na ional Jou nal
o Nume ical Me hods o Hea & Fluid Flow - INT J NUMER
METHOD HEAT FL F 13 (Feb. 2003), pp. 133–147. doi:10.1108/
09615530310456796.
50
CHAPTER 3
Assembling he p econdi ioning
ma ix
Pa o he con en s o his chap e ha e been published as:
R. de Olazábal, R. Bo ell and O. Lehmkuhl. ‘An algeb aic global
linele p econdi ione o incomp essible low sol e s’. In: Jou nal o
Compu a ional Physics 514 (2024),
and dissemina ed in he ollowing con e ences:
he 32nd Pa allel Compu a ional Fluid Dynamics Con e ence, Nice,
F ance, 2021,
he 15 h JLESC Wo kshop, Bo deaux, F ance, 2023,
he 1s Ma h 2 P oduc (M2P) con e ence o Eme ging Technologies
in Compu a ional Science o Indus y, Sus ainabili y and Inno a ion,
Tao mina, Sicily, 2023, and
he 23 d IACM Compu a ional Fluids Con e ence, San iago de Chile,
2025.
In he p e ious chap e , we in oduced an e icien pa alleliza ion s a egy
o he TDMA when applied o dis ibu ed ma ices. This app oach can be
implemen ed in he Linele P econdi ione , ex ending he con en ional app oach
o allow linele s o span mul iple subdomains. This gene aliza ion ensu es a
mo e balanced wo kload dis ibu ion and expands he me hod’s applicabili y o
ma ices wi h simila p ope ies.
P e ious app oaches o linele p econdi ione s in ol e allowing each
subdomain o de ine i s own linele s independen ly. This can hinde nume ical
pe o mance due o he lack o communica ion be ween subdomains o needing
o adap he mesh pa i ion o he linele geome y. While he la e imp o es
ma ix co espondence, i o en esul s in poo load balancing and u he
challenges wi h subdomain compa ibili y.
This chap e p esen s a me hod o cons uc ing and combining he slices
in o la ge linele s uc u es. Recall ha we de ine a "slice" as he local s uc u e
wi hin a subdomain, while a "linele " e e s o he global s uc u e o med by
51
3.1. Assembling he slices
me ging mul iple slices ac oss subdomains. Special a en ion mus be paid o
connec ing linele s om di e en domains, cons uc ing he linea sys em o he
linele p econdi ione when ma ix en ies a e dis ibu ed ac oss mul iple sub-
domains, and e icien ly pe o ming he necessa y in e -domain communica ions.
3.1 Assembling he slices
The assembly o he p econdi ioning ma ix
M
is composed o wo s eps. In he
i s s ep, each subdomain independen ly de ines i s se o slices. This is c ucial
o ob ain a easonable app oxima ion o he in e se o he sys em ma ix
A
,
as he e ec i eness o a linele p econdi ione depends s ongly on he p ope
choice o he se o linele s. Se e al known me hods can be used o pe o m
his s ep. Fo example, in [1], his is done by ollowing s eamlines, while in
[3], i is done by ollowing he s onges couplings. We will s ick wi h he
la e in his wo k, al hough he same implemen a ion can be used o he o me .
In he second s ep, linele slices ha belong o di e en subdomains a e
joined oge he . Howe e , no me hod has been ound in he li e a u e ha
pe o ms his s ep. The e o e, one o he p ima y con ibu ions o ou wo k is
de eloping a me hod ha can e icien ly pe o m his ope a ion, i.e., o ex end
he linele s beyond he bounda y be ween di e en subdomains.
3.1.1 Algeb aic slice assembly wi hin each subdomain
Le us s a his sec ion by b ie ly men ioning he me hod used o c ea e he
local slices. B ie ly, his s ep s a s by andomly selec ing a node as he sou ce
poin o a slice; hen, i con inues g owing his slice un il i can no longe
include addi ional nodes. Then, ano he node, no ye pa o any slice, is
chosen as he sou ce o a new slice. This p ocess is epea ed un il g owing
addi ional slices is no longe possible.
In he i s s ep, e e y subdomain builds i s own se o linele slices
independen ly. As men ioned abo e, his is done by keeping only he s onges
couplings while disca ding he es (see condi ions
(1.38)
). The algo i hm picks
one node andomly and hen g ows he slice ollowing he s onges couplings.
This p ocess is epea ed un il no mo e slices can be g own wi hin ha subdomain.
Schema ically, i consis s o he ollowing s eps:
1.
A node ha does no al eady belong o any slice is picked a andom. Le
us call i node i.
2.
Then, we look o he s onges coupling, i.e., he coupling
Aij
ha
sa is ies
|Aij|
|Aim|≥α o Nc−2 alues o m,m=j,i=j,α≥1.(3.1)
I such node
j
does no belong o any slice, i is aken o be he second
node o he slice. No e ha he longes linele s a e ob ained by se ing
52
3.1. Assembling he slices
α
= 1. This alue can be adjus ed o couple only hose nodes wi h a
su icien ly s ong coupling. The highe he alue o
α
, he s onge he
couplings ha he linele s admi . Wi hin his s ep, i also should be
checked ha o node
j
, i s coupling wi h
i
is among i s wo s onges
couplings. The eason why he nex node in he slice should sa is y his
condi ion o
Nc−
2 alues o
m
,
m=j
,
i=j
and no jus o e e y
alue o m=jwill be explained in he ollowing sec ion.
3.
Then we mo e o node
j
and epea he p ocess. I is impo an in his
s ep o igno e he coupling wi h node
i
, which al eady belongs o he
linele .
4.
This p ocess is epea ed un il a) we come ac oss a node al eady included
in a linele , b) condi ion
(3.1)
is no sa is ied, o c) he nex node is pa
o he halo (in Figu e
(3.1d)
his p ocess s ops when node
j
is coupled o
node kin he halo).
5.
I one o he p e ious hings happens, we go back o he i s node
i
and
s a sea ching again bu now igno ing he nodes al eady belonging o he
linele (Figu e 3.1d, he sea ch is s a ed again om node
i
igno ing he
p e ious coupling wi h node j).
6. The p ocess is i e a ed un il p e ious condi ions a e no longe sa is ied.
This p ocess is also exempli ied in Figu e 3.1.
When dealing wi h ill-condi ioned ma ices a ising om highly s e ched
bounda y laye s, i is wo h no icing ha some o he p e ious s eps could be
a oided by se ing o g ow he linele s in he o hogonal di ec ion o he mesh
s e ching. Ne e heless, we wan his me hod o be algeb aic and a oid any
in o ma ion ega ding he geome y o he mesh.
A he end o his s ep, each p ocess will ha e all he in o ma ion ela ed
o he s uc u e and couplings o i s own slices, oge he wi h he couplings
be ween hem and he halo. This las piece o da a is impo an because i will
allow us o expand he slices (i.e., he local sec ion o he linele s) beyond he
subdomain in e ace.
Also, a his poin , each subdomain has in o ma ion ega ding i s own
se o slices. Howe e , hey ha e no in o ma ion on wha happens in o he
subdomains no how he communica ions mus be pe o med. We need o
de ine he communica ion scheme o le di e en slices sha e in o ma ion.
Rema ks on choosing he nex node in he slice
P e iously, we no ed ha when g owing he slices, i is impo an o check ha
o he nex no node
j
o be included in he slice, i s coupling wi h he p e ious
node
i
in he slice is among he wo s onges couplings o bo h o hem. Now,
we gi e a mo e de ailed explana ion o why his condi ion is necessa y and how
o implemen i . Le us s a wi h some conside a ions abou equa ion
(3.1)
.
Suppose he case shown in Figu e 3.2, whe e he i s node picked a andom is
53
3.1. Assembling he slices
(a) A node no
pa o any exis ing
linele slice is selec-
ed andomly. Then,
he s onges coup-
ling needs o be iden-
i ied.
(b) We e i y ha o
node
j
, i s connec ion
wi h
i
is among i s
wo s onges coup-
lings.
(c) We mo e o node
j
and epea he p o-
cess.
(d) As node
j
is linked
o he halo, he p o-
cess hal s, es a ing
om node
i
bu ex-
cluding he couplings
al eady included in
he slice.
(e) The p ocess is e-
pea ed un il equa ion
(3.1)
is no longe sa -
is ied.
( ) The same p ocess
is epea ed o e e y
node in he subdo-
main.
Figu e 3.1: I e a i e p ocess o build slices (i.e., he local sec ion o he linele s
wi hin each subdomain). This i e a i e p ocess is pe o med o e e y node in he
subdomain un il condi ions
(3.1)
a e no longe sa is ied, esul ing in he cons uc ion
o a comple e se o slices.
54
3.1. Assembling he slices
Figu e 3.2: Ske ch showing nodes and couplings o an example mesh. Nodes
h
,
iand jshould belong o he same slice.
node
i
. Then, o g ow he slice, we need o ind he s onges coupling. Suppose
in his case he s onges coupling is
Aij
, be ween nodes
i
and
j
(i.e.,
Aij > Aim
∀m=j,i=j, wi h j).
In he second s ep, we need o check whe he coupling
Aij
sa is ies equa ion
(3.1). Suppose in his case we ake α= 2, so condi ion (3.1) eads
|Aij|
|Aim|≥2,m=j. (3.2)
Figu e 3.2 shows ha we should include nodes
h
,
i
, and
j
in he slice unde
cons uc ion. Howe e , equa ion
(3.2)
will no (and should no , as we shall
see) hold o he couplings be ween hese h ee nodes. Since each o hem has
wo s onges couplings, he p e ious condi ion will no be alid ( o example,
o node
i
, we ha e
Aij/Aih
= 10
/
9
<
2). Ne e heless, o ha e a slice joining
hese nodes, condi ion
(3.2)
mus be changed o apply only o couplings no
included in he slice. Since each node can be coupled o a mos wo o he nodes,
condi ion
(3.2)
should no hold o hese wo couplings; hence,
(3.2)
needs o
hold o
Nc−
2couplings, whe e
Nc
is he o al numbe o nodes coupled o
node i. Condi ion (3.1) hen becomes
|Aij|
|Aim|≥α o Nc−2 alues o m,m=j,i=j . (3.3)
The p e ious s a emen is alid whe he node
h
should be included in he
cu en slice.
A e inding ha node
i
has i s s onges coupling wi h
j
, one migh
conside aking node
j
as he nex node o he slice. Ne e heless, we need o
be ca e ul o a oid he case whe e he couplings be ween
Ajk
and
Ajl
we e
much s onge han he coupling
Aij
. In o he wo ds, i could be he case
ha node
i
has i s s onges coupling wi h node
j
, bu coupling wi h
i
is no
among he wo s onges couplings o node
j
(
Aij > Aim ∀m=j
,
i=j
, bu
Aji ≤Ajm ∀m=i
,
j=i
). Hence, we could ha e he cases shown in Figu es
55
3.1. Assembling he slices
(a) Ske ch o a case whe e, by building
he slices ollowing he s onges coup-
lings, node
j
should belong o he same
linele as nodes hand i.
(b) In his case, node
j
should no
be included in he same linele as
nodes
h
and
i
. Howe e , om he
pe spec i e o node
i
, his si ua ion is
indis inguishable om he one depic ed
in Figu e 3.3a.
Figu e 3.3: Illus a ion highligh ing he impac o s onges couplings on linele
cons uc ion. In one scena io, node
j
is expec ed o belong o he same linele
as nodes
h
and
i
(Figu e 3.3a). Howe e , in a di e en scena io, node
j
should
no be included in he same linele as nodes
h
and
i
(Figu e 3.3b), e en i node
i
has he same couplings as be o e. In o he wo ds, whe he o no node
j
should
be pa o his slice is de e mined by he couplings o node
j
a he han hose
o node i.
3.3a and 3.3b. Fo example, om hese igu es, i can be seen ha e en i
he coupling be ween nodes
i
and
j
is dominan when aking in o accoun all
nodes coupled o
i
, o node
j
o be able o be included in he same slice, he
same coupling be ween
i
and
j
needs also o be dominan when es ic ed o all
nodes coupled o
j
. Al hough in bo h igu es, node
i
“sees” he same couplings,
in Figu e 3.3a nodes
h
,
i
and
j
should be included in he same slice, bu in
Figu e 3.3b one slice should con ain nodes
h
and
i
, and ano he one should
con ain nodes
k
,
j
and
l
. The only di e ence in bo h scena ios is how node
j
(and no
i
) is coupled o i s neighbo s. This is why, be o e adding node
j
node
o he slice, we mus check ha om he poin o iew o
j
, coupling
Aij
is
also among he wo dominan couplings.
In o he wo ds coupling Aij will be included in he slice i :
1. Aij
is among he wo s onges couplings o node
i
:
|Aij |
|Aim|≥α
o
Nc−
2
alues o m,m=j,i=j.
2. Aij is among he wo g ea es couplings o node j:|Aij |
|Ajm|≥α o Nc−2
56
3.1. Assembling he slices
alues o m,m=i,i=j.
Algo i hm 3 ou lines he algo i hm used o build he slices. He e, we deno e
α ol
he ole ance imposed in condi ion
(3.3)
. The main idea behind his
app oach o slice assembly is o explo e he nodes and hei connec i i y gi en
by ma ix
A
and o iden i y and assemble slices based on he ela i e weigh o
he couplings Aij.
Algo i hm 3 Build slices
1: o i in nodes do
2: i node i is no included in any slice hen
3: IsA ailable=False
4: Look o he s onges coupling Aij .▷Le jbe he node wi h he
s onges coupling.
5: Compu e he minimum alue o α o
|Aij |
|Aim|≥α o Nc−2 alues o m,m=j,i=j.
6: i (Aij is no in he halo) and
(is among he wo s onges coupling o node
j
) hen
7: IsA ailable=T ue
8: SecondNode ←j
9: else
10: Look o he second s onges coupling Aij .▷Le jbe
his
node.
11: Compu e he minimum alue o α o
|Aij |
|Aim|≥α o Nc−2 alues o m,m=j,i=j.
12: i Aji is among he wo s onges coupling o node j hen
13: IsA ailable=T ue
14: while (IsA ailable) and (α≥α ol) and
(node jis no included in any slice) do
15: Add node j o he slice
16: Look o he s onges coupling Ajk.▷Le kbe his node.
17: Compu e he minimum alue o α o
|Ajk |
|Ajm|≥α o Nc−2 alues o m,m=j,j=k
.
18: i (node kis in halo) and (α≥α ol) hen
19: The coupling be ween node jand he halo is s o ed.
20: i Ajk is among he wo s onges coupling o node k hen
21: IsA ailable=T ue
22: j←k
23: i (kis in he halo) o (α < α ol) o
(IsA ailable==False) hen
24: j←SecondNode
This algo i hm begins by i e a ing h ough each node (line 1). Le ing
i
deno e he s a ing node, he p ocess checks whe he
i
is no pa o an exis ing
slice (line 2). I no ,
i
is designa ed as a po en ial s a ing poin . Then, in line
3, an auxilia y boolean IsA ailable is se o False. This a iable will de e mine
whe he he g owing p ocess can be epea ed om he nex node in he slice.
57
3.1. Assembling he slices
This a iable will be se o T ue i he nex node
j
is no in he halo and he
coupling
Aji
is among i s wo s onges couplings. This is because i
j
is in
he halo, he algo i hm canno con inue g owing he slice. Also, i
j
is no in
he halo, bu he coupling
Aji
is no among i s wo s onges couplings, he
algo i hm canno con inue g owing he slice ei he . Subsequen ly, he algo i hm
iden i ies he s onges coupling om
i
, deno ed as node
j
(line 4). I
j
is no
in he halo, he algo i hm e i ies ha he coupling
Aij
sa is ies condi ion I)
(3.3)
(line 5). I sa is ied, he algo i hm e i ies whe he he coupling
Aji
is
among he wo s onges couplings o node
j
(lines 6-8, emembe ha since we
assume symme y, hen
Aij
=
Aji
). On he con a y, i node
j
is in he halo
o he coupling condi ions a e no sa is ied, he algo i hm es a s he sea ch
om node
i
(lines 10-14). Then, he algo i hm en e s a loop o ex end he
slice, adding nodes based on he s onges coupling om he cu en node
j
(lines 16-30). The leng h o his slice and he nodes included in i a e upda ed
a e e y i e a ion (line 25). The loop con inues un il a) he cu en node is
al eady pa o a slice, b) he coupling condi ions a e no longe sa is ied, o c)
he nex node is in he halo. I a e mina ion condi ion is me , he algo i hm
e u ns o he ini ial node
i
and es a s he sea ch in he o he di ec ion (line
28), igno ing nodes al eady included in he slice. The p ocess i e a es ac oss all
nodes (line 1), epea ing he p ocess un il no node
i
is le unexplo ed. The
p ocess e mina es when all nodes ha e been add essed, and e e y slice is buil .
A he end o his s ep, each p ocess will ha e all he in o ma ion ela ed o
he s uc u e and couplings o i s slices, oge he wi h he couplings be ween
hem and he halo. This las piece o da a is c ucial because i will allow us o
expand he slices (i.e., he local sec ion o he linele s) beyond he subdomain
in e ace. These couplings a e indica ed in Figu e 3.1 as do ed ed lines. Also,
a his poin , each subdomain con ains in o ma ion abou i s slices. S ill, i
lacks in o ma ion on he slices o he subdomains ha e buil o he equi ed
communica ion p ocedu es (i.e., how he communica ions mus be pe o med).
The nex s ep is o de ine he communica ion scheme.
3.1.2 Geome ic slice assembly wi hin each subdomain
In he p e ious sec ion, we discussed how o build he slices wi hin each
subdomain by inspec ing he sys em ma ix
A
. O he me hods can be used o
build he slices, such as ollowing s eamlines [1], using he s onges couplings,
o using a geome ic app oach. Suppose we ollow a geome ic app oach ins ead
o p e e ing an algeb aic one. In ha case, we can assemble he slices by
compu ing he dis ance be ween nodes and selec ing he close node (no e
ha i is s ill impo an ha he equa ion
(3.1)
holds o any new node o he
slice) o , on he o he hand, we can de ine a ma ix o dis ances be ween nodes
and use his ma ix o build he slices. In ou case, we gene a e linele s ia a
geome ic app oach by de ining a ma ix
D
as in
(1.40)
. Then, we ollow he
same s eps as in he p e ious sec ion, bu ma ix
D
is now used o build he
slice ins ead o he linea sys em ma ix. In his case, ma ix
D
is only used o
build he slices, bu he p econdi ioning ma ix
M
en ies a e s ill illed wi h
he co esponding ma ix
A
. In o he wo ds, he co esponding en ies o he
Schu -Complemen ma ices (
(1.45)
-
(1.48)
) a e illed wi h he en ies o he
58
3.2. Halo cleaning
Algo i hm 5 Algo i hm used o symme ize in e domain couplings.
1: Bu e ←IdGlobal
2: Halo upda e on Bu e
3: Bu e HaloUpda ed ←Bu e
4: o i in haloCouplings do
5: Couplings(i) ▷Looks o he couplings o he halo
6: Bu e [i] ←bu e [Fi s NodeCoupling]
7: Halo upda e on Bu e
8: o i in haloCouplings do
9: Couplings(i)
10: i IdGlobal[i]==Bu e [Fi s NodeCoupling] hen
11: This coupling emains
12: else i nCouplings(i)==2 hen
13: i IdGlobal[i]==Bu e [Las NodeCoupling] hen
14: This coupling emains
15: o i in haloCouplings do
16: Couplings(i)
17: Bu e HaloUpda ed[i] ←Bu e HaloUpda ed[Las NodeCoupling]
18: Halo upda e on Bu e HaloUpda ed
19: o i in haloCouplings do
20: Couplings(i)
21: i IdGlobal[i]==Bu e HaloUpda ed[Fi s NodeCoupling] hen
22: This coupling emains
23: else i nCouplings(i)==2 hen
24: i IdGlobal[i]==Bu e HaloUpda ed[Las NodeCoupling] hen
25: This coupling emains
26: E e y o he coupling is disca ded
h ough all nodes, which a he same ime a e pa o a slice and a e coupled
o he halo. Fo each one o hese nodes, line 5 sea ches o he node in he
halo o which i is coupled. Line 6, he en y associa ed wi h ha node in
Bu e is eplaced by he alue o he node o which i is coupled, which we
call Fi s NodeCoupling. In Figu e 3.7, his is pe o med in he s ep deno ed
by Bu e [i]
←
bu e [Fi s NodeCoupling]. This is wha happens, o example,
o node
i
in he le column whe e, since i is coupled o node
m
(see Figu e
3.7), i s ID is eplaced by he ecei ed ID o node
m
. Suppose i is coupled
o wo o he nodes in he halo - such as he case wi h node
l
in Figu e 3.5
- only one o hese couplings is conside ed in his s ep. This is depic ed in
he le column o Figu e 3.7, whe e he ID o node
l
-which is coupled o
nodes
q
and
- is eplaced by
q
( he idea o ha ing a second a ay called
Bu e HaloUpda ed is o epea his p ocess wi h hose couplings ha ha e
no been conside ed p e iously). Then, a second halo upda e is pe o med,
and he p ocess o eplacing en ies in Bu e wi h he alues in he halo o
whom hey a e coupled is epea ed. Fo all hose nodes whose own ID has been
eco e ed, Fi s NodeCoupling should no be disca ded. This is shown in Figu e
3.7 a e he second Bu e [i]
←
bu e [Fi s NodeCoupling]: when he ecei ed
ID (and colo ) ma ches he one in a ay IdGlobal, hen Fi s NodeCoupling
65
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.8: Final s age o he case shown in Figu e 3.5. Only he symme ic
in e domain couplings emain, while all he non-symme ic couplings we e
disca ded a e applying he halo cleaning p ocess desc ibed in his sec ion.
is no disca ded. Then, he same s eps pe o med be ween lines 4 and 18
a e again pe o med be ween lines 19 and 33, wi h he only change ha i a
node is coupled wice o he halo, hen Las NodeCoupling is used ins ead o
Fi s NodeCoupling. Also, his is pe o med wi h he a ay Bu e HaloUpda ed
de ined in line 21 (a e he i s halo upda e in Figu e 5). Once mo e, o e e y
node whose ID has been eco e ed, Las NodeCoupling has no been disca ded.
On he con a y, Fi s NodeCoupling and Las NodeCoupling a e disca ded o
e e y coupling whose ID has no been eco e ed. Fo example, in Figu e 5,
since nodes
i
,
o
do no eco e hei ID a inal s ages 1 and 2, hei coupling o
he halo mus be disca ded. This ans o ms he ini ial s a e shown in Figu e
3.5 o he one shown in 3.8.
A his poin , e e y coupling be ween nodes om di e en subdomains is
symme ic. This will be c ucial o he nex s ep, which consis s o g owing he
linele s by joining slices om di e en subdomains. This is explained in he
nex sec ion.
3.3 G owing he linele s: joining slices om di e en
subdomains
In his sec ion, we will desc ibe how o couple slices om di e en subdomains
in o la ge s uc e s called linele s in he con ex o his wo k. This is one o
he mos impo an s eps in he cons uc ion o he p econdi ione , as i is he
s ep whe e he communica ion be ween di e en subdomains is de ined. Also,
he ope a ions pe o med he e will be he eason why his p econdi ione is
agnos ic o he domain pa i ion.
The e a e wo main ways o couple slices om di e en subdomains. The
i s one is o selec a speci ic se o slices o be he s a ing poin o hei
espec i e linele s. One eason o do his is i we wan he linele s o s a
a he bounda y o he mesh. In such a case, we would selec he slices
con aining he nodes a he bounda y o he mesh o be he s a ing poin o
hei espec i e linele s. The second way is o le he slices join one ano he
wi hou speci ying any s a ing poin . This is he app oach we will ollow in
66
3.3. G owing he linele s: joining slices om di e en subdomains
his wo k. Ne e heless, o he sake o comple eness, we will explain bo h
me hods.
3.3.1 Selec ing a speci ic se o slices o be he s a ing poin o
hei espec i e linele s
In his app oach, he linele s a e g own om a p ede e mined se o slices. We
can ake one o he inpu s o he sol e o be he ID o he nodes candida es o
belong o he i s slice o hei linele (conside ing ha hey do no need o be
a one ex eme o hei slice, hey may be, o example, a he middle o a slice),
any slice con aining such nodes will be conside ed he i s one o hei linele .
The nex s ep consis s o he coupling o slices om di e en subdomains.
This is done in an i e a i e p ocess by “g owing” he linele s s a ing om
ce ain slices. Since one o he inpu s was he ID o he nodes candida es o
belong o he i s slice o hei linele (conside ing ha hey do no need o be
a he end o hei slice), any slice con aining such nodes will be conside ed he
i s one o hei linele .
This s ep can be be e unde s ood by conside ing he ollowing example.
Conside he connec i i y g aph gi en by he couplings in ma ix
A
o be he
one shown in Figu e 3.9, whe e each box ep esen s a di e en subdomain. Blue
nodes a e hose belonging o he i s slice o hei linele .
In he i s s ep, local slices a e assembled. This is shown be ween igu es
3.9a and 3.9b and was p e iously shown in Figu e 3.1. Also, ed do ed lines in
Figu e 3.9 deno e he couplings be ween di e en subdomains (i.e., couplings
be ween local slices and he halo).
The p ocess o coupling di e en slices is pe o med i e a i ely ia successi e
halo upda es. S a ing om hose slices candida es o be he i s o hei linele
(depic ed as slices con aining blue nodes in Figu e 3.9, hese blue nodes may
ep esen , o example, he nodes o elemen s adjacen o he ai oil in Figu e
1.7), i any o hese slices a e coupled o he halo, each one o hem is joined
wi h he co esponding node in hei espec i e subdomain. This p ocess is
i e a ed
Np−
1 imes un il a mos
Np
slices a e joined oge he o un il he e
a e no mo e slices o join. Figu es 3.9c o 3.9 show how he linele s g ow o
c oss one in e ace pe s ep.
Hence, he i e a i e p ocess shown in Figu e 3.9 can be summa ized as
ollows:
1.
Each subdomain builds i s own se o slices as desc ibed in sec ion 3.1
(and u he de eloped in 3.1). I is also impo an ha he in o ma ion
abou hei couplings o he halo ( ed do ed lines in Figu e 3.9) is s o ed.
2.
To g ow he linele s, we mus selec he slices ha will be he i s o hei
linele . In his example, hey a e aken as such hose including blue nodes
in Figu es 3.9a o 3.9 . As p e iously men ioned, hey may e e o he
nodes o elemen s adjacen o he ai oil in Figu e 1.7.
67
3.3. G owing he linele s: joining slices om di e en subdomains
(a) Ske ch o a hyb id
mesh di ided in o se e al
subdomains. Blue nodes
indica e ha hei slice is
a candida e o be he i s
in hei espec i e linele .
(b) Each subdomain con-
s uc s i s slices independ-
en ly om one ano he .
When a slice has a coup-
ling wi h he halo, his
coupling is s o ed o be
used when g owing he
linele s. Red do ed lines
deno e couplings wi h he
halo.
(c) A e a i s halo up-
da e, linele s a e g own
by connec ing he i s
slices ( hose con aining
blue nodes) wi h he
slices hey a e coupled o
h ough he halo.
(d) A second halo up-
da e is pe o med o in-
co po a e h ee slices in o
he linele s by joining he
p e iously g own linele s
wi h he slices hey a e
coupled o ia he halo.
(e) A hi d halo upda e is
used o g ow he linele s
o include ou slices.
( ) A e he inal halo
upda e, any coupling no
included in a linele is
disca ded. Figu e 3.10
shows his inal s age.
Figu e 3.9: Example o he linele cons uc ion p ocess using slices om di e en
subdomains.
68
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.10: Final s age a e assembling he slices and g owing he linele s.
Only couplings wi hin he linele s a e kep , and e e y o he coupling is disca ded.
3.
An halo upda e ope a ion is epea ed
Np−
1 imes (
Np
= 5 in he
p e ious example). By his ope a ion, each slice knows o which o he
slice is coupled (change om do ed o con inuous lines in Figu e 3.9c). In
his s ep, he communica ion scheme o be used in sec ion 2.2.2 is buil .
4.
A e
Np−
1i e a ions, i any emaining coupling has no been used o
join slices, ha coupling is disca ded. Also, i no linele has g own om
any gi en bounda y node, his node is ea ed as a node no included in
any linele .
To apply he Schu Complemen me hod o sol e equa ion
(1.39)
o each
node included in a linele , we need o iden i y he in e ace nodes, which we
de ined o be he i s node o each local linele a e he in e ace (by in e ace,
i is mean he bounda y be ween di e en subdomains). Figu e 3.10 shows
hese as emp y do s. This way, e e y inne node is coupled o o he inne o
in e ace nodes, jus as he Schu algo i hm equi es. Also, no e ha he i s
slice o he linele does no ha e in e ace nodes. These nodes a e aken o be
he i s one o he i− h linele slice o i > 1.
A e his s ep, each p ocess knows o which linele i s slices belong and hei
posi ion in hei espec i e linele . The nex s ep is o de ine he communica ion
scheme. To sol e he linea sys em o he in e ace nodes o each linele
(equa ion
(1.47)
), we assign each linele o a speci ic p ocess. This p ocess
sol es equa ion
(1.47)
o he co esponding linele . Hence, he i s s ep is o
map each linele o a gi en p ocess. In ou case, we op ed o dis ibu e he
linele s as e enly as possible.
Also, wi hin his app oach, linele loops a e na u ally a oided.
This is he me hod o g owing he linele s we ha e de eloped and published
in he wo k published in [2]. Howe e , he linele g ow h me hod has been
ex ended and gene alized, no equi ing a s a ing poin speci ica ion. This new
app oach is explained in he nex sec ion and he one we will con inue o use in
69
3.3. G owing he linele s: joining slices om di e en subdomains
he p esen wo k.
Fo a mo e de ailed desc ip ion o his app oach, he eade can e e o
Appendix B.
3.3.2 G owing he linele s wi hou speci ying a s a ing poin
An al e na i e app oach adop ed in his wo k in ol es g owing linele s wi hou
speci ying a s a ing slice. This me hod is mo e gene al and emo es he need
o use inpu . Howe e , a en ion mus be aken o p e en he o ma ion o
linele loops.
Conside he example in Figu e 3.11, whe e a slice wi h global ID
j
is
coupled on one side o a slice wi h global ID
i
and on he o he o a slice wi h
global ID
k
. Du ing a halo upda e communica ion s ep, coupling in o ma ion is
exchanged. Then, he p ocess managing slice
i
has he in o ma ion abou i s
coupling wi h slices jand k, and he same wi h he p ocess managing slices k
and j.
Subsequen halo upda es u he p opaga e his in o ma ion. Fo ins ance,
he p ocess handling slice
j
in o ms he p ocess managing slice
k
o i s coupling
wi h slice
i
. As a esul , he p ocess esponsible o slice
k
now has he
in o ma ion o bo h i s di ec coupling wi h slice
j
and he indi ec coupling
be ween slices
j
and
i
and simila ly wi h he p ocess managing slice
i
. A his
poin , p ocesses con aining
i
,
j
, and
k
ha e he comple e in o ma ion abou he
global s uc u e o he linele hey belong o.
Because he linele IDs a e globally unique, he coupling in o ma ion also
includes he anks o he p ocesses managing he coupled slices. This gua an ees
ha all da a necessa y o cons uc ing he communica ion scheme is eadily
a ailable. This p ocess is illus a ed in Figu e 3.11.
The e o e, his app oach begins by assigning global IDs o all slices coupled
o he halo. To g ow he slices, conside he example in Figu e 3.12, whe e en
slices a e coupled and numbe ed as shown. The i s s ep in ol es pe o ming
an allga he communica ion ac oss all p ocesses o sha e in o ma ion abou he
numbe o slices each p ocess has. This allows e e y p ocess o assign a unique
global ID o i s slices. To do his, each p ocess assigns a a iable called nSlices,
co esponding o i s numbe o slices. Then, an allga he communica ion is
pe o med wi h all he p ocesses. This way, each p ocess knows he numbe o
slices o e e y o he p ocess. Then, each p ocess assigns a global ID o each
slice, ensu ing no ID is epea ed. Fo example, i he a ay esul ing om his
allga he communica ion is [3, 2, 4, 1, 5], hen he slices o he i s p ocess
will be assigned global IDs 1, 2, and 3, he slices o he second p ocess will be
assigned global IDs 3 and 4, and so on. No e ha he e is no need o assign an
ID o he slices ha a e no coupled o he halo. Wi hin ou implemen a ion,
we will assign a global ID o e e y slice, e en hose no coupled o he halo,
bu wi h a ca ea : slices no coupled o he halo will ha e a global ID equal o
-1, while posi i e slices a e ese ed only o hose coupled o he halo. Also,
70
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.11: Example o he communica ion scheme used o g ow he linele s
wi hou speci ying a s a ing poin .
slices wi h only one coupling o he halo will ha e he lowes ID. La e , we will
see ha his will help p io i ize linele s g owing om he bounda y laye . This
global numbe ing o he linele s is done by doing he a o emen ioned allga he
bu , ins ead o nSlices, wi h ano he a iable nSlicesCoupled, which s o es he
numbe o slices coupled o he halo. By he end o his s ep, e e y p ocess
con aining a slice will ha e an a ay, which we called IdGlobal o size nSlices
con aining he global ID o each slice: local slice
i
has a global ID o IdGlobal[i].
No e om Figu e 3.11 and he explana ion p o ided ha he in o ma ion
lows in wo di ec ions, and he e o e, wo di e en ‘channels o communica ion’
a e used. The main idea is o assign o each slice a wo-dimensional a ay,
wi h each dimension co esponding o a communica ion channel h ough which
in o ma ion lows in a speci ic di ec ion. This is he basis o his p ocess.
Conside he example in Figu e 3.12, which depic s a se o en coupled
and numbe ed slices. The uppe sec ion o he Figu e illus a es he coupled
nodes o ming slices and hei co esponding global IDs, while he lowe sec ion
displays he en ies o he AuxField a ay assigned o each slice. The halo en ies,
o which he nodes a he ex eme o he slices a e coupled, a e ep esen ed as
do ed en ies. In his example, he i s deg ee o eedom acili a es he low
o in o ma ion om le o igh , while he second handles he low om igh
o le .
The e o e, he p ocess begins by de ining an auxilia y ield wi h wo deg ees
o eedom, called AuxField, whe e each node is mapped o wo en ies: node
i
is mapped o
AuxField
[
i,
1] and
AuxField
[
i,
2]. This a ay se es as he basis
o successi e halo upda es. Fo e e y node loca ed a he ex emi y o a slice,
he co esponding en ies in AuxField a e illed wi h he global ID o hei slice.
Al hough all en ies associa ed wi h he slice could be illed, only he en ies
equi ed o communica ion du ing he halo upda e a e necessa y, as only hey
in luence he esul .
We mus now de ine a pa ame e
Np
, which ep esen s he numbe o imes
71
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.12: Example o he a iables used o g ow he linele s wi hou speci ying
a s a ing poin .
his p ocess will be i e a ed. In he ollowing example, we conside
Np
= 2, as
his choice highligh s speci ic issues ha mus be add essed wi h his app oach.
Figu e 3.13 illus a es he low o in o ma ion o e wo halo communic-
a ion s eps o he case depic ed in Figu e 3.12. The colo ed en ies in he
AuxField a iable indica e ele an en ies, while all o he s a e byp oduc s
o he me hod and a e no essen ial. In he i s s ep, he en ies in he
AuxField a ay associa ed wi h he nodes a he ex emes o he slices a e
illed wi h hei co esponding slice’s global ID. Then, a halo upda e is pe -
o med. The subsequen s ep is o upda e he en ies o he AuxField a ay
associa ed wi h he nodes a he ex emes o he slices. In his s ep, i is
impo an o be consis en wi h he di ec ion in which he in o ma ion mus low.
Fo example, conside he slice wi h global ID 7 ( he second slice om he
le in Figu e 3.13). A e he i s halo upda e, i he le node ecei es an
inpu wi h a alue o 4, i mus be p opaga ed o he igh node o pass i o he
adjacen slice ( o he igh ) in he nex halo upda e. Simila ly, i he igh node
ecei es a alue o 9, his alue mus be copied o he le node o p opaga ion
o he adjacen slice ( o he le ) in he subsequen halo upda e. Fo a be e
unde s anding o he p ocess, in he diag am o igu e 3.13, we ha e kep ha
he low o in o ma ion o he igh goes h ough he i s channel while he
low o he le goes h ough he second one. Howe e , his is no necessa y.
I is su icien ha , o example, he igh node is upda ed wi h bo h inpu s
co esponding o he wo deg ees o eedom o he le halo node. I i ecei es
wo en ies o di e en alues, he ele an in o ma ion is in he one whose
alue i has no ye ecei ed.
Special a en ion is equi ed when dealing wi h single-node slices. In such
cases, he ele an inpu (i.e., he one no p e iously ecei ed) mus be copied
om each halo node in o each deg ee o eedom o he inne node. While his
s ep migh ini ially seem con using, i will become clea e wi h he explana ion
o he nex s ep.
A e his s ep, each slice has in o ma ion abou i s couplings wi h adjacen
slices and hei espec i e IDs. Fo example, slice 4 ‘knows’ is coupled wi h slice 7
on he igh , slice 7 ‘knows’ is coupled wi h slice 4 on he le , slice 9 on he igh ,
and so on. A his poin , we in oduce a new a iable ha will be use ul mo ing
o wa d: an a ay called CoupledSlices, wi h dimensions 2(
Np
+ 1)
×nSlices
,
whe e
nSlices
is he numbe o slices in he cu en p ocess, and he a ay has
72
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.13: De ailed example o he p ocess used o g ow he linele s wi hou
speci ying a s a ing poin .
Figu e 3.14: De ailed example o he p ocess used o g ow he linele s wi hou
speci ying a s a ing poin .
only one deg ee o eedom. Each slice in his a ay co esponds o 2(
Np
+ 1)
en ies in CoupledSlices. The en ies co esponding o he
i
- h slice o he
p ocess a e loca ed in CoupledSlices[2(
Np
+1)(i-1)+1:2(
Np
+1)i]. We assign
he global ID o he slice o he i s en y: CoupledSlices[2(
Np
+1)(i-1)+1] =
IdGlobal[i], and ini ialize he emaining en ies o ze o. This a ay will s o e
he o de in which he slices a e coupled. Figu e 3.14 illus a es he e olu ion
o he CoupledSlices sec ion co esponding o each slice o he example shown
in Figu e 3.12.
A e he i s halo upda e, CoupledSlices is upda ed acco ding o he
couplings ecei ed. Fo example, since slice 4 ecei ed only he coupling wi h
slice 7, i adds his alue o CoupledSlices. Simila ly, slice 7 ecei ed he
coupling wi h slice 4 and slice 9, so hese wo alues a e added. The coupling
ecei ed om he le is placed o he le o 7 (a he i s en y, i.e., a he
2(
Np
+1)(i-1)+1 en y), and he coupling om he igh is added o he igh
73
3.3. G owing he linele s: joining slices om di e en subdomains
Figu e 3.15: Example o he e olu ion o he CoupledSlices a ay om he
example shown in igu e 3.12 a e he i s halo upda e.
o 7 (i.e., eplacing he i s ze o in CoupledSlices[2(
Np
+1)(i-1)+1:2(
Np
+1)i])
in CoupledSlices. I is impo an o no e ha al hough i does no ma e
whe he couplings om he le o igh a e placed on he le o igh side
o CoupledSlices, consis ency mus be main ained, and he same app oach
should be ollowed a e each halo upda e. The idea is ha any new coupling
in o ma ion is added o CoupledSlices. Howe e , i a coupling alue al eady
exis s in CoupledSlices, i is no added again, ensu ing ha no en ies a e
epea ed o a gi en slice. The hi d column om he igh in Figu e 3.14 shows
he s a e o CoupledSlices a e he second halo upda e is comple ed. To build
he communica ion scheme co ec ly, each slice mus be awa e o i s posi ion in
he linele o which i belongs, which leads us o he nex s ep.
Now ha each slice knows i s couplings and hose o i s neighbo ing slices, he
nex s ep is o iden i y he di e en linele s and de e mine each slice’s posi ion
wi hin i s espec i e linele . I is impo an o no e ha he CoupledSlices
a ay, as shown in he hi d column o Figu e 3.14, does no ye e lec he
global s uc u e o he linele s. Fo example, we migh ini ially hink ha slice
4 (in he i s ow) belongs o he linele o med by slices 4, 7, and 9, while
slice 7 belongs o he linele o med by slices 4, 7, 9, and 2. This would c ea e
an inconsis ency in he linele s uc u e, po en ially dis up ing he solu ion o
he co esponding linea sys em. To ackle his, an addi ional s ep is needed o
ensu e he consis ency o he in o ma ion associa ed wi h each p ocess slice. In
his s ep, each slice iden i ies he minimum alue in i s espec i e CoupledSlices
sec ion, i.e., he slice wi h he smalles global ID o which i is coupled. This
alue is hen communica ed o he neighbo ing slices h ough a halo upda e,
simila o he p e ious p ocess in ol ing AuxField. No e ha only one deg ee
o eedom is equi ed o his s ep. An example o his p ocess is illus a ed in
Figu e 3.12.
Now, suppose a e his halo upda e, i does no ecei e om i s neighbo
74
CHAPTER 4
Implemen a ion and pe o mance
Pa o he con en s o his chap e ha e been published as:
R. de Olazábal, R. Bo ell and O. Lehmkuhl. ‘An algeb aic global
linele p econdi ione o incomp essible low sol e s’. In: Jou nal o
Compu a ional Physics 514 (2024),
and dissemina ed in he ollowing con e ences:
he 15 h JLESC Wo kshop, Bo deaux, F ance, 2023, and
he 1s Ma h 2 P oduc (M2P) con e ence o Eme ging Technologies
in Compu a ional Science o Indus y, Sus ainabili y and Inno a ion,
Tao mina, Sicily, 2023.
In his sec ion, we p esen nume ical esul s and compa e he pe o mance
o he diagonal p econdi ione , he LLP, i.e., he linele p econdi ione when
communica ions a e no allowed, and he Global Linele P econdi ione (GLP,
i.e., when communica ions a e allowed). I has been seen in p e ious wo ks [6]
ha o aniso opic meshes wi h highly s e ched elemen s, LLP educes he
numbe o PCG i e a ions. Also, we obse ed ha when his aniso opy is due
o mesh s e ching, GLP signi ican ly educes he numbe o i e a ions in he
solu ion o he p essu e equa ion and, consequen ly, he o al CPU ime.
To e alua e he p oposed me hod, we sol ed Poisson’s equa ion using he
ini e elemen me hod in a la ge-scale, uns eady, incomp essible low p oblem
wi h a Reynolds numbe o
Re
= 10
5
. The Reynolds numbe o he choice
o u bulence modeling app oach a ec s only he .h.s.. In ou case, we ha e
ob ained he .h.s. o LES. Howe e , we expec a e y simila beha io i
we had used RANS. I is impo an o highligh ha RANS can be used in
meshes wi h highe aniso opy han hose usually employed in LES. This is
pa icula ly ad an ageous o he p oposed me hod, as i has demons a ed
as e con e gence in cases wi h highe aspec a ios.
I is impo an o cla i y ha we ocused on sol ing Poisson’s equa ion o
he p essu e co ec ion a he han he comple e se o go e ning equa ions in
add essing hese challenges. As such, he ensuing esul s co espond exclusi ely
81
4.1. Implemen a ion
o he solu ion o he p essu e co ec ion equa ion.
4.1 Implemen a ion
The p esen sol e has been de eloped wi hin Alya, he mul i-physics ini e
elemen simula ion so wa e designed and main ained by he Ba celona Supe -
compu ing Cen e (BSC). Alya is op imized o high-pe o mance compu ing
(HPC) en i onmen s, ensu ing scalabili y on bo h CPUs and GPUs h ough
hyb id pa alleliza ion models ha combine MPI, OpenMP, and OpenACC.
The so wa e has unde gone igo ous e i ica ion, alida ion, and op imiza ion,
demons a ing i s eliabili y in sol ing complex luid dynamics p oblems, among
o he physical phenomena [7,1,3,4,5]. Fu he mo e, Alya is one o he wel e
simula ion codes included in he Uni ied Eu opean Applica ions Benchma ks
Sui e (UEABS), unde sco ing i s adhe ence o he highes HPC s anda ds.
These a ibu es make i a sui able pla o m o implemen ing and es ing he
p esen sol e in scien i ic and enginee ing applica ions.
Alya ollows a modula a chi ec u e consis ing o a co e ke nel and mul iple
specialized modules. The ke nel p o ides undamen al unc ionali ies o
sol ing disc e ized PDE, while he modules de ine he speci ic physics o a
gi en p oblem. The so wa e suppo s a di e se ange o physical simula ions,
including incomp essible and comp essible CFD, compu a ional solid mechanics
(CSM), hea ans e h ough di usion, con ec ion, and adia ion, pa icle
anspo , elec ophysiology, eac i e species anspo , and nuclea eac ions.
Depending on he mul i-physical na u e o a gi en p oblem, hese modules can
be coupled o enable complex simula ions, such as sp ay combus ion modeling,
which in eg a es u bulen low, eac ing chemical species, hea ans e , and
e apo a ing d ople s.
4.2 Wo k low
The wo k low o Alya ollows a s anda d s uc u e used in enginee ing sol e s
o nonlinea PDEs. The p ocess begins wi h a p ep ocessing s age in which
he inpu da a is ead, memo y is alloca ed, communica ion pa e ns a e
es ablished, and da a s uc u es a e ini ialized. Nex , a ime-s epping loop is
used o ansien p oblems. Fo nonlinea sys ems, an addi ional i e a i e loop
uns a each ime s ep o assemble and sol e he nonlinea algeb aic sys em,
epea ing un il he e o alls below a se h eshold. Once he e o alls below
a speci ied h eshold, he simula ion mo es o he nex ime s ep. A e all ime
s eps ha e been comple ed, he pos -p ocessing phase dealloca es memo y and
ends he simula ion.
4.3 Compu a ional Resou ces
All simula ions p esen ed in his hesis we e execu ed on he Ma eNos um V
supe compu e a BSC. As a p e-exascale Eu oHPC sys em, Ma eNos um V
o e s a peak compu a ional powe o 314 pe a lops, le e aging a combina ion
o Bull Sequana XH3000 and Leno o ThinkSys em a chi ec u es. The sys em
82
4.4. Nume ical e i ica ion o he sol e
is di ided in o ou specialized pa i ions o ca e o a ious HPC wo kloads.
The p ima y pa i ions include:
•
The Gene al Pu pose Pa i ion (GPP), deli e ing 45 pe a lops wi h 6480
s anda d nodes and 72 high-bandwid h memo y (HBM) nodes, powe ed
by In el Sapphi e Rapids p ocesso s.
•
The Accele a ed Pa i ion (ACC), o e ing 230 pe a lops h ough 1120
nodes equipped wi h N idia Hoppe GPUs and In el Sapphi e Rapids
p ocesso s.
Addi ional pa i ions ea u ing N idia G ace CPUs and nex -gene a ion
accele a o s will u he ex end he sys em’s capabili ies. Ma eNos um V
employs a high-pe o mance a - ee ne wo k opology and an ad anced s o age
sys em p o iding 248 pe aby es o capaci y, wi h ead/w i e speeds o 1.6 TB/s
and 1.2 TB/s, espec i ely. A long- e m a chi e supplemen s his wi h an
addi ional 402 pe aby es o ape-based s o age.
4.4 Nume ical e i ica ion o he sol e
To ensu e accu acy, eliabili y, and obus ness, he sol e unde wen a com-
p ehensi e e i ica ion and alida ion p ocess using mul iple es cases. I s
pe o mance and scalabili y we e also es ed ac oss di e en mesh sizes and
elemen ypes.
In his sec ion, we p esen nume ical esul s and analyze he pe o mance
o h ee p econdi ione s: he diagonal p econdi ione , he LLP, which ope -
a es wi hou in e -domain communica ion, and he GLP, which allows o
communica ion be ween subdomains. P e ious s udies [6,2] ha e shown ha
o highly aniso opic meshes wi h highly s e ched elemen s, LLP e ec i ely
educes he numbe o PCG i e a ions. Mo eo e , when he aniso opy esul s
om mesh s e ching, GLP has been obse ed o u he accele a e con e gence
in sol ing he p essu e equa ion, signi ican ly educing o al compu a ional ime.
To e alua e he p oposed app oach, we applied i o sol e Poisson’s equa ion
using he ini e elemen me hod wi hin a la ge-scale, uns eady, incomp essible
low simula ion a a Reynolds numbe o
Re
= 10
5
. No ably, when sol ing he
p essu e equa ion, he esul ing ma ix co esponds o a Laplacian ope a o
ha depends solely on he mesh s uc u e a he han he low p ope ies. The
Reynolds numbe and he u bulence modeling app oach in luence only he
.h.s. o he equa ion. In ou case, his e m was ob ained om a LES. Howe e ,
we an icipa e ha simila beha io would be obse ed i a RANS model we e
used. I is wo h emphasizing ha RANS is ypically applied o meshes
wi h e en highe aniso opy han hose used in LES, making i pa icula ly
well-sui ed o he p oposed me hod, as i s con e gence bene i s become mo e
p onounced in cases wi h g ea e aspec a ios. Fu he mo e, we es ed he sol e
using a andom ec o on he .h.s. and obse ed a simila con e gence beha io .
83
4.4. Nume ical e i ica ion o he sol e
I is impo an o cla i y ha we ocused on sol ing Poisson’s equa ion o
he p essu e co ec ion a he han he comple e se o go e ning equa ions. As
such, he ollowing esul s co espond exclusi ely o he solu ion o he p essu e
co ec ion equa ion.
I is also wo h men ioning ha , in ou expe ience, he linele p econdi ione ,
designed o exploi he mesh aniso opy in i s a o , has demons a ed imp o ed
pe o mance wi h inc easing aspec a ios. Speci ically, a highe aspec a io
co ela ed wi h a educed numbe o i e a ions equi ed o con e gence, indic-
a ing as e con e gence a es (we de ine he aspec a io as he geome ical
aspec a io, ep esen ing he a io o he maximum o minimum leng h o a
gi en elemen ). Consequen ly, he p esen ed esul s a e pa icula ly ad an -
ageous o scena ios cha ac e ized by highe aspec a ios. The simula ions o
his case we e execu ed u ilizing 48 o 768 p ocesses, wi h he pa ame e
Np
(p e iously discussed in chap e 3) selec ed o each case o ensu e he inclusion
o e e y linele slice. Impo an ly, as he numbe o p ocesses inc eased, all
p oblem pa ame e s emained cons an , allowing us o conduc a obus , s ong
scalabili y es .
In o de o s udy he con e gence o he GLP, we compa e he alue o he
no malized esidual gi en by
Res =||A.xi−b||
||b|| ,(4.1)
o GLP, LLP , and he diagonal p econdi ione , whe e
xi
is he alue o he
inc emen in p essu e a he
i− h
i e a ion. Since we a e sol ing o he
inc emen in p essu e, we ake he ini ial alue o he unknown (x0) o be
0.
In he ollowing analyses, we employ h ee di e en meshes o s udy he
p oposed gene aliza ion o he linele p econdi ione . These include a hyb id
mesh comp ising 1.64 million elemen s, wi h 60% o hem loca ed wi hin he
bounda y laye and a maximum aspec a io o 17
.
05. Addi ionally, we u ilize a
s uc u ed mesh wi h 0.98 million elemen s, all con ined wi hin he bounda y
laye and wi h he same maximum aspec a io o 17
.
05. To explo e he e icacy
o he p oposed imp o emen s unde mo e challenging condi ions, we in oduce
an ex eme case o a s uc u ed mesh wi h 5 million elemen s. All elemen s
a e loca ed wi hin he bounda y laye in his con igu a ion, and he maximum
aspec a io eaches 284.68.
Figu e 4.1 shows he con e gence o he esidual h oughou each i e a ion o
he P econdi ioned Conjuga e G adien (PCG) me hod o wo o he analyzed
cases: 1.64 and 0.98 million elemen meshes. No e how in Figu e 4.1a he e is
no signi ican di e ence in he con e gence du ing he i s s eps, bu he e is
a h eshold a e which GLP accele a es i s con e gence. On he o he hand,
when all he elemen s a e loca ed wi hin he bounda y laye , which is he case
depic ed in Figu e 4.1b, i can be seen ha he imp o emen in con e gence is
almos immedia e.
I we ins ead plo he numbe o i e a ions equi ed o each me hod o
educe he esidual by six o de s o magni ude (i.e., a esidual such ha he
84
4.4. Nume ical e i ica ion o he sol e
(a) Case un on a mesh wi h 1.64 million elemen s, wi h 60%
in he bounda y laye . GLP eaches a esidual o 10
−6
1.62
imes as e han LLP and 4.54 imes as e han he diagonal
p econdi ione .
(b) Case un on a mesh wi h 0.98 million elemen s en i ely loca ed
wi hin he bounda y laye . GLP eaches a esidual o 10
−6
2.97
imes as e han LLP and 9.27 imes as e han he diagonal
p econdi ione .
Figu e 4.1: Compa ison o PCG con e gence using GLP, LLP, and diagonal
p econdi ione s in PCG Me hod on meshes wi h a ying elemen coun s and
bounda y laye co e age. Residual is plo ed agains he numbe o i e a ions.
Simula ions execu ed on 384 p ocesses.
85
4.4. Nume ical e i ica ion o he sol e
quo ien be ween he esidual a he i- h i e a ion and he RHS equals 10
−6
,
see equa ion (4.1)), we ob ain he g aphs shown in Figu e 4.2. I is impo an
o no e ha GLP equi es only 534 i e a ions o each he desi ed ole ance,
whe eas he LLP equi es a leas 682 i e a ions, and he diagonal p econdi ione
equi es be ween 3043 and 3046 i e a ions. The e o e, he p oposed me hod
educes he numbe o i e a ions by a ac o o a leas 1.27 compa ed o he
LLP and a ac o o 5.69 compa ed o he diagonal p econdi ione . I is also
no ewo hy o men ion ha bo h LLP and GLP exhibi no ably accele a ed
con e gence compa ed o he scena io whe e only 60% o elemen s a e wi hin
he bounda y laye (see Fig. 4.2a). On he o he hand, he con e gence o
he LLP wo sens when he numbe o p ocesses inc eases. Fu he mo e, his
imp o emen has been obse ed o inc ease o highe aspec a ios o meshes
wi h a la ge numbe o elemen s.
Howe e , e en hough he GLP akes a lowe amoun o i e a ions o
con e ge o a desi ed ole ance, i should also be aken in o accoun ha each
i e a ion akes mo e CPU ime since mo e ope a ions (mainly communica ions)
a e equi ed. Thus, o he cases p e iously shown in Figu e 4.1, Figu e 4.3
shows he con e gence o he esidual a e e y PCG i e a ion s. execu ion
ime. Figu e 4.3a shows he con e gence in ime o he case p e iously shown
in Figu e 4.1a, and Figu e 4.3b shows he same ela ion o he case p e iously
shown in Figu e 4.1b. No e how in hese cases he GLP cu e aligns closely
wi h LLP due o he longe du a ion o each GLP i e a ion.
Addi ionally, he da a p esen ed in Figu es 4.3a and 4.3b can be exp essed
in e ms o execu ion ime equi ed o each a speci ic ole ance. This is
demons a ed in Figu es 4.4a and 4.4b, which show he ime pe i e a ion o
each p econdi ione . Figu e 4.4a indica es ha he diagonal p econdi ione
has he sho es ime pe i e a ion, ollowed by LLP, and inally, GLP has he
longes ime. Howe e , he o al ime equi ed o achie e a ce ain ole ance
is de e mined by he ime pe i e a ion and he numbe o i e a ions needed.
In mos cases, he numbe o i e a ions equi ed ollows an in e se ela ion,
wi h GLP equi ing he leas numbe o i e a ions, ollowed by LLP, and inally,
he diagonal p econdi ione . As a esul , hese wo ac o s de e mine which is
he as es p econdi ione . Figu e 4.4b shows he o al ime equi ed by he
PCG sol e o con e ge o a esidual o 10
−6
. No e ha GLP akes less ime o
achie e he desi ed solu ion since i e a ions a e signi ican ly educed in his case.
I is impo an o no e ha , o all he s udied cases, GLP always educed
he numbe o i e a ions equi ed o each a ce ain ole ance. Ne e heless,
his does no always mean an imp o emen in ime. Le us show o he cases
whe e GLP ep esen s li le o no imp o emen in ime and one whe e i does
ep esen a signi ican imp o emen . Figu e 4.5a compa es he esidual agains
he numbe o i e a ions o he h ee p econdi ione s unde s udy: Diagonal,
LLP, and GLP, o a hyb id mesh o 1.64 million elemen s whe e 40% o
hem a e wi hin he bounda y laye and using 384 p ocesses. No e how GLP
imp o es he con e gence: Fo all he s udied cases, i always akes he lowes
numbe o i e a ions. Ne e heless, i we compa e he ime equi ed o con e ge,
we see ha he e is li le o no imp o emen a all. This is shown in Figu e
4.5b. Hence, e en hough GLP con e ges in a lowe amoun o i e a ions,
86
4.4. Nume ical e i ica ion o he sol e
(a) Case un on a mesh wi h 1.64M elemen s, 60% o hem
wi hin he bounda y laye .
(b) Case un on a mesh wi h 0.98M elemen s, all wi hin he
bounda y laye .
Figu e 4.2: Compa ison o he numbe o i e a ions equi ed o each a ole ance
o 10
−6
o GLP, LLP, and diagonal p econdi ione on meshes wi h a ied
elemen coun s and bounda y laye co e age.
87
4.4. Nume ical e i ica ion o he sol e
(a) Con e gence o he esidual in ime o he case shown
in Figu e 4.2a. GLP achie es a esidual o 10
−6
1.17 imes
as e han LLP and 2.07 imes as e han he Diagonal
p econdi ione .
(b) Con e gence o he esidual in ime o he case shown
in Figu e 4.2b. GLP achie es a esidual o 10
−6
2.12 imes
as e han LLP and 4.08 imes as e han he Diagonal
p econdi ione .
Figu e 4.3: Compa ison o he esidual con e gence in ime be ween GLP, LLP,
and diagonal p econdi ione . Simula ions execu ed on 384 p ocesses.
88
4.4. Nume ical e i ica ion o he sol e
(a) The diagonal p econdi ione exhibi s he sho es ime
pe i e a ion, ollowed by LLP. As expec ed, GLP akes he
longes ime pe i e a ion.
(b) Time equi ed o each a ole ance o 10
−6
o he GLP,
LLP , and diagonal p econdi ione . GLP equi es less ime
due o i s as e con e gence.
Figu e 4.4: Compa ison o i e a ion ime and o al con e gence ime o GLP,
LLP, and diagonal p econdi ione s in achie ing 10−6 ole ance.
89
4.4. Nume ical e i ica ion o he sol e
(a) Residual s. he numbe o i e a ions. GLP achie es
con e gence in he lowes numbe o i e a ions among all
es ed p econdi ione s: 10
−6
1.40 imes as e (in e ms o
he numbe o i e a ions) han LLP and 3.80 imes as e
han he Diagonal p econdi ione .
(b) Con e gence o he esidual o e ime. In his case, GLP
o e s li le o no imp o emen o e he LLP. GLP eaches a
esidual o 10−61.07 imes as e han LLP and 1.81 imes
as e han he Diagonal p econdi ione .
Figu e 4.5: Residual con e gence compa ison in PCG i e a ions and ime: GLP,
LLP, and diagonal p econdi ione o a mesh o wi h 1.64 million elemen s o
40% o hem loca ed wi hin he bounda y laye . The simula ion was un in 384
pa allel p ocesses.
he ime equi ed o con e ge is almos he same as in he LLP case. These
obse a ions emphasize he ade-o be ween i e a ions and ime o di e en
p econdi ioning me hods.
To u he unde s and he beha io o he p oposed me hod, i we un he
same case as be o e bu his ime in 768 p ocesses, we ha e he esul s shown
in Figu e 4.6. Figu e 4.6a shows he same beha io as he one shown in Figu e
90
4.4. Nume ical e i ica ion o he sol e
(a)
(b)
Figu e 4.10: Imp o emen in he p econdi ioned esidual con e gence ime o
di e en ole ances and pe cen ages o elemen s wi hin he bounda y laye . The
colo scheme is he same as in Figu e 4.9
97
4.5. Analysis o pe o mance
wi h he numbe o pa i ions, deg ading he p econdi ione and consequen ly
inc easing he numbe o i e a ions equi ed o each a ce ain ole ance. This
is consis en wi h ou expe ience, whe e we ha e consis en ly obse ed ha
GLP is he p e e ed op ion o high aniso opic meshes wi h a high pe cen age
o elemen s wi hin he bounda y laye .
Two main conside a ions should be no ed:
•
Based on ou expe ience, a "high" pe cen age o elemen s wi hin he
bounda y laye is ypically conside ed o be 40%. The ime imp o emen
becomes mo e signi ican o meshes wi h mo e han 40% o elemen s in he
bounda y laye . Beyond 60%, he imp o emen is e en mo e p onounced.
I he expec ed imp o emen is small, implemen ing GLP may no be
wo hwhile. Fo example, in Figu e b) o Figu e 4.9, o a mesh wi h
50% o he elemen s in he bounda y laye and a ole ance o 10
−3
, GLP
equi es 92.1% o he ime needed by LLP. In his case, he imp o emen
migh no jus i y he addi ional compu a ional e o .
•
The numbe o pa i ions is no he only ac o in luencing pe o mance—
he quali y o he mesh pa i ioning ela i e o he linele s also plays
a c ucial ole. Fo example, in a high-pa i ion scena io, he mesh
may be pa i ioned such ha LLP disca ds ewe couplings, esul ing in
as e con e gence. Con e sely, o low-pa i ion scena ios, i he mesh is
pa i ioned so ha he numbe o couplings ha LLP disca ds is high,
his can lead o slowe con e gence.
These obse a ions suppo he idea ha he p oposed me hod p esen s a
p omising al e na i e o he s a e-o - he-a linele p econdi ione s, pa icula ly
o high aniso opic meshes wi h a la ge p opo ion o elemen s in he bounda y
laye . Howe e , he imp o emen in ime is mos no able o high ole ances
and meshes wi h a signi ican bounda y laye pe cen age. Addi ionally, he
numbe o pa i ions and he quali y o mesh pa i ioning a e key ac o s ha
in luence he sol e ’s pe o mance.
4.5 Analysis o pe o mance
This sec ion p esen s an analysis o he pe o mance o he p econdi ione .
We s a by analyzing he execu ion o he sol e by inspec ing he aces.
This p o ides a clea ep esen a ion o he wo kload dis ibu ion and he
o e head in oduced by he p oposed me hod. We hen s udy he pe o mance
o he p oposed me hod by using s anda d pe o mance me ics. The analysis
ocuses on speed-up, load balance, communica ion e iciency, and o e all pa allel
pe o mance.
4.5.1 Execu ion Analysis
The execu ion and implemen a ion o he p oposed me hod we e analyzed
using Ex ae, a pe o mance analysis ool used o p obe and egis e pa allel
execu ions. Ex ae gene a es ace iles ha can be isualized using Pa a e , a
98
4.5. Analysis o pe o mance
Figu e 4.11: T ace o he pa allel execu ion, showing he ime spen in each
s ep o he PCG o he second i e a ion o he i s ime s ep.
pe o mance analysis and isualiza ion ool. Bo h ools we e de eloped a he
Ba celona Supe compu ing Cen e .
The analysis was conduc ed on a ca i y low p oblem wi h a mesh o 340000
elemen s, wi h 50% o hem wi hin he bounda y laye . The simula ion was
execu ed using 32 p ocesses, wi h a pa ame e
Np
= 100, such ha e e y linele
slice is included in he global linele s uc u e. The slices we e g own wi h he
algeb aic me hod and a pa ame e
α
= 2. The esul s o he second i e a ion
o he i s ime s ep a e shown in Figu e 4.11. Fo he PCG ope a ions, we use
he same no a ion as in sec ion 1.5.3.
A key aspec o no e is ha he ime spen in SpM and do p oduc
compu a ions includes he communica ion equi ed o hei execu ion. This
explains why he do p oduc
Ts
akes signi ican ly longe o ce ain p ocesses:
while some ha e al eady comple ed he p econdi ioning s ep, o he s a e s ill
in he p ocess o inishing i . S ep 6 (communica ion o in e ace alues) is
subdi ided in o wo phases: mapping da a o he sending bu e and pe o ming
he ac ual communica ion. Addi ionally, in Alya, he i s h ead co esponds o
he mas e and does no pa icipa e in p econdi ione compu a ions. The e o e,
i is excluded om he p esen analysis.
99
4.5. Analysis o pe o mance
Figu e 4.12: Zoomed-in ace o he p econdi ione execu ion o he second
i e a ion, co esponding o he whi e ec angle in Figu e 4.11.
Ano he impo an obse a ion is he load imbalance be ween p ocesses 2–16
and 17–32, which esul s om he non-uni o m dis ibu ion o he bounda y
laye . P ocesses handling a la ge po ion o he bounda y laye con ain a highe
concen a ion o linele s. I a p ocess has linele s, i applies he linele p econdi-
ione ; o he wise, i eso s o he diagonal p econdi ione . Since he linele
p econdi ione equi es mo e compu a ions han he diagonal one, p ocesses
wi h a highe densi y o linele s ake longe o comple e he p econdi ioning s ep.
To u he illus a e his, Figu e 4.12 p o ides a zoomed-in iew o he
whi e ec angle in Figu e 4.11, highligh ing he ime spen in each s ep o he
p econdi ione . The igu e con as s he execu ion o a p ocess wi h linele s
( h ead 16) agains one wi hou linele s ( h ead 17). F om Figu es 4.11 and 4.12,
i can be seen ha he wo kload associa ed wi h sol ing o he in e ace (s ep
5) is e enly dis ibu ed ac oss all p ocesses. Howe e , a e he communica ion
s ep (s ep 6), h ead 17 p oceeds di ec ly o he diagonal p econdi ione , while
h ead 16 sol es he linea sys em associa ed wi h he inne nodes o i s slices.
In ou app oach, his las s ep is done by i e a ing o e he slices. Du ing each
i e a ion, he p ocess compu es he .h.s.
bp−Apsxs
(s ep 7), sol es o he
inne nodes (s ep 8), and maps he solu ion back o i s o iginal o de ing (s ep
10). Figu e 4.12 also p o ides a mo e de ailed iew o he i e a i e p ocess used
o sol e o he inne nodes wi hin he slices.
100
4.5. Analysis o pe o mance
4.5.2 Scalabili y Analysis and Pe o mance Me ics
Gi en he complexi y o he p oblems add essed in his wo k, compu a ional
e iciency is c i ical o minimizing ene gy consump ion, cos s, and p ocessing
ime. Thus, we mus ensu e ha he implemen a ion scales well wi h he
numbe o CPU co es alloca ed o he ask. To his end, we analyze di e en
pe o mance me ics o he sol e .
Le
Np e
deno e he minimum numbe o co es used in a pe o mance es ,
wi h he co esponding elapsed ime gi en by
T e
. I he same p oblem is
execu ed on
Np
co es wi h an elapsed ime o
TNp
, he ela i e speed-up is
de ined as:
SU(T) = T e
TNp
(4.2)
In ou case,
TNp
is conside ed o be he maximum elapsed ime ac oss all
p ocesses. We also conside he speed-up o he mean execu ion ime ac oss all
p ocesses, which is gi en by eplacing
TNp
by
TNp
=
⟨Ti⟩Np
i=1
in equa ion
(4.2)
:
SU(⟨T⟩) = T e
⟨Ti⟩Np
i=1
(4.3)
Howe e , a high speed-up does no gua an ee e icien esou ce u iliza ion.
To assess his, h ee key me ics p o ide a mo e in-dep h analysis: load balance,
communica ion e iciency, and pa allel e iciency.
Load balance quan i ies how e enly he compu a ional load is dis ibu ed
among di e en p ocessing uni s, he e o e educing bo lenecks due o idle
p ocesses. I is compu ed as he a io be ween a e age use ul compu a ion ime
ac oss all p ocesses,
Tw
=
⟨Ti
w⟩Np
i=1
, and he maximum use ul compu a ion ime
(also ac oss all p ocesses), Tmax
w= maxNp
i=1 Ti
w:
LB =Tw
Tmax
w
(4.4)
On he o he hand, communica ion e iciency is he maximum, ac oss all
p ocesses, o he a io be ween use ul compu a ion ime and o al un ime.
I measu es he p opo ion o ime spen on compu a ions ela i e o
communica ion o e head. I is de ined as:
CE =Tmax
w
TNp
(4.5)
Poo communica ion e iciency iden i ies cases whe e excessi e ime is spen
on communica ion a he han execu ing use ul compu a ions. Two p ima y
ac o s in luence his me ic: 1) synch oniza ion delays, whe e p ocesses e-
main idle a communica ion poin s due o wo kload imbalance, and 2) da a
ans e o e head, which a ises when p ocesses exchange la ge olumes o da a
ela i e o he ne wo k’s ne wo k capaci y o wi h high- equen communica ion.
Finally, pa allel e iciency assesses he o e all u iliza ion o compu a ional
esou ces o achie e his speed-up. I is exp essed as:
101
4.5. Analysis o pe o mance
PE =Tw
TNp
=LB ×CE (4.6)
No e ha he pa allel e iciency can be compu ed as he p oduc o load
balance and communica ion e iciency. Thus, he impo ance o minimizing
communica ion o e head and ensu ing a balanced wo kload o u ilize esou ces
in pa allel compu ing e icien ly.
In o de o e alua e he pe o mance o he sol e , we conduc ed a se ies o
es s on a la ge-scale, uns eady, incomp essible low p oblem wi h a Reynolds
numbe o
Re
= 10
5
. The simula ions we e execu ed using 56 o 1344 p ocesses,
wi h he pa ame e
Np
(discussed in sec ion 3) selec ed o each case o ensu e
he inclusion o e e y linele slice. Impo an ly, as he numbe o p ocesses
inc eased, all p oblem pa ame e s emained cons an , allowing us o conduc a
obus , s ong scalabili y es . Fo he ollowing esul s, we used a ca i y low
p oblem wi h meshes o 2.25 million elemen s wi h 40% o 90% o elemen s
wi hin he bounda y laye . No e ha mo e elemen s wi hin he bounda y laye
imply 1) la ge linele s and/o 2) mo e linele s. Since he main aspec o he
sol e we a e discussing is he implemen a ion o he newly de eloped p econdi-
ione , in his sec ion, we only conside compu a ion and communica ion imes
spen wi hin he p econdi ione .
Figu es 4.13 and 4.14 show he pa allel pe o mance analysis o he ca i y
low p oblem wi h 40% o elemen s wi hin he bounda y laye . The esul s a e
p esen ed as a unc ion o he numbe o p ocesses, wi h he ollowing me ics:
speed-up, mean ime speed-up-, load balance, and pa allel e iciency. Ideally,
he speed-up should align wi h he e e ence dashed ed line, which ep esen s
pe ec linea scalabili y. Any de ia ion om his ideal beha io indica es
ine iciencies s emming om communica ion o e head and wo kload imbalance.
Simila ly, communica ion e iciency should ideally emain close o 1, bu i
na u ally deg ades as in e -p ocess communica ion cos s inc ease. A pe ec ly
balanced wo kload would yield a load balance alue o 1, while lowe alues
indica e dispa i ies in compu a ional wo kload dis ibu ion ac oss p ocesses.
Finally, pa allel e iciency in eg a es he e ec s o speed-up, communica ion
e iciency, and load balance, o e ing a comp ehensi e measu e o how e ec i ely
he p econdi ione scales ac oss mul iple p ocesses. Table 4.1 p o ides a
compa ison o he LLP and GLP pe o mance me ics. The esul s show ha
o mos cases, he GLP ou pe o ms he LLP in e ms o speed-up, mean ime
speed-up, and load balance while main aining a simila pa allel e iciency. The
esul s o he diagonal p econdi ione a e shown as a e e ence o compa ison.
The main eason o he speed-up imp o emen in he GLP o e he LLP is
ha he wo k is sp ead mo e e enly among p ocesses, as shown by he load
balance me ic. This is pa icula ly impo an in he con ex o he p esen
sol e , as he p econdi ione is usually he mos compu a ionally expensi e
pa o he simula ion. In he LLP, he p ocesses con aining slices mus sol e
a TDMA, while he es o he p ocesses apply a do p oduc ela ed o he
diagonal p econdi ione . This leads o idle p ocesses and a poo load balance.
In con as , he GLP allows all p ocesses o dis ibu e he wo k mo e e enly
102
4.5. Analysis o pe o mance
since no only he p ocesses con aining slices need o sol e a TDMA, bu also
he p ocesses in cha ge o sol ing o he in e ace nodes need o apply a TDMA.
Howe e , o a la ge numbe o p ocesses, he communica ion o e head may
become a bo leneck, as e idenced by he dec ease in he speed-up and pa allel
e iciency wi h espec o he LLP case.
Also, no e ha he quali y o he mesh pa i ion is c ucial o he pe o m-
ance o he GLP, as he communica ion scheme is de e mined by he linele
pa i ions, which a e de e mined by he mesh pa i ion. The e o e, a poo load
balance in he linele p econdi ione (LLP o GLP) can be a consequence o
a mesh pa i ion ha cu s he linele s in a way ha he slices a e no e enly
dis ibu ed among he p ocesses. This is he eason why we compa e he LLP
and GLP pe o mance since he GLP is a gene aliza ion o he LLP ha allows
communica ion be ween slices. The e o e, he ac ha he GLP ou pe o ms
he LLP in mos cases is an indica o o 1) a good implemen a ion o he
GLP and 2) ha he p esen sol e is able o ake ad an age o he GLP
communica ion scheme. Finally, no e ha since he diagonal and he LLP does
no equi e communica ions, hei communica ion e iciency is always 1, and
he e o e hei load balance and pa allel e iciency a e he same.
P LLP GLP
SU(T)SU(⟨T⟩)LB/PE SU(T)SU(⟨T⟩)LB CE PE
56 1.00 1.00 0.42 1.00 1.00 0.44 0.89 0.95
112 1.55 1.82 0.35 1.80 1.92 0.41 0.90 0.91
168 2.31 2.60 0.37 2.52 2.97 0.38 0.91 0.89
224 2.70 3.29 0.35 3.68 3.92 0.40 0.94 0.84
336 3.56 4.60 0.30 4.09 5.89 0.33 0.90 0.81
448 4.02 5.65 0.30 5.07 7.85 0.33 0.88 0.74
672 5.84 8.29 0.29 6.75 11.90 0.32 0.87 0.76
896 7.20 10.26 0.29 7.62 15.92 0.30 0.86 0.64
1344 9.20 14.43 0.27 9.24 23.85 0.29 0.78 0.52
Table 4.1: Compa ison o LLP and GLP pe o mance me ics o a ca i y low
p oblem wi h 2.25 million elemen s, 40% o hem wi hin he bounda y laye .
P e e s o he numbe o p ocesses.
A mo e de ailed analysis o he GLP ’s pe o mance is p esen ed in Figu es
4.15, 4.16 and 4.17, whe e we also include he communica ion e iciency. I is
obse ed ha communica ion e iciency dec eases as he numbe o p ocesses
inc eases, wi h his e ec being mo e p onounced o meshes con aining a highe
pe cen age o nodes wi hin he bounda y laye . This beha io is a ibu ed
o he ising communica ion o e head, which escala es wi h he numbe o
p ocesses and he numbe and leng h o he linele s. As he numbe o in e ace
nodes o be communica ed g ows, so does he associa ed o e head. This is also
e lec ed in he minimal o no imp o emen in speed-up and he poo pa allel
e iciency obse ed when unning on a la ge numbe o p ocesses.
Howe e , i is impo an o highligh ha he load balance emains con-
103
4.5. Analysis o pe o mance
(a) Speed-up
(b) Mean ime speed-up
Figu e 4.13: Speed-up as a unc ion o he numbe o p ocesses o a ca i y low
p oblem wi h 2.25 million elemen s, 40% o hem wi hin he bounda y laye .
104
4.5. Analysis o pe o mance
(a) Load balance
(b) Pa allel e icency
Figu e 4.14: Load balance and pa allel pe o mance as a unc ion o he numbe
o p ocesses o a ca i y low p oblem wi h 2.25 million elemen s, 40% o hem
wi hin he bounda y laye .
105
4.6. P ep ocessing s age
(a) Speed-up
Figu e 4.15: Speed-Up o he GLP as a unc ion o he numbe o p ocesses
o a ca i y low p oblem wi h 2.25 million elemen s Di e en pe cen ages o
elemen s wi hin he bounda y laye .
sis en ly main ained ac oss all numbe s o p ocesses, imp o ing wi h a highe
pe cen age o elemen s wi hin he bounda y laye . Two key ac o s explain
his: 1) as he pe cen age o elemen s wi hin he bounda y laye inc eases,
so does he numbe o nodes wi hin he linele s, and 2) he communica ion
s a egy—whe e each linele is assigned o a speci ic p ocess o handle i s
in e ace nodes— esul s in a mo e e en dis ibu ion o he load as he numbe
o linele nodes inc eases.
4.6 P ep ocessing s age
The p ep ocessing s age is he i s s ep in he sol e ’s wo k low. Du ing
his s age, he inpu da a is ead, memo y is alloca ed, da a s uc u es a e
ini ialized, and he communica ion scheme is de ined. I he cos o his s ep
is oo high, any po en ial ad an ages gained in sol ing he p oblem may be
diminished o e en los . The e o e, minimizing he cos o his s age is equi ed
o ensu e o e all e iciency.
Be o e analyzing GLP ’s con e gence, we compa e he p ep ocessing ime
o GLP and LLP using he same cases discussed in Sec ion 4.5. Figu e 4.18
p esen s he a io o p ep ocessing ime o GLP ela i e o LLP. Unlike LLP,
GLP equi es addi ional ope a ions, such as assembling he local linea sys em,
compu ing con ibu ions o he in e ace sys em, mapping hese con ibu ions,
and handling communica ion. The e o e, p ep ocessing akes longe o GLP,
wi h i s ime inc easing as he numbe o domain pa i ions g ows. The igu e
indica es ha GLP ’s p ep ocessing ime anges be ween 1.34 and 4.33 imes
ha o LLP .
106
4.7. Conclusions
Re e ences
[1]
Had ien Calme , Albe o M. Gamba u o, Alis e J. Ba es, Ma iano
Vázquez, Guillaume Houzeaux and Denis J. Doo ly. ‘La ge-scale CFD
simula ions o he ansi ional and u bulen egime o he la ge human
ai ways du ing apid inhala ion’. In: Compu e s in Biology and Medicine
69 (2016), pp. 166–180. issn: 0010-4825. doi:h ps://doi.o g/10.1016/j.
compbiomed.2015.12.003.u l:h ps://www.sciencedi ec .com/science/
a icle/pii/S0010482515003881.
[2]
R. de Olazábal, R. Bo ell and O. Lehmkuhl. ‘An algeb aic global
linele p econdi ione o incomp essible low sol e s’. In: Jou nal o
Compu a ional Physics 514 (2024), p. 113237. issn: 0021-9991. doi:h ps:
//doi.o g/10.1016/j.jcp.2024.113237.u l:h ps://www.sciencedi ec .com/
science/a icle/pii/S0021999124004868.
[3]
S. Gö e , D. Mi a, M. Za ala-Ake, J.B.W. Kok, M. Vázquez and G.
Houzeaux. ‘Hea loss p edic ion o a con ined p emixed je lame using a
conjuga e hea ans e app oach’. In: In e na ional Jou nal o Hea and
Mass T ans e 107 (2017), pp. 882–894. issn: 0017-9310. doi:h ps:
/ / doi . o g / 10 . 1016 / j . ijhea mass ans e . 2016 . 10 . 122.u l:h ps :
//www.sciencedi ec .com/science/a icle/pii/S0017931016317495.
[4]
O. Lehmkuhl, G. Houzeaux, H. Owen, G. Ch ysoken is and I. Rod iguez.
‘A low-dissipa ion ini e elemen scheme o scale esol ing simula ions o
u bulen lows’. In: Jou nal o Compu a ional Physics 390 (2019), pp. 51–
65. issn: 0021-9991. doi:h ps://doi.o g/10.1016/j.jcp.2019.04.004.u l:
h ps://www.sciencedi ec .com/science/a icle/pii/S0021999119302372.
[5]
Al onso San iago, Miguel Za ala-Aké, Rica d Bo ell, Guillaume Houzeaux
and Ma iano Vázquez. ‘HPC compac quasi-New on algo i hm o in e ace
p oblems’. In: Jou nal o Fluids and S uc u es 96 (2020), p. 103009.
issn: 0889-9746. doi:h ps : / / doi . o g / 10 . 1016 / j . j luids uc s . 2020 .
103009.u l:h ps : / / www. sciencedi ec . com / science / a icle / pii /
S088997461930934X.
[6]
O lando So o, Rainald Lohne and Fe nando Camelli. ‘A Linele
p econdi ione o incomp essible low sol e s’. In: In e na ional Jou nal
o Nume ical Me hods o Hea & Fluid Flow - INT J NUMER
METHOD HEAT FL F 13 (Feb. 2003), pp. 133–147. doi:10.1108/
09615530310456796.
[7]
Ma iano Vázquez, Guillaume Houzeaux, Seid Ko ic, An oni A igues,
Jazmin Aguado-Sie a, Ru h A ís, Daniel Mi a, Had ien Calme , Fe nando
Cucchie i, He be Owen, Ahmed Taha, E an De ing Bu ness, José Ma ía
Cela and Ma eo Vale o. ‘Alya: Mul iphysics enginee ing simula ion owa d
exascale’. In: Jou nal o Compu a ional Science 14 (2016). The Rou e
o Exascale: No el Ma hema ical Me hods, Scalable Algo i hms and
Compu a ional Science Skills, pp. 15–27. issn: 1877-7503. doi:h ps:
//doi.o g/10.1016/j.jocs.2015.12.007.u l:h ps://www.sciencedi ec .
com/science/a icle/pii/S1877750315300521.
113
CHAPTER 5
Nume ical Applica ions – Sol ing
Real-Wo ld Scena ios
Pa o he con en s o his chap e ha e been published as:
R. de Olazábal, R. Bo ell and O. Lehmkuhl. ‘An algeb aic global
linele p econdi ione o incomp essible low sol e s’. In: Jou nal o
Compu a ional Physics 514 (2024),
and dissemina ed in he ollowing con e ences:
he 32nd Pa allel Compu a ional Fluid Dynamics Con e ence, Nice,
F ance, 2021,
he 15 h JLESC Wo kshop, Bo deaux, F ance, 2023,
he 1s Ma h 2 P oduc (M2P) con e ence o Eme ging Technologies
in Compu a ional Science o Indus y, Sus ainabili y and Inno a ion,
Tao mina, Sicily, 2023, and
he 23 d IACM Compu a ional Fluids Con e ence, San iago de Chile,
2025.
In his chap e , we assess he pe o mance o he GLP p econdi ione in
h ee la ge-scale compu a ional luid dynamics (CFD) simula ions. These
cases ep esen h ee eal-wo ld low p oblems in ol ing di e en geome ies,
Reynolds numbe s, and u bulence modeling app oaches. The objec i e is o
e alua e how GLP pe o ms in p ac ical scena ios, pa icula ly in e ms o
con e gence beha io , compu a ional cos , and pa allel e iciency.
The i s es case in es iga es he low o e he 30P30N h ee-elemen
high-li ai oil, a well-es ablished benchma k in ae odynamics. This simula ion
employs LES a a Reynolds numbe o
Rec
= 750
,
000 and in ol es a hyb id
uns uc u ed mesh wi h s uc u ed in la ion laye s a ound he ai oil su aces.
This con igu a ion has been ex ensi ely analyzed in [6]. The second case is a
DNS o low h ough he S an o d di use a
Re
= 10
,
000, which se es as a
canonical case o analyzing low sepa a ion and u bulence in a con ined domain.
I was ini ially in oduced by Che y e al. [1] and u he analyzed in [5].
114
5.1. 30P30N Ai oil
Finally, he hi d case examines he D i Ae model, a de ailed ep esen a ion
o a ealis ic au omobile, de eloped a he Ins i u e o Ae odynamics and Fluid
Mechanics a Technische Uni e si ä München (TUM) [2,3,4]. The simula ion
is pe o med a
Re
= 4
.
87
×
10
6
(based on ehicle leng h), using Wall-modelled
La ge Eddy Simula ion (WMLES). The D i Ae case is he la ges conside ed
in his s udy, comp ising 427 million elemen s and equi ing up o 14,000
compu a ional pa i ions. In all h ee cases, we employ an open in eg a ion
ule. Gi en ha GLP bene i s signi ican ly om a geome ic app oach (see,
o example, Figu e 1.4), we op ed o a geome ic-based linele cons uc ion
me hod in hese simula ions.
The chap e is o ganized as ollows. Sec ions 5.1, 5.2, and 5.3 p esen
de ailed analyses o he 30P30N ai oil, S an o d di use , and D i Ae model
cases, espec i ely. Each sec ion includes an e alua ion o sol e con e gence,
compu a ional cos , and he impac o GLP on nume ical s abili y. These
analyses compa e he pe o mance o GLP agains he LLP and he diagonal
p econdi ione .
Following his, Sec ion 5.4 p o ides a comp ehensi e pe o mance analysis,
compa ing p ep ocessing and p econdi ioning cos s ac oss all h ee cases. He e,
we highligh he communica ion o e head and load balance e ec s ha in luence
o e all compu a ional e iciency.
Finally, Sec ion 5.5 p esen s a de ailed scalabili y s udy, examining speed-up,
communica ion e iciency, load balance, and pa allel e iciency. This analysis
ex ends he indings om Chap e 4, ein o cing key insigh s in o how GLP
scales ac oss di e en p oblem sizes and pa i ion coun s.
The esul s ob ained in his chap e se e as a alida ion o he GLP
me hod in la ge-scale CFD applica ions. While signi ican imp o emen s in
con e gence a e demons a ed, ade-o s ela ed o p ep ocessing ime and
communica ion o e head a e also iden i ied. These indings p o ide guidance
o u u e op imiza ions, pa icula ly in hyb id p econdi ioning s a egies and
domain decomposi ion echniques. The chap e concludes wi h a discussion o
hese insigh s and hei implica ions o he e icien solu ion o eal-wo ld luid
dynamics p oblems.
5.1 30P30N Ai oil
Be o e analyzing o e all con e gence ends, we i s examine he esidual
e olu ion du ing each i e a ion o he P econdi ioned Conjuga e G adien
(PCG) me hod o he high-li wing case a he i s ime s ep. Figu e 5.1
p esen s he esidual con e gence o he diagonal, LLP, and GLP p econdi ion-
e s, wi h he la e wo e alua ed using bo h geome ic and algeb aic app oaches.
A key obse a ion om Figu e 5.1 is ha while he algeb aic GLP exhibi s a
signi ican imp o emen o e he algeb aic LLP, i s ad an age o e he diagonal
p econdi ione diminishes a e a esidual o 3
×
10
−4
, beyond which he diagonal
115
5.1. 30P30N Ai oil
Figu e 5.1: Residual con e gence o e ime o he PCG me hod applied o he
30P30N case a
Rec
= 750000. The igu e compa es he algeb aic and geome ic
app oaches o cons uc ing he linele s. The simula ion was pe o med on 1680
p ocesses.
p econdi ione becomes he as es . This beha io is consis en wi h Figu es 4.9
and 4.10, which illus a e ha he bene i s o GLP o e diagonal and LLP
p econdi ione s become mo e p onounced when he p opo ion o elemen s
wi hin he bounda y laye exceeds 40%. In his case, app oxima ely 43% o he
elemen s lie wi hin he bounda y laye , meaning he expec ed imp o emen is
ela i ely mode a e.
This obse a ion ein o ces he indings om p e ious analyses—GLP is pa -
icula ly e ec i e in scena ios wi h a high pe cen age o elemen s in he bounda y
laye . Fu he mo e, his esul demons a es ha he conclusions d awn om
sec ion emain alid e en o complex geome ies and highe Reynolds numbe s.
Ano he c ucial poin is he ad an age o he geome ic app oach o e he
algeb aic one. The geome ic GLP con e ges signi ican ly as e , pa icula ly a
lowe ole ances. Fo ins ance, i eaches a esidual ole ance o 10
−6
in 105.73
seconds, while he diagonal p econdi ione , he second as es me hod, equi es
202.80 seconds. This ansla es in o a speed-up o app oxima ely 1.92 imes,
o , con e sely, he geome ic GLP comple ing in jus 52% o he ime equi ed
by he diagonal p econdi ione . These indings align wi h Figu es 1.4 and 1.5,
u he highligh ing he supe io e iciency o he geome ic app oach.
5.1.1 Compu a ion and Communica ion Time Analysis
To be e unde s and he compu a ional cos o he GLP p econdi ione , we
analyze he mean ime pe PCG i e a ion equi ed o bo h compu a ion and
communica ion. This is compa ed agains he LLP p econdi ione as a unc ion
o he numbe o p ocesses. Figu e 5.2 p esen s he mean ime pe i e a ion
o di e en p ocess coun s: 896, 1344, 1792, 2688, and 3584. These alues
116
5.1. 30P30N Ai oil
Figu e 5.2: Mean ime pe i e a ion o he LLP and GLP p econdi ione s o
he 30P30N case a Rec= 750000.
ep esen he a e age o e all h eads.
These esul s indica e ha he communica ion s ep is he p ima y bo le-
neck in he GLP p econdi ione . Meanwhile, he compu a ion s ep akes a
mos wice he ime he LLP p econdi ione equi es. This end is u he
illus a ed in Figu e 5.4a, which shows he a io o he ime consumed by each
s ep o he GLP compa ed o he co esponding mean ime in he LLP app oach.
The inc eased cos o he GLP p econdi ione aligns wi h he addi ional
compu a ional s eps in ol ed (see Sec ion 2.2.2). Unlike LLP, he GLP app oach
explici ly sol es he p econdi ioning sys em o in e ace nodes. Fu he mo e,
in he GLP me hod, mo e one-node slices may be inco po a ed in o he linele
sys em. These slices a e simply ea ed wi h a diagonal p econdi ione in he
LLP app oach. Figu es 5.2 and 5.4a highligh ha , on a e age, he compu a ion
ime equi ed by GLP is app oxima ely wice ha o LLP.
Addi ionally, as expec ed, communica ion ime inc eases wi h he numbe
o p ocesses, sugges ing ha pa allel o e head becomes mo e p onounced a
la ge scales.
5.1.2 Compu a ional Cos pe PCG I e a ion
To u he analyze he pe o mance o he GLP p econdi ione , we examine he
ime aken pe i e a ion o he PCG me hod. Figu e 5.3 p esen s he o al ime
equi ed o a single PCG i e a ion o bo h LLP and GLP. This da a can be
compa ed wi h Figu e 5.4b, which illus a es he a io o he o al ime aken
by GLP ela i e o LLP. The esul s indica e ha he GLP p econdi ione is,
in his case, a mos 3.56 imes slowe han LLP.
117
5.1. 30P30N Ai oil
Figu e 5.3: Time pe i e a ion o he LLP and GLP p econdi ione s o he
30P30N case a Rec= 750000.
To gain u he insigh , Figu e 5.4a p o ides a b eakdown o he mean ime
pe i e a ion, dis inguishing be ween compu a ion and communica ion imes.
I also includes he a io o o al ime o GLP ela i e o LLP.
These esul s a e consis en wi h p e ious obse a ions (see Figu e 4.8).
The plo in Figu e 5.4a was compu ed by a e aging he imes o e all p ocesses,
p o iding an o e iew o he gene al beha io . A simila pa e n is obse ed in
Figu e 4.4, wi h he added dis inc ion ha compu a ion and communica ion
imes a e explici ly compa ed.
Howe e , his a e aged pe spec i e may no ully cap u e he o al compu a-
ional cos o GLP. The slowes p ocess dic a es he o e all ime in he p esence
o load imbalance. Gi en he na u e o he linele -based p econdi ione s
(GLP and LLP), load imbalance can a ise when linele s a e concen a ed in
speci ic egions o he mesh, and he mesh pa i ioning does no accoun o
he linele dis ibu ion. This e ec is illus a ed in Figu e 4.11. Fu u e wo k
can be done o conside load-balancing s a egies in la ge-scale implemen a ions.
On he o he hand, a simila analysis can be conduc ed o he p ep ocessing
s ep. Figu e 5.5 p esen s he a e age ime equi ed pe p ocess o comple e he
p ep ocessing s age o he LLP and GLP p econdi ione s. The esul s indica e
ha , on a e age, he compu a ional cos o he p ep ocessing s ep in GLP
is a mos 1.95 imes highe han ha o LLP. Howe e , he communica ion
o e head signi ican ly ampli ies he o e all cos , making he GLP p econdi ione
up o 7.35 imes mo e expensi e because o he communica ion ime.
Howe e , when compa ing he o al ime spen in he p ep ocessing s age,
he load imbalance p esen in his s age makes he GLP a mos 2.42 imes mo e
expensi e han he equi ed o he LLP. This sugges s ha while he addi ional
communica ion s eps equi ed by GLP inc ease he p ep ocessing cos , hei
118
5.1. 30P30N Ai oil
(a) Quo ien be ween he mean compu a ion and communica ion
imes o LLP and GLP, alongside he o al ime a io o GLP
o e LLP.
(b) To al ime quo ien o he LLP and GLP p econdi ione s.
Figu e 5.4: Compa ison o he mean ime pe i e a ion o he LLP and GLP
p econdi ione s in he 30P30N case.
119
5.1. 30P30N Ai oil
impac is pa ially mi iga ed by load-balancing e ec s ac oss p ocesses.
A e e i ying ha he beha io o he GLP p econdi ione aligns wi h he
esul s ob ained in he p e ious sec ions, we now analyze i s o e all impac on
he ime equi ed o sol e he Na ie -S okes equa ions.
Figu e 5.6 p esen s he ime pe ime s ep equi ed o sol e he Na ie -S okes
equa ions o he 30P30N case a
Rec
= 750000 using he diagonal, LLP, and
GLP p econdi ione s, conside ing 3584 pa i ions and ole ances o 10
2
and
10
6
. The esul s indica e ha when employing he GLP, he con e gence
demons a es he mos s able pe o mance, apidly eaching a s eady-s a e
con e gence ime wi h minimal luc ua ions.
In con as , he LLP p econdi ione consis en ly equi es mo e ime pe s ep
and exhibi s highe a iabili y. On he o he hand, he diagonal p econdi ione
shows la ge luc ua ions in con e gence ime, pa icula ly in he ea ly i e a ions.
This sugges s ha he GLP p econdi ione accele a es con e gence and enhances
nume ical s abili y in his speci ic scena io.
Ano he way o isualize hese esul s is o plo he cumula i e ime equi ed
o sol e he Na ie -S okes equa ions. Figu es 5.7a and 5.7b show he cumula i e
ime equi ed o pe o m
n
ime s eps o he cases p esen ed in Figu es 5.6a
and 5.6b. These esul s con i m ha he GLP p econdi ione is he mos
e icien , ollowed by he LLP.
A b oade analysis can be pe o med by a ying he numbe o p ocesses
and ole ances. Table 5.1 p esen s he a io o he ime equi ed by GLP o ha
equi ed by LLP o pe o m
n
ime s eps, whe e
n
is he las compa able ime
s ep. This concep is illus a ed in Figu e 5.8. Since he choice o p econdi ione
in he p essu e co ec ion equa ion in luences he o e all ime equi ed o sol e
he Na ie -S okes equa ions, he LLP and GLP cases may no comple e he
same numbe o ime s eps wi hin he gi en simula ion ime ame. Thus, we
ake he las compa able ime s ep eached by bo h p econdi ione s.
Fo ins ance, in Figu e 5.8, he LLP execu es 209 ime s eps, while he
GLP comple es 303. The e o e, we compu e he quo ien be ween he ime
equi ed by bo h cases o each 209 ime s eps. Fo his scena io, he GLP
akes only 29.6% o he ime equi ed by he LLP, demons a ing a subs an ial
imp o emen .
Table 5.1 u he con i ms ha he mos signi ican ad an ages o he GLP
a e ob ained when using lowe ole ances and a highe numbe o p ocesses.
This end is consis en wi h he indings in Sec ion 5.1 (see Figu es 4.9 and
4.10). Blank spaces in he able indica e missing da a.
120
5.1. 30P30N Ai oil
(a) Quo ien be ween he mean ime spen in compu a ion and
communica ion o he p ep ocessing s age in he LLP and GLP
p econdi ione s.
(b) To al ime quo ien o he p ep ocessing s age in he GLP
and LLP p econdi ione s.
Figu e 5.5: Compa ison o he p ep ocessing s age o he LLP and GLP
p econdi ione s in he 30P30N case.
121
5.1. 30P30N Ai oil
(a) Compa ison o he ime pe ime s ep o sol e he Na ie -S okes
equa ions o he 30P30N case wi h a ole ance o 10
2
in he p essu e
co ec ion equa ion and 3584 p ocesses.
(b) Same as Figu e 5.6a bu wi h a ole ance o 106.
Figu e 5.6: Compa ison o he ime pe ime s ep o sol ing he Na ie -
S okes equa ions o he 30P30N case when using he diagonal, LLP, and
GLP p econdi ione s.
122
5.3. D i Ae Model
Figu e 5.11: Time pe ime s ep equi ed o sol e he p essu e co ec ion
equa ion o he D i Ae case using 14,000 p ocesses and a ole ance o 10−5.
Figu e 5.12: Cumula i e ime equi ed o sol e he p essu e co ec ion equa ion
o he D i Ae case wi h 14,000 p ocesses and a ole ance o 10−5.
129
5.4. Pe o mance Analysis
Figu e 5.13: Cumula i e ime equi ed o sol e he Na ie -S okes equa ions o
he D i Ae case wi h 14,000 p ocesses and a ole ance o 10−5.
5.4 Pe o mance Analysis
In his sec ion, we e alua e he pe o mance o he GLP p econdi ione o he
h ee s udied cases, compa ing i agains he LLP app oach, which se es as he
benchma k. Since GLP is a gene aliza ion o LLP, any imp o emen obse ed
in GLP should be measu ed ela i e o LLP.
5.4.1 P ep ocessing S age
We begin by analyzing he p ep ocessing s age. Figu e 5.14 p esen s he a io
be ween he p ep ocessing ime equi ed o GLP and LLP ac oss all es cases.
As expec ed, he p ep ocessing ime scales wi h he p oblem size, wi h he
D i Ae model— he la ges case—exhibi ing he longes p ep ocessing ime.
No ably, o his case, communica ion o e head signi ican ly con ibu es o he
o al p ep ocessing ime, leading o an inc ease o o e an o de o magni ude
compa ed o LLP.
To be e unde s and his phenomenon, Figu e 5.15 illus a es he mean
p ep ocessing ime pe p ocess o he D i Ae case. He e, he communica ion
s ep is he dominan cos , con i ming ha communica ion o e head is he
p ima y bo leneck in he p ep ocessing s age.
On he o he hand, Figu e 5.16 p esen s he co esponding esul s o he
S an o d di use case. Unlike he D i Ae model, he p ep ocessing ime, in
his case, is mo e balanced be ween compu a ion and communica ion. This
is expec ed, as he S an o d di use is he smalles case s udied and in ol es
ewe pa i ions. The smalle numbe o pa i ions esul s in less communica ion
o e head, leading o a p ep ocessing ime close o ha o LLP.
These obse a ions a e consis en wi h he esul s discussed in Sec ion 4.6 o
Chap e 4, whe e we analyzed he p ep ocessing cos s o a ca i y low case. In
130
5.4. Pe o mance Analysis
Figu e 5.14: Compa ison o p ep ocessing imes o GLP and LLP ac oss he
h ee s udied cases.
Figu e 5.15: Mean p ep ocessing ime pe p ocess o he D i Ae case, showing
he con ibu ion o compu a ion and communica ion.
131
5.4. Pe o mance Analysis
Figu e 5.16: Mean p ep ocessing ime pe p ocess o he S an o d di use case,
highligh ing he balanced cos be ween compu a ion and communica ion.
ha analysis, we ound ha p ep ocessing ime exhibi ed signi ican a iabili y
due o communica ion o e head, which was s ongly in luenced by he domain
pa i ioning. The cu en esul s ein o ce hese indings: while compu a ion
ime emains ela i ely s able ac oss di e en cases, he communica ion cos is
he main ac o a ec ing p ep ocessing e iciency.
As p e iously no ed in Chap e 4, op imizing he domain pa i ioning
s a egy is one po en ial a enue o imp o ing p ep ocessing pe o mance. Since
GLP is agnos ic o mesh pa i ioning, an op imized pa i ioning app oach
ha minimizes communica ion o e head while main aining load balance could
signi ican ly enhance pe o mance. This is especially ele an o la ge-scale
cases like D i Ae , whe e communica ion cos s domina e. Fu u e wo k could
explo e pa i ioning s a egies ha ake in o accoun he spa ial dis ibu ion o
linele s, ensu ing ha linele s a e no excessi ely agmen ed ac oss pa i ions.
5.4.2 P econdi ioning S ep
In his sec ion, we compa e he compu a ional cos o he p econdi ioning s ep
o he h ee s udied cases. Figu e 5.17 p esen s he quo ien o he o al ime
spen in he p econdi ioning s ep o each case.
I can be obse ed ha he di e ence in compu a ional cos be ween GLP
and LLP is less p onounced in his s ep compa ed o he p ep ocessing s age.
The o e head emains wi hin he same o de o magni ude as he LLP case,
ein o cing he scalabili y o he GLP me hod. Howe e , i is impo an o
highligh ha he mos compu a ionally expensi e case is he 30P30N ai oil,
which, pa adoxically, exhibi s he bes con e gence imp o emen s. This is
because a comp omise is made be ween o e cos in ime and imp o emen in
con e gence. Depending on he case, one may ou pe o m he o he . This
aligns wi h he esul s shown in Figu e 5.1, demons a ing ha he addi ional
cos pe i e a ion o he GLP does no necessa ily co ela e wi h ine iciency.
132
5.4. Pe o mance Analysis
Figu e 5.17: Compa ison o he mean ime pe p ocess in he p econdi ioning
s ep.
Figu e 5.18: Mean ime pe p ocess in he compu a ion and communica ion
s eps o he S an o d di use case.
Figu es 5.18 and 5.19 p o ide addi ional insigh s by b eaking down he
p econdi ioning s ep in o compu a ion and communica ion componen s o he
S an o d di use and D i Ae cases, espec i ely.
These igu es show ha in he di use and D i Ae cases, he communica ion
s ep does no domina e he o e all cos as much as in he 30P30N case. Despi e
his, in he o me wo cases, he GLP me hod does no signi ican ly educe
i e a ions compa ed o LLP, sugges ing ha he unde lying p oblem cha ac-
133
5.5. Scalabili y Analysis and Pe o mance Me ics
Figu e 5.19: Mean ime pe p ocess in he compu a ion and communica ion
s eps o he D i Ae case.
e is ics play a c ucial ole in de e mining he e ec i eness o he p econdi ione .
This beha io is consis en wi h he indings p esen ed in Chap e 4, whe e
he ca i y low case exhibi ed simila ends. In ha case, he p econdi ioning
s ep in GLP incu ed an inc eased communica ion cos , mainly due o he
communica ion be ween slices o linele s ac oss di e en domain pa i ions.
This u he suppo s he idea ha he p ima y sou ce o o e head in GLP
a ises om in e p ocess communica ion, which is di ec ly in luenced by he
domain decomposi ion s a egy.
The e o e, as in Chap e 4, he esul s ein o ce he need o op imized
pa i ioning s a egies o be e accommoda e linele s uc u es and mi iga e
communica ion cos s. Fu u e wo k could explo e adap i e pa i ioning ech-
niques ha accoun o linele connec i i y o u he imp o e he e iciency o
GLP in la ge-scale simula ions.
5.5 Scalabili y Analysis and Pe o mance Me ics
To conclude his sec ion, we e alua e he scalabili y and pe o mance o he
p econdi ioning s ep o he h ee s udied cases: he D i Ae model, he
30P30N ai oil, and he S an o d di use . This analysis ollows he me hodology
desc ibed in Chap e 4 (Sec ion 4.5.2), whe e simila me ics we e compu ed
o a ca i y low case.
5.5.1 Speed Up
Figu e 5.20 and able 5.5 p esen he speed-up esul s o he h ee cases. The
S an o d di use exhibi s he closes beha io o ideal scaling, wi h bo h he
134
5.5. Scalabili y Analysis and Pe o mance Me ics
Figu e 5.20: Speed-up o he D i Ae , 30P30N ai oil, and S an o d di use
cases. GLP and LLP compa ison.
GLP and LLP p econdi ione s demons a ing s ong scalabili y. Also, he GLP
p econdi ione has sligh ly wo se scalabili y han he LLP, consis en wi h he
ac ha he GLP has a highe communica ion cos .
In con as , he D i Ae and 30P30N cases show mo e signi ican de i-
a ions om linea speed up, wi h he 30P30N ai oil pe o ming he wo s .
This deg ada ion aligns wi h p e ious obse a ions (Figu e 5.2), whe e he
p econdi ioning s ep in he 30P30N case was ound o be hea ily domina ed by
communica ion ime. Fu he mo e, he 30P30N ai oil exhibi s he la ges gap
be ween he GLP and LLP cases, emphasizing he addi ional compu a ional
o e head in oduced by he GLP p econdi ione .
Fo he D i Ae model, speed up emains easonable bu does no each he
le els obse ed in he di use case. This can be a ibu ed o he shee size o
he p oblem (427 million elemen s) and he impac o load imbalance a highe
p ocess coun s.
5.5.2 Communica ion E iciency
Figu e 5.21 shows he communica ion e iciency o he h ee cases. I can be
seen ha he D i Ae model achie es he bes communica ion e iciency, emain-
ing abo e 0.9 e en a high co e coun s. This indica es ha wi hin he es ed
numbe o pa i ions, he compu a ional wo kload is well dis ibu ed and ha
communica ion o e head does no signi ican ly hinde pe o mance. In con as ,
135
5.5. Scalabili y Analysis and Pe o mance Me ics
P D i Ae 30P30N Di use
GLP LLP GLP LLP GLP LLP
896 1.0 1.0 1.0 1.0
1344 1.0 1.0 1.20 1.20 1.42 1.47
1792 1.27 1.24 1.44 1.33 1.77 1.83
2688 1.77 1.79 1.81 1.93 2.53 2.64
3584 2.32 2.27 2.10 2.39 3.39 3.53
Table 5.5: Compa ison o GLP and LLP Speed Up o di e en es cases and
p ocesso coun s (P ).
Figu e 5.21: Communica ion e iciency o he D i Ae , 30P30N ai oil, and
S an o d di use cases.
he 30P30N case expe iences a no iceable decline in communica ion e iciency,
d opping o 0.68 o he highes p ocess coun s. This end is consis en wi h
p e ious indings showing ha he 30P30N ai oil spends he la ges ac ion o
ime in communica ion, which nega i ely a ec s i s scalabili y. On he o he
hand, he S an o d di use case main ains a ela i ely s able communica ion
e iciency o a ound 0.83 ac oss all p ocess coun s, sugges ing ha i s wo kload
dis ibu ion is mo e balanced. The same esul s can be seen in de ail in able 5.6.
5.5.3 Load Balance
Figu e 5.22 and able 5.7 p esen he load balance esul s o he h ee cases.
The S an o d di use exhibi s he bes load balance, wi h he GLP and LLP
con igu a ions main aining alues abo e 0.55. In con as , he 30P30N case
136
5.5. Scalabili y Analysis and Pe o mance Me ics
P D i Ae 30P30N Di use
896 0.81 0.85
1344 0.95 0.77 0.85
1792 0.95 0.85 0.83
2688 0.95 0.73 0.83
3584 0.97 0.68 0.85
Table 5.6: Communica ion E iciency compa ison o D i Ae , 30P30N, and
Di use cases ac oss di e en p ocesso coun s (P ).
expe iences he wo s load imbalance, wi h alues d opping o 0.32 as he
numbe o p ocesses inc eases. This is likely due o he highly aniso opic
na u e o i s mesh, which makes i mo e challenging o dis ibu e compu a ional
wo kload e enly ac oss p ocesses.
In e es ingly, despi e achie ing he highes communica ion e iciency, he
D i Ae model also su e s om a no iceable load imbalance, wi h alues
luc ua ing be ween 0.40 and 0.51. This sugges s ha ce ain p ocesses a e
unde u ilized due o wo kload a ia ions. The obse ed imbalance is consis en
wi h he beha io shown in Figu e 4.11, whe e linele s concen a ed in speci ic
egions o he domain lead o inc eased compu a ional cos s o he p ocesses
handling hose sec ions. Howe e , he GLP p econdi ione helps mi iga e his
issue by edis ibu ing some o he compu a ional wo kload associa ed wi h
in e ace nodes ac oss all p ocesses, leading o imp o ed balance. This e ec
is e lec ed in Figu e 5.22, whe e he GLP con igu a ion achie es be e load
balance han LLP o bo h he S an o d di use and D i Ae cases (see also
Figu e 4.14b). The 30P30N case, howe e , shows only a ma ginal imp o emen
in load balance wi h GLP compa ed o LLP, indica ing ha he p econdi ione ’s
balancing e ec is less p onounced in his scena io.
P D i Ae 30P30N Di use
GLP LLP GLP LLP GLP LLP
896 0.50 0.55 0.65 0.60
1344 0.51 0.46 0.42 0.44 0.63 0.60
1792 0.48 0.44 0.34 0.37 0.59 0.56
2688 0.45 0.42 0.34 0.35 0.57 0.54
3584 0.44 0.40 0.32 0.33 0.55 0.54
Table 5.7: Load balance compa ison o GLP and LLP ac oss di e en p ocesso
coun s (P ) o D i Ae , 30P30N, and Di use es cases.
5.5.4 Pa allel E iciency
Pa allel e iciency, shown in Figu e 5.23 and in Table 5.8, e lec s he combined
e ec s o speed-up, communica ion e iciency, and load balance, p o iding
a comp ehensi e measu e o scalabili y. The S an o d di use achie es he
highes pa allel e iciency among he h ee cases, wi h bo h GLP and LLP
137
5.5. Scalabili y Analysis and Pe o mance Me ics
Figu e 5.22: Load balance o he D i Ae , 30P30N ai oil, and S an o d di use
cases. GLP and LLP compa ison.
con igu a ions main aining alues abo e 0.47. The D i Ae model ollows,
hough i s e iciency declines as he numbe o p ocesses inc eases. As expec ed,
he 30P30N case exhibi s he lowes pa allel e iciency, d opping below 0.22 a
he highes co e coun s.
These esul s align wi h he ends obse ed in Figu e 5.2, highligh ing he
subs an ial communica ion o e head incu ed by he 30P30N case. In all h ee
cases, he GLP con igu a ion consis en ly demons a es lowe pa allel e iciency
han he LLP. This con i ms p e ious indings ha while he GLP signi ican ly
accele a es con e gence, i s added compu a ional and communica ion o e head
educes o e all e iciency.
P D i Ae 30P30N Di use
GLP LLP GLP LLP GLP LLP
896 0.40 0.55 0.55 0.60
1344 0.48 0.46 0.33 0.44 0.53 0.60
1792 0.46 0.44 0.29 0.37 0.49 0.56
2688 0.43 0.42 0.25 0.35 0.47 0.54
3584 0.42 0.40 0.22 0.33 0.47 0.54
Table 5.8: Pa allel e iciency compa ison o GLP and LLP ac oss di e en
p ocesso coun s (P ) o he D i Ae , 30P30N, and Di use es cases.
138
