UPCommons
Po al del coneixemen obe de la UPC
h p://upcommons.upc.edu/e-p in [s
This is a pos -pee - e iew, p e-copyedi e sion o an a icle published in
Compu a ional mechanics
. The inal au hen ica ed e sion is a ailable
online a : h ps://doi.o g/10.1007/s00466-020-01860-y.
Compu a ional Mechanics manusc ip No.
(will be inse ed by he edi o )
A Scalable F amewo k o he Pa i ioned
Solu ion o Fluid-S uc u e In e ac ion
P oblems
Ali eza Nase i1, Amin To oun e oush2, Ignacio Gonz´alez1,
Mi iam Mehl2, Ca los Da id P´e ez-Sega a1
1Hea and Mass T ans e Technological Cen e (CTTC)
Uni e si a Poli ´ecnica de Ca alunya - Ba celonaTech (UPC)
ESEIAAT, C/ Colom 11, 08222 Te assa (Ba celona), Spain
2Ins i u e o Pa allel and Dis ibu ed Sys ems (IPVS)
Uni e si y o S u ga
Uni e si ¨a ss aße 38, D-70569 S u ga , Ge many
The da e o eceip and accep ance will be inse ed by he edi o
Abs ac In his wo k, we p esen a scalable and e icien pa allel sol e
o he pa i ioned solu ion o luid-s uc u e in e ac ion p oblems h ough
mul i–code coupling. Two ins ances o an in–house pa allel so wa e, Te -
moFluids, a e used o sol e he luid and he s uc u al sub–p oblems, cou-
pled oge he on he in e ace ia he p eCICE coupling lib a y. Fo luid
low, he A bi a y Lag angian-Eule ian o m o he Na ie -S okes equa ions
is sol ed on an uns uc u ed con o ming g id using a second–o de ini e–
olume disc e iza ion. A pa allel dynamic mesh me hod o uns uc u ed
meshes is used o ack he mo ing bounda y. Fo he s uc u al p oblem,
he nonlinea elas odynamics equa ions a e sol ed on an uns uc u ed g id
using a second–o de ini e– olume me hod. A semi–implici FSI coupling
me hod is used which seg ega es he luid p essu e e m and couples i
s ongly o he s uc u e, while he emaining luid e ms and he geome -
ical nonlinea i ies a e only loosely coupled. A obus and ad anced mul i–
ec o quasi–New on me hod is used o he coupling i e a ions be ween he
sol e s. Bo h he luid and he s uc u al sol e use dis ibu ed–memo y pa -
allelism. The in a–sol e communica ion equi ed o da a upda e in he
solu ion p ocess is ca ied ou using non–blocking poin – o–poin communi-
ca o s. The in e –code communica ion is ully pa allel and poin – o–poin ,
a oiding any cen al communica ion uni . Inside each single–physics sol e ,
he load is balanced by di iding he compu a ional domain in o ai ly equal
blocks o each p ocess. Addi ionally, a load balancing model is used a
he in e –code le el o minimize he o e all idle ime o he p ocesses. Two
2 Ali eza Nase i e al.
p ac ical es cases in he con ex o hemodynamics a e s udied, demon-
s a ing he accu acy and compu a ional e iciency o he coupled sol e .
S ong scalabili y es esul s show a pa allel e iciency o 83% on 10,080
CPU co es.
Key wo ds Fluid-S uc u e In e ac ion; Pa i ioned Me hod; Mul i-Code
Coupling; Scalabili y; High Pe o mance Compu ing
1 In oduc ion
Nume ical me hods o sol e luid-s uc u e in e ac ion (FSI) p oblems can
be b oadly di ided in o wo ca ego ies, monoli hic and pa i ioned. In a
monoli hic app oach, he luid and he solid equa ions a e disc e ized and
sol ed as a single la ge sys em, inhe en ly accoun ing o hei mu ual in-
e ac ion (see e.g. [1,2]). In a pa i ioned app oach, on he o he hand, he
FSI p oblem is di ided in o wo domains, o luid and solid. As a esul ,
hese me hods use sepa a e sol e s o luid and s uc u al sub-p oblems and
adop a coupling echnique o accoun o he in e ac ion o he domains.
One o he big ad an ages o he pa i ioned app oach is he possibili y
o use he mos adap ed and well- alida ed nume ical me hods o each
sub-p oblem. Mo eo e , i allows using p e iously de eloped and compu a-
ionally op imized luid and s uc u al sol e codes, hus sa ing excessi e
so wa e de elopmen e o [3,4]. Howe e , he pa i ioned app oach in o-
duces a new challenge o he p oblem which is he coupling be ween he
sepa a e sol e s. This challenge has wo di e en aspec s. The i s aspec
conce ns he me hodology, i.e., coupling dis inc se s o (disc e ized) pa -
ial di e en ial equa ions and ensu ing he physical equilib ium condi ions
on he in e ace. The second aspec is ela ed o he implemen a ion, i.e.,
coupling wo pa allel codes wi h di e en modules and s uc u es, wi h he
aim o achie ing an e icien and scalable o e all so wa e.
Mode n scien i ic and enginee ing p oblems a e o en e y complex and
equi e a huge compu a ional e o . The e o e, any simula ion so wa e
mus be able o e icien ly un on massi ely pa allel compu e s. The pa al-
lel e iciency o a sol e is c ucial in o de o be able o use he a ailable
esou ces adequa ely and pe o m a complex calcula ion. Du ing he ecen
yea s, e icien pa allel codes ha e been de eloped o many single-physics
p oblems, pa icula ly luid and s uc u e sys ems. While he monoli hic
app oach o sol e FSI p oblems equi es de eloping a new sol e and imple-
men ing a so wa e, ollowing a pa i ioned app oach c ea es he oppo u-
ni y o exploi he p e iously-de eloped e icien codes o FSI simula ions.
Ne e heless, using e icien single-physics simula ion codes does no au o-
ma ically gua an ee achie ing a good pa allel e iciency o a coupled mul i-
physics simula ion. Mul i-code coupling in oduces se e al new challenges.
One pa icula di icul y is he da a exchange be ween sepa a e codes which
o en use di e en da a s uc u es and could e en be w i en in di e en
Ti le Supp essed Due o Excessi e Leng h 3
languages. An e icien ly pa allel mechanism o da a exchange be ween he
codes is c ucial o achie ing pa allel e iciency on he coupled amewo k.
Mo eo e , by coupling wo pa allel codes, a new le el o load balancing is
in oduced o he p oblem, as each code would be esponsible o a di e en
amoun o calcula ions on a di e en numbe o CPUs.
Recen e o s ha e been made o de elop e icien and scalable mul i-
physics sol e s (pa icula ly o FSI p oblems) using ei he monoli hic [5–
7] o pa i ioned [8–10] app oaches. Monoli hic FSI sol e s a e na u ally
mo e sui able o massi e pa alleliza ion, since he whole coupled p oblem
is sol ed as a single sys em o equa ions using a single sol e . The e o e,
his app oach does no ace wo main compu a ional challenges o he pa i-
ioned app oach, i.e., pa allel communica ion be ween sepa a e sol e s and
he in e -sol e load balancing. Ne e heless, achie ing a high scalabili y
o monoli hic me hods is also a e y challenging ask, as FSI p oblems a e
highly complex and nonlinea . In his wo k, we ocus on he pa i ioned
app oach in o de o c ea e a scalable and e icien FSI sol e using exis ing
single–physics sol e s. Loss o pa allel e iciency in mul i-code coupling is
conside ed a main d awback o pa i ioned me hods. This is due o he
challenging issues on da a s uc u ing, domain decomposi ion, pa allel da a
communica ion and in e -sol e load balancing. Cajas e al. [9] p esen ed
a pa allel pa i ioned sol e o FSI p oblems, based on mul i-code cou-
pling. The single-physics sol e s we e wo ins ances o an in-house code,
communica ing di ec ly ia MPI messages. The in e -code communica ion
was pa allel and poin - o-poin while each sol e used a mas e -wo ke ap-
p oach in e nally. An in e -code load balancing me hod was p oposed based
on o e loading he a ailable co es in o de o minimize he idle ime. The
coupled amewo k was shown o scale well o 1280 MPI p ocesses on 768
CPU co es [9]. Hewi e al. [10] de eloped a mul i-code coupled ame-
wo k using open sou ce single-physics sol e s (OpenFOAM o luid and
Pa aFEM o s uc u e). The communica ion be ween he sol e s was ca -
ied ou sequen ially using a mas e ank. The sol e was shown o scale
well on 1,536 co es o a coupled FSI p oblem. This appea s o be he high-
es scalabili y epo ed o a pa i ioned FSI sol e in he li e a u e. Apa
om sequen ial communica ion, a majo p oblem o he amewo k in [10] is
ha i mus use he same numbe o co es o he luid and he solid sol e .
This p oblem a ises om he complexi y o he wo-laye ed da a communi-
ca ion s uc u e (inside each sol e and be ween hem), which is essen ial in
a pa i ioned mul i-physics sol e . Using he same numbe o co es o he
sol e s is a se ious d awback, as i e ec i ely ules ou any load balancing
be ween he sol e s, g ea ly educing he compu a ional e iciency o he
coupled amewo k. A compu a ionally e icien pa i ioned FSI sol e ha
could scale on se e al housand p ocesso s is s ill missing in he li e a u e.
Towa ds ha goal, we ocus on he common issues o pa allel pa i ioned
sol e s and p opose e ec i e solu ions o o e come hem.
In ou mul i-code coupled sol e , he communica ion be ween he sepa-
a e codes is managed by using a communica ion lib a y. The communica-
4 Ali eza Nase i e al.
ion lib a y ecei es he da a om each code ia an adap e and con ains
unc ions o acili a e he exchange o da a. I also accele a es he cou-
pling i e a ions be ween he sol e s in he s ongly-coupled con igu a ion.
Adap e s a e used o connec he sol e s and adjus he da a s uc u e while
ans e ing da a om one sol e , h ough he communica ion lib a y, o he
o he sol e . This app oach leads o a obus and powe ul scheme o cou-
ple di e en codes o mul i-physics simula ions. An immedia e ad an age
o his me hod is ha one o bo h o he single-physics sol e s could be
eplaced by o he sol e s wi h ela i ely small changes in he code, in he
bes case limi ed o he adap e . Examples o such communica ion lib a ies
can be ound in [11,8,12,13]. Examples o mul i-physics simula ion so wa e
using communica ion lib a ies can be ound in [8,14,13,15]. The e a e o he
examples in he li e a u e whe e he da a exchange is handled di ec ly and
he communica ion unc ions a e included in he single-physics codes hem-
sel es. This app oach is shown o be e icien when wo ins ances o he
same code a e coupled (e.g. [9]). Howe e , i is no as obus and powe ul
as he i s app oach, especially in cases whe e wo di e en codes a e being
coupled. Mo eo e , i does no allow o exchange one o bo h sol e s. E en
hough in he cu en wo k we use wo ins ances o he same sol e o he
single–physics sub-p oblems, we ollow he i s app oach (using a coupling
lib a y) o c ea e a obus and lexible amewo k.
A he coupling me hodology le el, pa i ioned me hods a e gene ally
di ided in o explici (o loosely-coupled) and implici (o s ongly-coupled)
schemes. Explici me hods sol e he luid and s uc u e equa ions only once
pe ime s ep, using da a om he p e ious solu ion o he pa ne sol e .
The e o e, explici me hods do no sa is y he exac equilib ium condi ions
a he in e ace, which causes ins abili y issues in many FSI p oblems. The
so-called added-mass ins abili y is pa icula ly s ong in FSI p oblems wi h
incomp essible low, a slende in e ace, and simila densi ies o luid and
solid [16,17]. Implici me hods, on he o he hand, en o ce he equilib ium
condi ion a he in e ace h ough coupling i e a ions be ween luid and
s uc u al sol e s. These me hods a e s able o p oblems wi h s ong added-
mass e ec . Howe e , hei compu a ional cos is gene ally high due o he
epe i i e solu ion o he go e ning equa ions a each ime s ep [3,4]. In
he ecen ly in oduced semi-implici coupling app oach [18–20], he luid
p essu e e m is seg ega ed and s ongly coupled o he s uc u e, while
he emaining luid e ms a e only loosely coupled. S ong coupling o he
luid p essu e and s uc u al de o ma ion elimina es he added-mass ins a-
bili y issue, while loose coupling o he emaining luid e ms helps a oiding
excessi e compu a ional cos [18,19].
In his wo k, wo ins ances o a pa allel in-house code, Te moFluids [21],
a e used o sol e he luid and he s uc u e p oblem. Te moFluids is a o-
bus gene al-pu pose so wa e o luid and s uc u e p oblems, using s a e-
o - he-a me hods o u bulen low [22,23], mul iphase low [24,25], and
complex he mal sys ems [26,27], wi h a high compu a ional e iciency and
pa allel scalabili y [28–30]. I p esen s a conse a i e disc e iza ion o he

Ti le Supp essed Due o Excessi e Leng h 5
go e ning equa ions on uns uc u ed g ids based on a ini e- olume me hod.
I is also equipped wi h dynamic-mesh schemes o ack he mo ing bound-
a y. The coupling o he codes is ca ied ou using he p eCICE coupling
lib a y [12]. The p eCICE lib a y p o ides communica ion, da a mapping
and equa ion coupling o su ace coupled mul i-physics applica ions in a
modula manne . I o e s a ully pa allel poin - o-poin communica ion,
ad anced quasi-New on i e a i e coupling schemes and a ious ad anced
mapping me hods (bo h consis en and conse a i e). A semi-implici FSI
coupling app oach p oposed in [19] is applied, which e ec i ely seg ega es
he luid p essu e e m and couples i s ongly o he s uc u e. The e-
maining luid e ms and he geome ical non-linea i ies a e ea ed explic-
i ly, educing he compu a ional cos o he nume ical solu ion. P ac ical
es cases in he con ex o biological low ( low inside de o mable essels)
a e s udied and he scalabili y o he o e all amewo k is e alua ed.
The emainde o his a icle is o ganized as ollows. In Sec ion 2, he
go e ning equa ions o each sub-p oblem and he coupling condi ions a e
p esen ed. Sec ion 3 desc ibes he p oposed nume ical me hods. Sec ion 4
p esen s he pa alleliza ion me hod o each single-physics sol e , as well
as he in e -code communica ions and load balancing. Nume ical es s a e
p esen ed in Sec ion 5, while Sec ion 6 summa izes and concludes he a icle.
2 Go e ning Equa ions
In his sec ion, we p esen he luid and s uc u e go e ning equa ions and
he coupling condi ions on hei common in e ace. The luid and he s uc-
u al domain a e e e ed o as Ω ( )⊂R3×(0, T) and Ωs( )⊂R3×(0, T),
espec i ely, whe e ∈(0, T ) deno es ime. The luid-s uc u e in e ace is
he common bounda y o he domains, deno ed by Γ( ) = ∂Ω ( )∩∂Ωs( ).
An A bi a y Lag angian-Eule ian (ALE) o mula ion oge he wi h a con-
o ming mesh echnique is used o sol e he luid low in a mo ing domain.
A Lag angian o mula ion is used o he s uc u al equa ions.
The uns eady low o an incomp essible iscous luid is go e ned by
he Na ie -S okes equa ions. An ALE o mula ion o hese equa ions in a
mo ing domain is gi en by
∂u
∂ +c· ∇u=1
ρ
∇ · σ (1)
∇ · u= 0 (2)
whe e uis he luid eloci y and ρ he luid densi y. Vec o cis he ALE
con ec i e eloci y c=u−w, which is he luid eloci y ela i e o a
domain mo ing wi h a eloci y w. The s ess enso σ o an incomp essible
New onian luid is de ined as
σ =−pI+µ (∇u+∇uT) (3)
6 Ali eza Nase i e al.
whe e pis he luid p essu e, I he uni enso and µ is he dynamic is-
cosi y o he luid.
The s uc u al domain is go e ned by he conse a ion laws o mass and
momen um, whose Lag angian o m is gi en by
ρ0
s=ρsJ(4)
∂
∂ ρs
∂d
∂ =∇ · σs(5)
whe e supe sc ip 0 e e s o he e e ence ma e ial (unde o med) con igu-
a ion o he body, ρsis he s uc u al densi y and dis he displacemen
om he e e ence con igu a ion. The enso σsis he Cauchy s ess enso ,
which can be ela ed o he displacemen ield by he hype elas ic cons i u-
i e model o Sain Venan -Ki chho
σs=B
2J[2µs(B−I) + λs (B−I)] (6)
whe e Bis he le Cauchy-G een de o ma ion enso B=F·FT, and µs
and λsa e he Lam´e’s pa ame e s. The ma e ial de o ma ion enso Fis
e alua ed as F=I+∇dand i s de e minan is deno ed by J= de (F).
The physical equilib ium on he luid-solid common bounda y (kine-
ma ic and dynamic equilib ium) cons i u es he coupling condi ions on he
in e ace. Fo a non-slip ype in e ace hey ead
uΓ=∂dΓ
∂ (7)
σsnΓ=σ nΓ(8)
a Γ, whe e nΓis he uni no mal ec o on he in e ace.
3 Nume ical Me hods
In his sec ion we p esen he nume ical me hods o disc e iza ion and
solu ion o he single-physics p oblems, as well as he coupling me hod.
Th oughou he de elopmen o ou coupled amewo k and he single-
physics sol e s, an emphasis was made on making he so wa e simple,
modula and, as a as possible, ma ix- ee. Mo eo e , he luid sol e was
mainly de eloped o u bulen low simula ions. The e o e, many aspec s o
he disc e iza ion and he nume ical me hods co espond o he pa icula
conside a ions o u bulen lows.
Ti le Supp essed Due o Excessi e Leng h 7
3.1 Fluid Sol e
Fo luid low, a ac ional-s ep p ojec ion me hod along wi h an explici
ime ad ancemen is used o sol e he eloci y-p essu e coupling o he mo-
men um equa ion. This leads o a h ee s ep solu ion o he luid go e ning
equa ions om ime s ep n o n+ 1, wi h a ime inc emen o ∆
up=un−∆ [3
2(cn· ∇un−µ
ρ
∆un)−1
2(cn−1· ∇un−1−µ
ρ
∆un−1)] (9)
∆
ρ
∆pn+1 =∇ · up(10)
un+1 =up−∆
ρ
∇pn+1 (11)
in Ωn+1
, whe e upis a p edic ed eloci y ield which does no sa is y he in-
comp essibili y condi ion (Eq. (2)). This in e media e eloci y ield is hen
p ojec ed on o a di e gence- ee ield h ough he co ec ion a Eq. (11).
An explici Adams-Bash o h me hod is used o he con ec i e and di u-
si e e ms in Eq. (9). Using an explici me hod is pa icula ly p e e ed in
u bulen low simula ions whe e small ime s eps a e indispensable. F om
a compu a ional poin o iew, i a oids sol ing a nonlinea sys em o he
luid eloci y ield. I also o e s ad an ages in pa alleliza ion as only he
in o ma ion om he p e ious ime s ep is equi ed (only one episode o
da a upda e be ween nodes is equi ed a each ime s ep).
A ini e- olume me hod is used o he spa ial disc e iza ion o he luid
equa ions on a colloca ed, uns uc u ed mesh wi h second-o de symme y-
p ese ing schemes. Symme y-p ese ing schemes conse e he kine ic en-
e gy o he low a he disc e e le el which is c ucially impo an in u bulen
low simula ions [31,32]. A Jacobi–p econdi ioned conjuga e g adien sol e
is used o sol e he Poisson equa ion o p essu e. Mo e de ails o he nu-
me ical me hods o luid low equa ions can be ound in [33,32].
We use a con o ming mesh echnique o ack he mo ing bounda y,
hus he luid mesh needs o mo e in o de o adap o he new loca ion
o he in e ace. The ansla ed mesh and he e alua ed domain eloci y
wmus sa is y he Geome ic Conse a ion Law (GCL) [34,35]. The GCL
gua an ees ha no olume is los while mo ing he g id, and a cons an
ield is p ese ed by he ALE scheme. Fo any con ol olume (CV) in he
luid domain, he GCL is s a ed as
∂
∂ −Zs
w·dA= 0 (12)
whe e and ss and o he olume and he bounda y su ace o he CV,
espec i ely, and Ais he a ea ec o poin ing ou wa d.
A pa allel mo ing mesh echnique based on he adial basis unc ion
in e pola ion me hod [36] is used o mo e he luid g id in acco dance wi h
8 Ali eza Nase i e al.
he new loca ion o he in e ace and de ine he disc e ized luid domain a
he new ime s ep Ωn+1
. The me hod uses he known displacemen on he
in e ace o e alua e an in e pola ed alue o he in e io e ices o he
luid g id. A g ea ad an age o his me hod is ha i does no need he
connec i i y o he mesh elemen s and can be applied o bo h s uc u ed
and uns uc u ed g ids. Mo eo e , i only equi es sol ing a linea sys em o
equa ions whose size is limi ed o he numbe o e ices on he luid-solid
in e ace. A de ailed desc ip ion o he mo ing mesh me hod can be ound
in [36,19].
A e he luid mesh is mo ed, he domain eloci y is e alua ed a he
su aces o each con ol olume. We e alua e he su ace eloci ies based on
he GCL law in o de o exac ly sa is y i . The ime a e o change o olume
o a CV is equal o he sum o olumes swep by i s aces. In his wo k, we
e alua e he domain eloci y w ace a each ace based on he olume swep
by ha ace. Wi h a second-o de backwa d disc e iza ion, i eads
wn+1
ace =3
2(δ
A∆ n)n+1 −1
2(δ
A∆ n)n(13)
whe e Ais he su ace a ea, n he uni no mal ec o o he ace, ∆ he
ime s ep, and δ is he olume swep by he ace a one ime s ep. A mo e
de ailed desc ip ion o he e alua ion o he domain eloci y ield and he
sa is ac ion o he geome ic conse a ion law can be ound in [20].
3.2 Solid Sol e
Aiming a low-induced de o ma ions and oscilla ions o he s uc u e, he
solid equa ions a e sol ed using an implici ime in eg a ion, whe e bo h
he ine ial and su ace o ces in Eq. (5) a e e alua ed a he cu en ime
ins an n+1. A cell-cen e ed ini e- olume me hod wi h a o al Lag angian
app oach is used o he spa ial disc e iza ion. The momen um equa ion
is in eg a ed on he unde o med con igu a ion. A e applying he Gauss
heo em on he s ess di e gence and ela ing he unde o med and cu en
a ea ec o s wi h Nanson’s o mula, he momen um balance can be w i en
as
Z 0
ρ0
s
∂2d
∂ 2d 0=Zs0
σs(JF−Tn0)ds0(14)
The accele a ion o he ine ial e m is compu ed acco ding o he ape-
zoidal ule
∂2d
∂ 2=4
∆ 2dn+1 −4
∆ 2dn−4
∆ ∂d
∂ n
−∂2d
∂ 2n
(15)
The dependencies be ween di e en di ec ions o he displacemen and
he geome ical and ma e ial non-linea i ies ound in he igh -hand side
o Eq. (14) a e de e ed o he sou ce e m o he sys em. The e o e, ou e
Ti le Supp essed Due o Excessi e Leng h 15
b a y [12]. In e –code communica ion ia p eCICE can be ei he based on
MPI po s (MPI-2.0) o on lowe le el TCP/IP socke s. In he cu en wo k
we ha e used TCP/IP socke s, because he MPI po s unc ionali y is miss-
ing in he implemen ed MPI e sions on he supe compu e ha we used
o ou es s (Supe MUC supe compu e a he Leibniz Supe compu ing
Cen e o he Ba a ian Academy o Sciences). To es ablish communica-
ion channels be ween he pa icipan s’ anks, he luid and he solid mesh
pa i ions a e ini ially analyzed o ind logical connec ions be ween hem.
Once he communica ion channels a e es ablished, da a a e exchanged in an
asynch onous way o a oid unnecessa y blocking. Es ablishing he commu-
nica ion channels is ca ied ou only once a he ini ializa ion s age. Fo he
es o he un- ime, he same channels a e used o da a exchange. Since
using a cen al communica ion ins ance can deg ade he scalabili y o ou
amewo k, we use a ully pa allel poin - o-poin communica ion scheme,
hus no cen al se e –like uni is used. This way, da a exchange happens
locally be ween he connec ed anks. I is ob ious ha , in case o ha ing
a big mesh o a high numbe o anks and using a cen al communica ion
ins ance, a single mas e ank o se e would be a se e e bo leneck.
Figu e 2 schema ically shows he pa allel s uc u e o he coupled ame-
wo k and he di e en le els o communica ion.
4.3 Load Balancing
Load imbalance can be a se ious sou ce o ine iciency in pa allel simula ion
codes. Inside each single–physics sol e , he load is balanced by di iding he
compu a ional domain in o ai ly equal blocks o each p ocess, as explained
in Sec ion 4.1. Howe e , in a pa i ioned FSI simula ion, he e exis s a new
le el o load balancing be ween he single–physics sol e s. In his wo k wi h
a simul aneous execu ion o he sol e s (Jacobi- ype p oblem desc ibed in
Sec ion 3.3), bo h sol e s mus inish an i e a ion and send he ou pu o he
pa ne sol e be o e he nex i e a ion can s a . This means, in case he
a ailable CPUs a e no dis ibu ed op imally among he sol e s, one sol e
will be wai ing o he pa ne o inish i s own compu a ions. In con as o
he load balancing wi hin a single–physics sol e , load balancing ac oss he
codes is mo e di icul since we do no know a p io i he ela ion be ween
he wo k pe cell o di e en sol e s.
In his wo k, we ollow he app oach p oposed in [55] o add ess his
issue. We i s model he sol e pe o mance agains he numbe o anks
o each domain, and hen sol e an in ege op imiza ion p oblem o ind
he app op ia e pai o ank numbe s o he domains ha minimizes he
wai ing ime. Since analy ical modeling o he sol e s’ pe o mance is e y
complex, we use an empi ical eg ession-based app oach ins ead, aiming o
ind an app op ia e pe o mance model.
We un msimula ions wi h di e en numbe o co es, yielding a se o
da a poin s consis ing o pai s (n, n) mapping he numbe o anks n o he

16 Ali eza Nase i e al.
Rank n-1
Rank n-2
Rank 2
Rank 1
Rank 0
Rank m-1
Rank 1
Rank 0
adap e adap e adap e adap e adap e
adap e adap e adap e
p eCICE Communica o
Poin - o-Poin Communica ion
In e -Code
In a-Sol e Poin - o-Poin Communica ion
In a-Sol e Poin - o-Poin Communica ion
Fluid Sol e Solid Sol e
MPI Comm Wo ld 1
MPI Comm Wo ld 2
SIMD wi hin
each ank
.
Fig. 2 Pa alleliza ion model o he coupled amewo k and he di e en le els
o communica ion.
un- ime n(in his wo k, m=5 simula ions, each o only 3 ime s eps). In
o de o ind a unc ion (n) o each sol e , which p edic s he un- ime o
any n, we use he Pe o mance Model No mal Fo m (PMNF) [56], de ined
in (Eq. (26)), as a basis o ou p edic ion model:
(n) =
q
X
k=1
ckniklogjk
2(n),(26)
whe e nis he numbe o anks used by he sol e , qis he numbe o e ms
used o he empi ical un- ime app oxima ion, ckis he weigh o each e m
in he app oxima ion unc ion, and ikand jka e empi ical coe icien s.
Ti le Supp essed Due o Excessi e Leng h 17
Calo oiu e al. [56] sugges , ha he sea ch space gi en by q= 2, ik∈
{±0
4,±1
4,...,±12
4}and jk∈ {0,±1,±2}is sui able o many applica ions.
To ind he op imal model, we simply check all he combina ions wi hin he
sea ch space, calcula e a c oss- alida ion-based loss o each combina ion
( he accumula i e e o on he alida ion da a se ) and pick he combina ion
wi h he smalles loss. By applying a PMNF eg ession, we a e able o
gene a e pe o mance models (n) o each sol e in ol ed in he simula ion.
As men ioned be o e, ou goal is o ind an op imal assignmen o co es
o each sol e o a limi ed o al numbe o a ailable co es P, such ha
he o e all un- ime F(n , ns) is minimized (n and nsa e numbe o anks
used by he luid and he solid sol e espec i ely). This can be exp essed
by he ollowing op imiza ion p oblem:
minimize
n ,ns
F(n , ns) wi h F(n , ns) = max( s(ns), (n ))
subjec o n +ns≤P.
(27)
I he unc ions sand a e app oxima ed by he PMNF eg ession,
his op imiza ion p oblem is a nonlinea , possibly non-con ex in ege op i-
miza ion. We assume ha sand a e bo h mono onically dec easing, i.e.,
assigning mo e co es o a sol e ne e inc eases he un- ime. Wi h his, we
can simpli y he cons ain o
P=n +ns.(28)
The op imiza ion p oblem can hen be sol ed by checking all possible
alues o nsand n in o de o choose he pai ha minimizes he o al un-
ime. This pai o ank coun s is hen used o di ide he a ailable p ocesso s
be ween he luid and he s uc u al sol e o he FSI simula ion. Fo mo e
de ails, please e e o [55].
5 Nume ical Tes s
Nume ical es s a e p o ided in his sec ion o demons a e he pa allel scal-
abili y o he coupled amewo k in sol ing p ac ically ele an FSI p ob-
lems. The es cases a e in he con ex o hemodynamics. The i s es case
is a benchma k p oblem ep esen ing he p opaga ion o p essu e wa es in-
side a 3D de o mable ube. The second es case is he blood low inside a
pa ien -speci ic ao a conside ing he elas ic de o ma ion o he ao a wall.
The scalabili y es s a e ca ied ou on he Supe MUC supe compu e a
he Leibniz Supe compu ing Cen e o he Ba a ian Academy o Sciences
and Humani ies in Ga ching [57]. Supe MUC consis s o 2.6GHz In el Xeon
E5-2697- 3 (Haswell) p ocesso s. Each compu ing node con ains wo p o-
cesso s wi h 14 co es pe p ocesso (28 co es pe node) and 64GB o RAM.
The nodes a e connec ed ia Mellanox In iniband FDR14 in e connec . The
GNU GCC compile was used o compile bo h he Te moFluids sol e and
18 Ali eza Nase i e al.
he p eCICE lib a y. In addi ion, an In el MPI implemen a ion compa ible
wi h he GCC compile was used o in a-sol e pa alleliza ion.
5.1 Tes case 1: 3D Flow Inside a De o mable Tube
This benchma k p oblem was p oposed by [58] and s udied, among o he s,
by [59,44,60,61]. The p oblem is a 3D incomp essible low inside a s aigh
ube wi h a de o mable wall, mo i a ed by he ype o p oblems encoun e ed
in hemodynamics. The ube has a leng h o l= 0.05m, an inne adius
o R0= 0.005m, and a wall hickness o h= 0.001m. The luid densi y
and iscosi y a e ρ = 1000kg/m3and µ = 0.003Pa ·s, espec i ely. The
s uc u al densi y is ρs= 1200kg/m3, he Young modulus E= 3×105N/m2,
and he Poisson a io ν= 0.3.
The ube is clamped a bo h ends and he luid is ini ially a es . An
o e p essu e o 1333.2Pa is applied a he ube inle du ing a pe iod o 0.003s
and a cons an p essu e o 0Pa a e wa ds. The p essu e a he ou le is 0Pa
du ing he whole simula ion. A Neumann bounda y condi ion is used o he
luid eloci y a bo h he inle and he ou le bounda ies. The ou e su ace
o he ube wall is ea ed as a ac ion- ee bounda y.
An uns uc u ed e ahed al mesh is used o he luid domain. The
solid domain mesh is cons uc ed by ex uding he luid mesh on he ou e
bounda y (s uc u ed mesh). Th ee di e en mesh esolu ions a e used o
he nume ical es s. The esolu ion o he mesh o he luid and he solid
domain is p o ided in Table 1. Two meshes (M1 and M2 in Table 1) a e used
o sol e he p oblem om = 0 un il = 0.02s wi h cons an ime s ep sizes
o ∆ = 10−4and ∆ = 5 ×10−5s, espec i ely (which means doubling he
esolu ion in each spa ial di ec ion and ime). Figu e 3 p esen s he adial
displacemen a he mid-leng h o he ube du ing he simula ion ime. As
seen in he igu e, he esul s ob ained by mesh M1 and M2 (and hei
co esponding ime s ep) a e ai ly close. Small di e ences a e isible nea
he peaks and he oughs. Resul s wi h he coa se mesh and he la ge ime
s ep appea o be mo e damped, which is expec ed due o a la ge nume ical
dissipa ion. Figu e 3 also con ains nume ical esul s om Eken e al. [60]
and Lozo skiy e al. [61]. Al hough he ou se s o esul s ollow he same
end, he esul s om [60,61] appea o be mo e damped, compa ed o
he p esen solu ion. This could be due o he ex a nume ical dissipa ion
in [60,61], as bo h wo ks use a i s -o de Eule disc e iza ion in ime wi h
a ime s ep ∆ = 10−4s (compa ed o a second-o de ime disc e iza ion in
he cu en wo k). Figu e 4 shows he eloci y ec o s inside he de o med
ube a wo ins an s = 0.005 and = 0.01s. The colo con ou in he
solid domain is he on Mises equi alen s ess. The de o ma ion o he wall
is magni ied by a ac o o en o be be e isible. The igu e shows he
p opaga ion o he p essu e wa e wi h a ini e eloci y inside he ube.
Fo scalabili y es s, we ha e used a much ine g id (mesh M3 in Ta-
ble 1). A comple e simula ion is no ca ied ou o hese es s, only he i s
Ti le Supp essed Due o Excessi e Leng h 19
Table 1 Compu a ional g ids used o he de o mable ube es case ( es case 1).
Mesh name No. o cells
Fluid S uc u e
M1 15K 10K
M2 120K 80K
M3 9M 6M
-0.1
-0.05
0
0.05
0.1
0.15
0 0.005 0.01 0.015 0.02
mid-leng h adial displacemen (mm)
ime (s)
mesh M2, ∆ =5e-5
mesh M1, ∆ =1e-4
Eken e al. (2016)
Lozo skiy e al. (2019)
Fig. 3 Radial displacemen a he mid-leng h o he ube ( es case 1) o wo
di e en mesh esolu ions and ime s ep sizes, compa ed o nume ical esul s
om [60,61].
en ime s eps a e sol ed. The solu ion esul s a he en i e ield ( o bo h
luid and solid domains) a e w i en in o h5 ou pu iles a e e y ime s ep.
Figu e 5 shows he a e age un– ime pe ime s ep o di e en numbe s
o CPU co es using mesh M3. This un– ime includes he ime o w i ing
ou pu da a iles, bu does no include he ini ializa ion ime o he sol e s.
The ideal (linea ) educ ion o he un- ime by inc easing he co e coun is
also shown o compa ison. The numbe o co es in he ho izon al axis o
he g aph indica es he o al sum o co es used o he luid and he solid
sol e . The dis ibu ion o he a ailable co es be ween he sol e s is based
on he load balancing model desc ibed in Sec ion 4.3, and i is p o ided in
Table 2. As seen in he igu e, a e y good educ ion in he compu a ional
ime is achie ed by inc easing he numbe o co es. Resul s show a e y good
scalabili y o up o 1400 co es, wi h an almos ideal speed-up o up o 1120
co es. Figu e 6 shows he pa allel e iciency a each poin o he g aph. The
pa allel e iciency o he s ong scalabili y on mco es is e alua ed as
20 Ali eza Nase i e al.
Fig. 4 P opaga ion o he p essu e wa e inside he de o mable ube ( es case
1). Shown in he igu e a e luid eloci y ec o s inside he de o med domain and
he on Mises equi alen s ess a he wall. Top: = 0.005s; Bo om: = 0.01s.
De o ma ions a e magni ied by a ac o o en.
e iciency = l×l
m×m(29)
whe e mis he un- ime on mco es, and lis he un- ime on he smalles
numbe o co es, indica ed by l. I possible, he pa allel e iciency is measu ed
agains he sequen ial un- ime (l= 1). Howe e , due o he limi a ion
o memo y on a single co e, la ge p oblems canno be sol ed sequen ially.
The e o e, he smalles numbe o co es which can be used o sol e he
p oblem is used as he basis o e alua e he e iciency (l= 280 o his es
case). As seen in he igu e, he pa allel e iciency is 96% o 1120 co es which
indica es an almos ideal pa allel scalabili y. Fo 1400 co es he e iciency is
s ill high, a 82%. Fo highe co e numbe s he pa allel e iciency deg ades.
This limi appea s o be mos ly de e mined by he size o he mesh used o
he es s. The size o he compu a ional g id ( o al o 15M cells o mesh
M3) is ela i ely small o be di ided among o e a housand p ocesses.
Di iding mesh M3 among 1400 co es means each p ocess owns oughly 10K

Ti le Supp essed Due o Excessi e Leng h 21
cells, which is oo ew calcula ions and co esponds o a e y low a i hme ic
in ensi y (compa ed o communica ion). By epea ing he scalabili y es s
using ine g ids, a g ea e scalabili y can be achie ed on highe numbe s o
co es. La ge compu a ional g ids a e used o he nex es case. Mo eo e ,
he ou pu da a w i ing could limi he scalabili y as i essen ially con ains
sequen ial s eps. We ha e included he ou pu esul w i ing ime in he
scalabili y es s o he cu en es case, because i is a necessa y pa o
any p ac ical simula ion.
Table 2 Dis ibu ion o he CPU co es be ween he luid and he solid sol e
based on he in e -code load balancing model, o es case 1 (de o mable ube)
and mesh M3.
To al No. o co es 280 420 560 700 840 980 1120 1260 1400
Fluid sol e 153 230 301 363 417 461 496 524 545
Solid sol e 127 190 259 337 423 519 624 736 855
280
420
560
700
840
980
1120
1260
1400
Numbe o co es
102
4 × 101
6 × 101
2 × 102
Run ime pe ime s ep [s]
Tube mesh M3
Ideal scalabili y
Fig. 5 S ong scalabili y es esul s o es case 1 (de o mable ube) and mesh
M3: a e age un– ime pe ime s ep o di e en numbe s o co es.
A e e y ime s ep, an a e age o oughly 17 coupling i e a ions we e e-
qui ed o achie e con e gence on he coupled p oblem (using F SI = 10−5).
This numbe emained ai ly cons an by inc easing he numbe o p o-
cesses, as seen in Table 3. This shows ha he o e all me hodology o he
coupled FSI p oblem is ma hema ically scalable. Table 3 also con ains he
da a o he ini ializa ion ime o he coupled amewo k. The epo ed ime
22 Ali eza Nase i e al.
280
420
560
700
840
980
1120
1260
1400
Numbe o co es
85
90
95
100
105
E iciency [%]
Ideal E iciency
E iciency
Fig. 6 S ong scalabili y es esul s o es case 1 (de o mable ube) and mesh
M3: pa allel e iciency a di e en co e coun s.
includes he ini ializa ion imes o he single–physics sol e s, as well as he
ime o es ablishing he in e –code communica ion channels and ini ializing
he coupling lib a y modules. I does no include he p e-p ocessing s eps
such as mesh gene a ion o pa i ioning using he METIS lib a y. The da a
in Table 3 show ha he ini ializa ion ime emains ai ly cons an and only
sligh ly inc eases when inc easing he numbe o p ocesses.
Table 3 S ong scalabili y o es case 1 (de o mable ube) and mesh M3. Resul s
in he able a e he a e age numbe o coupling i e a ions pe ime s ep, and he
o al ini ializa ion ime o he coupled amewo k.
No. o co es 280 420 560 700 840 980 1120 1260 1400
coupling i e a ions (a e age) 17.4 17.6 16.8 17.4 16.8 16.8 17.2 17.4 17.2
ini ializa ion ime [s] 329.6 335.8 339.2 345.6 351.7 353.9 359.2 365.4 370.6
5.2 Tes case 2: Pa ien –Speci ic Ao a
The second es case is a simula ion o blood low inside a pa ien -speci ic
ao a wi h a mild ho acic ao ic coa c a ion. The 3D geome y o he ao a
is ob ained by con as agen magne ic esonance angiog aphy, p o ided by
he 2nd CFD challenge o he STACOM 2013 con e ence [62]. Figu e 7 shows
he p o ided geome y and he loca ion o he bounda ies. The hickness
Ti le Supp essed Due o Excessi e Leng h 23
o he ao a wall and i s mechanical p ope ies we e no p o ided in he
challenge da a. The e o e, we assume alues in he ypical physiological
ange. A uni o m hickness o h= 2mm is assumed o he wall along wi h
densi y ρs= 1200kg/m3, Young modulus E= 3 ×105N/m2, and Poisson
a io ν= 0.3. The densi y and he iscosi y o he blood a e conside ed
o be ρ = 1000kg/m3and µ = 0.004Pa ·s. These alues o he wall
hickness and he p ope ies o he wall and blood a e simila o he alues
used in [63].
Fig. 7 Tes case 2, 3D geome y o a pa ien -speci ic ao a p o ided in [62], and
he loca ion o he bounda ies.
Fo he inle bounda y o he ao a, a Di ichle bounda y condi ion is
used o he eloci y, using measu ed physiological low a e da a p o ided
in [62] o he es condi ion. A Neumann bounda y condi ion is used o
he luid p essu e a he inle . Fo he ou le bounda ies, explici RCR
Windkessel bounda y condi ions [64] a e used o model he e ec o he es
o he ascula ne wo k. The Windkessel pa ame e s a e chosen as hose
epo ed in [65]. Fo he solid, a ze o–displacemen (clamped) bounda y
condi ion is se a he inle and a he ou le s, while a ac ion- ee bounda y
condi ion is used on he ou e su ace o he wall.
The objec i e o he cu en wo k is no a deep s udy o he biophysical
phenomena ea u ed in his es case, bu a he o show he capabili y o
he de eloped amewo k o sol e such a complex p oblem on a massi ely
pa allel con igu a ion. Ne e heless, he ob ained luid eloci y and p es-
24 Ali eza Nase i e al.
su e ields, as well as he s uc u al displacemen s we e seen o be in he
easonable physiological ange. Figu e 8 shows he solu ion a wo ins an s,
= 0.05s and = 0.1s. The igu e con ains he eloci y ec o plo in-
side he de o med ao ic wall. The colo con ou s in he s uc u al domain
co espond o he on Mises equi alen s ess.
Fig. 8 Tes case 2, luid eloci y ec o s inside he de o med ao ic wall and he
on Mises equi alen s ess a he wall. Le : a = 0.05s; Righ : a = 0.1s.
The s ong scalabili y o he de eloped amewo k is e alua ed using he
compu a ional g ids M1 and M2 (Table 4). Bo h luid and solid g ids a e
uns uc u ed e ahed al meshes. The i s en ime s eps a e sol ed o he
scalabili y es s. Unlike he p e ious es case, he ou pu esul w i ing s ep
is disabled, hus he un- imes only include he solu ion ime o he coupled
p oblem. Figu e 9 shows he a e age un– ime pe ime s ep o di e en
numbe s o co es o mesh M1, wi h and wi hou in e -code load balancing.
Simila o he p e ious es case, he numbe o co es indica es he o al
numbe o p ocesses o he luid and he solid sol e oge he . Fo he case
wi hou load balancing, he a ailable co es a e di ided be ween he sol e s
p opo ional o hei espec i e mesh size. Fo he load balancing case, he
di ision o he co es is based on he load balancing model in Sec ion 4.3,
and i is p o ided in Table 5. As seen in Figu e 9, a e y good educ ion
in compu a ional ime is achie ed by inc easing he numbe o co es, up
Ti le Supp essed Due o Excessi e Leng h 31
load balancing is conside ed, i.e., inside each sol e and ac oss he codes.
The a ailable p ocesso s a e di ided be ween he wo sol e s based on a
load balancing model ha minimizes he o al idle ime o he p ocesses.
Bo h he luid and he s uc u al sol e disc e ize and sol e he co e-
sponding go e ning equa ions on uns uc u ed 3D meshes. A semi–implici
FSI coupling me hod is used, in which he luid p essu e e m is seg ega ed
and s ongly coupled o he s uc u e, while he emaining e ms a e only
loosely coupled. An e icien mul i– ec o quasi–New on me hod is used o
sol e he coupled in e ace p oblem.
Two nume ical es cases in he con ex o hemodynamics a e conside ed
and he s ong scalabili y o he coupled amewo k is e alua ed. The i s
es case is a benchma k FSI p oblem, sol ing an incomp essible low inside
a de o mable ube. The simula ion esul s o his es case a e compa ed
o o he nume ical esul s om he li e a u e in o de o e i y he accu-
acy o he solu ion me hods and he o e all amewo k. Scalabili y es s
a e ca ied ou using a compu a ional mesh consis ing o 9 million uns uc-
u ed e ahed al cells o he luid and 6 million s uc u ed hexahed al cells
o he solid. Tes esul s showed a pa allel e iciency o 82% on 1400 CPU
co es. This limi appea s o be mos ly de e mined by he size o mesh, as o
highe numbe o co es he a i hme ic in ensi y o he p oblem becomes oo
small. The numbe o coupling i e a ions a each ime s ep emained nea ly
cons an o di e en numbe o co es, which shows he de eloped me hods
a e ma hema ically scalable. Mo eo e , he ini ializa ion ime o he ame-
wo k emained ai ly cons an and inc eased only sligh ly o di e en co e
coun s.
The second es case sol es he blood low inside a pa ien –speci ic ao a.
Two di e en meshes we e used o he scalabili y es s, consis ing o un-
s uc u ed e ahed al cells o bo h he luid and he solid domain. Tes
esul s using a mesh wi h 20 million cells o he luid and 9 million cells o
he solid, showed a pa allel e iciency o 79% on 3920 CPU co es. Simila o
he p e ious es case, his limi appea s o be de e mined by he mesh size
and small a i hme ic in ensi y o highe co e numbe s. Mo eo e , we ha e
e alua ed he e ec o ou in e -code load balancing model on he pe o -
mance o he coupled sol e . Applying he load balancing me hod educed
he un- ime by an a e age o 15%, almos uni o mly ac oss he di e en
o al co e coun s. The scalabili y es s we e epea ed using a mesh wi h 95
million cells o he luid and 41 million cells o he solid. Resul s demon-
s a ed a pa allel e iciency o 83% on 10,080 CPU co es. The numbe o
FSI coupling i e a ions a each ime s ep emained cons an o di e en
co e coun s, demons a ing he ma hema ical scalabili y o he me hods.
The ini ializa ion ime o he amewo k inc eased only mode a ely o up
o 6720 co es, howe e , i g ew apidly o highe numbe o co es. Fu he -
mo e, i was shown ha he ini ializa ion ime o he amewo k inc eases
linea ly by inc easing he size o he mesh.

32 Ali eza Nase i e al.
7 Acknowledgemen
This wo k was inancially suppo ed by
–Minis e io de Econom´ıa y Compe i i idad, Sec e a ´ıa de Es ado de In-
es igaci´on, Desa ollo e Inno aci´on, Spain (ENE2017-88697-R),
–p io i y p og am 1648 - So wa e o Exascale Compu ing 214 (ExaFSA -
Exascale Simula ion o Fluid-S uc u e-Acous ics In e ac ions) o he
Ge man Resea ch Founda ion,
–and a FI PhD schola ship by he Ag`encia de Ges i´o d’Aju s Uni e si a is
i de Rece ca (AGAUR) o Gene ali a de Ca alunya (Spain).
The pe o mance measu emen s we e ca ied ou on he Supe MUC supe -
compu e a Leibniz Rechenzen um (LRZ) de Baye ischen Akademie de
Wissenscha en. The au ho s wish o hank LRZ o he compu ing ime
and he echnical suppo .
Re e ences
1. Y. Bazile s, V. M. Calo, Y. Zhang, T. J. R. Hughes, Isogeome ic luid-
s uc u e in e ac ion analysis wi h applica ions o a e ial blood low, Compu-
a ional Mechanics 38 (4-5) (2006) 310–322. doi:10.1007/s00466-006-0084-3.
2. K. Takizawa, Y. Bazile s, T. E. Tezduya , Space– ime and ALE–VMS ech-
niques o pa ien -speci ic ca dio ascula luid–s uc u e in e ac ion modeling,
A chi es o Compu a ional Me hods in Enginee ing 19 (2) (2012) 171–225.
doi:10.1007/s11831-012-9071-3.
3. J. Deg oo e, Pa i ioned simula ion o luid-s uc u e in e ac ion,
A chi es o Compu a ional Me hods in Enginee ing 20 (2013) 185–238.
doi:10.1007/s11831-013-9085-5.
4. G. Hou, J. Wang, A. Lay on, Nume ical me hods o luid-s uc u e in e -
ac ion - A e iew, Communica ions in Compu a ional Physics 12 (2) (2012)
337–377. doi:10.4208/cicp.291210.290411s.
5. F. Kong, X.-C. Cai, A scalable nonlinea luid–s uc u e in e ac ion sol e
based on a schwa z p econdi ione wi h isogeome ic uns uc u ed coa se
spaces in 3d, Jou nal o Compu a ional Physics 340 (2017) 498–518.
doi:10.1016/j.jcp.2017.03.043.
6. S. Depa is, D. Fo i, G. G andpe in, A. Qua e oni, FaCSI: A block pa allel
p econdi ione o luid–s uc u e in e ac ion in hemodynamics, Jou nal o
Compu a ional Physics 327 (2016) 700–718. doi:10.1016/j.jcp.2016.10.005.
7. F. Kong, V. Khey e s, E. Finol, X.-C. Cai, Simula ion o uns eady blood
lows in a pa ien -speci ic complian pulmona y a e y wi h a highly pa al-
lel monoli hically coupled luid-s uc u e in e ac ion algo i hm, In e na ional
jou nal o nume ical me hods in biomedical enginee ing 35 (7) (2019) e3208.
doi:10.1002/cnm.3208.
8. S. Ka aoka, S. Minami, H. Kawai, T. Yamada, S. Yoshimu a, A pa allel
i e a i e pa i ioned coupling analysis sys em o la ge-scale acous ic luid–
s uc u e in e ac ions, Compu a ional Mechanics 53 (6) (2014) 1299–1310.
doi:10.1007/s00466-013-0973-1.
Ti le Supp essed Due o Excessi e Leng h 33
9. J. Cajas, G. Houzeaux, M. V´azquez, M. Ga cia, E. Casoni, H. Calme , A. A -
igues, R. Bo ell, O. Lehmkuhl, D. Pas ana, e al., Fluid-s uc u e in e ac-
ion based on hpc mul icode coupling, SIAM Jou nal on Scien i ic Compu ing
40 (6) (2018) C677–C703. doi:10.1137/17M1138868.
10. S. Hewi , L. Ma ge s, A. Re ell, P. Pankaj, F. Le e o-
Flo encio, OpenFPCI: A pa allel luid–s uc u e in e ac ion ame-
wo k, Compu e Physics Communica ions 244 (2019) 469–482.
doi:h ps://doi.o g/10.1016/j.cpc.2019.05.016.
11. J. La son, R. Jacob, E. Ong, The model coupling oolki : a new o an90
oolki o building mul iphysics pa allel coupled models, The In e na ional
Jou nal o High Pe o mance Compu ing Applica ions 19 (3) (2005) 277–292.
doi:10.1177/1094342005056115.
12. H.-J. Bunga z, F. Lindne , B. Ga zhamme , M. Mehl, K. Scheu ele,
A. Shukae , B. Ueke mann, p eCICE–a ully pa allel lib a y o mul i-
physics su ace coupling, Compu e s & Fluids 141 (2016) 250–258.
doi:10.1016/j.comp luid.2016.04.003.
13. D. Thomas, M. L. Ce quaglia, R. Boman, T. D. Economon, J. J. Alonso,
G. Dimi iadis, V. E. Te apon, CUPyDO - An in eg a ed Py hon en i on-
men o coupled luid-s uc u e simula ions, Ad ances in Enginee ing So -
wa e 128 (2019) 69–85.
14. H.-J. Bunga z, F. Lindne , M. Mehl, K. Scheu ele, A. Shukae , B. Ueke -
mann, Pa i ioned luid–s uc u e–acous ics in e ac ion on dis ibu ed da a:
Coupling ia p ecice, in: H.-J. Bunga z, P. Neumann, W. E. Nagel (Eds.),
So wa e o Exascale Compu ing - SPPEXA 2013-2015, Sp inge In e na-
ional Publishing, Cham, 2016, pp. 239–266. doi:10.1007/978-3-319-40528-5-
11.
15. M. L. Ce quaglia, D. Thomas, R. Boman, V. Te apon, J.-P. Pon ho , A
ully pa i ioned lag angian amewo k o si p oblems cha ac e ized by ee
su aces, la ge solid de o ma ions and displacemen s, and s ong added-mass
e ec s, Compu e Me hods in Applied Mechanics and Enginee ing 348 (2019)
409–442. doi:10.1016/j.cma.2019.01.021.
16. P. Causin, J. F. Ge beau, F. Nobile, Added-mass e ec in he de-
sign o pa i ioned algo i hms o luid-s uc u e p oblems, Compu e
Me hods in Applied Mechanics and Enginee ing 194 (2005) 4506–4527.
doi:10.1016/j.cma.2004.12.005.
17. C. F¨o s e , W. A. Wall, E. Ramm, A i icial added mass ins abili ies in se-
quen ial s agge ed coupling o nonlinea s uc u es and incomp essible iscous
lows, Compu e Me hods in Applied Mechanics and Enginee ing 196 (2007)
1278–1293. doi:10.1016/j.cma.2006.09.002.
18. M. A. Fe n´andez, J.-F. Ge beau, C. G andmon , A p ojec ion semi-implici
scheme o he coupling o an elas ic s uc u e wi h an incomp essible luid,
In e na ional Jou nal o Nume ical Me hods in Enginee ing 69 (4) (2007)
794–821. doi:10.1002/nme.1792.
19. A. Nase i, O. Lehmkuhl, I. Gonzalez, E. Ba ons, C. D. P´e ez-Sega a,
A. Oli a, A semi-implici coupling echnique o luid–s uc u e in e ac ion
p oblems wi h s ong added-mass e ec , Jou nal o Fluids and S uc u es 80
(2018) 94–112. doi:10.1016/j.j luids uc s.2018.03.012.
20. A. Nase i, I. Gonzalez, A. Amani, C. D. P´e ez-Sega a, A. Oli a, A
second–o de ime accu a e semi–implici me hod o luid–s uc u e in-
e ac ion p oblems, Jou nal o Fluids and S uc u es 86 (2019) 135–155.
doi:10.1016/j.j luids uc s.2019.02.007.
34 Ali eza Nase i e al.
21. Te mo Fluids S.L., Te mo luids, h p://www. e mo luids.com (2020).
22. I. Rod iguez, R. Bo ell, O. Lehmkuhl, C. D. Pe ez Sega a, A. Oli a, Di ec
nume ical simula ion o he low o e a sphe e a Re = 3700, Jou nal o Fluid
Mechanics 679 (2011) 263–287. doi:10.1017/j m.2011.136.
23. I. Rod ´ıguez, O. Lehmkuhl, J. Chi a, R. Bo ell, A. Oli a, On he low pas a
ci cula cylinde om c i ical o supe -c i ical eynolds numbe s: Wake opol-
ogy and o ex shedding, In e na ional Jou nal o Hea and Fluid Flow 55
(2015) 91 – 103. doi:10.1016/j.ijhea luid low.2015.05.009.
24. N. Balc´aza , L. Jo e, O. Lehmkuhl, J. Cas o, J. Rigola, A ini e-
olume/le el-se me hod o simula ing wo-phase lows on uns uc-
u ed g ids, In e na ional jou nal o mul iphase low 64 (2014) 55–72.
doi:10.1016/j.ijmul iphase low.2014.04.008.
25. E. Gu i´e ez, F. Fa e, N. Balcaza , A. Amani, J. Rigola, Nume ical app oach
o s udy bubbles and d ops e ol ing h ough complex geome ies by using a
le el se –mo ing mesh–imme sed bounda y me hod, Chemical Enginee ing
Jou nal 349 (2018) 662–682. doi:10.1016/j.cej.2018.05.110.
26. P. Galione, O. Lehmkuhl, J. Rigola, A. Oli a, Fixed-g id nume ical model-
ing o mel ing and solidi ica ion using a iable he mo-physical p ope ies–
applica ion o he mel ing o n-oc adecane inside a sphe ical capsule,
In e na ional Jou nal o Hea and Mass T ans e 86 (2015) 721–743.
doi:10.1016/j.ijhea mass ans e .2015.03.033.
27. E. Ba ons, C. Olie , E. Gu ie ez, A. Nase i, C. D. P´e ez-Sega a, A
ini e olume me hod o sol e he os g ow h using dynamic meshes,
In e na ional Jou nal o Hea and Mass T ans e 124 (2018) 615–628.
doi:10.1016/j.ijhea mass ans e .2018.03.104.
28. G. Colome , R. Bo ell, F. X. T ias, I. Rod ´ıguez, Pa allel algo i hms o Sn
anspo sweeps on uns uc u ed meshes, Jou nal o Compu a ional Physics
232 (1) (2013) 118–135. doi:10.1016/j.jcp.2012.07.009.
29. L. Jo e, R. Bo ell, O. Lehmkuhl, A. Oli a, Pa allel load balancing s a egy
o olume-o - luid me hods on 3-d uns uc u ed meshes, Jou nal o Compu-
a ional Physics 282 (2015) 269–288. doi:10.1016/j.jcp.2014.11.009.
30. R. Bo ell, J. Chi a, O. Lehmkuhl, G. Oya zun, I. Rod ´ıguez, A. Oli a,
Op imising he Te mo luids CFD code o pe ascale simula ions, In e na-
ional Jou nal o Compu a ional Fluid Dynamics 30 (6) (2016) 425–430.
doi:10.1080/10618562.2016.1221503.
31. R. Ve s appen, A. Veldman, Symme y-p ese ing disc e iza ion o u -
bulen low, Jou nal o Compu a ional Physics 187 (1) (2003) 343–368.
doi:10.1016/S0021-9991(03)00126-8.
32. F. X. T ias, O. Lehmkuhl, A. Oli a, C. D. P´e ez-Sega a, R. Ve s appen,
Symme y-p ese ing disc e iza ion o Na ie -S okes equa ions on colloca ed
uns uc u ed g ids, Jou nal o Compu a ional Physics 258 (2014) 246–267.
doi:10.1016/j.jcp.2013.10.031.
33. L. Jo e, O. Lehmkuhl, J. Ven osa, F. X. T ias, A. Oli a, Conse a ion p op-
e ies o uns uc u ed ini e- olume mesh schemes o he na ie -s okes equa-
ions, Nume ical Hea T ans e , Pa B: Fundamen als 65 (1) (2014) 53–79.
doi:10.1080/10407790.2013.836335.
34. P. Thomas, C. Lomba d, Geome ic conse a ion law and i s applica ion o
low compu a ions on mo ing g ids, AIAA jou nal 17 (10) (1979) 1030–1037.
doi:10.2514/3.61273.
35. M. Lesoinne, C. Fa ha , Geome ic conse a ion laws o low p oblems wi h
mo ing bounda ies and de o mable meshes, and hei impac on ae oelas-
Ti le Supp essed Due o Excessi e Leng h 35
ic compu a ions, Compu e me hods in applied mechanics and enginee ing
134 (1-2) (1996) 71–90. doi:10.1016/0045-7825(96)01028-6.
36. O. Es uch, O. Lehmkuhl, R. Bo ell, C. D. P. Sega a, A. Oli a, A pa allel
adial basis unc ion in e pola ion me hod o uns uc u ed dynamic meshes,
Compu e s and Fluids 80 (2013) 44–54. doi:10.1016/j.comp luid.2012.06.015.
37. P. Ca di , I. Demi dˇzi´c, Thi y yea s o he ini e olume me hod o solid
mechanics, a Xi p ep in a Xi :1810.02105 (2018).
38. H. Jasak, E o analysis and es ima ion o he ini e olume me hod wi h
applica ions o luid lows, Ph.D. hesis, Impe ial College London (Uni e si y
o London) (1996).
39. A. J. Macleod, Accele a ion o ec o sequences by mul i-dimensional ∆2
me hods, Communica ions in Applied Nume ical Me hods 2 (4) (1986) 385–
392.
40. ˇ
Z. Tuko i´c, A. I anko i´c, A. Ka aˇc, Fini e- olume s ess analysis in mul i-
ma e ial linea elas ic body, In e na ional Jou nal o Nume ical Me hods in
Enginee ing 93 (4) (2013) 400–419.
41. P. Ca di , Tuko i´c, H. Jasak, A. I anko i´c, A block-coupled Fini e Volume
me hodology o linea elas ici y and uns uc u ed meshes, Compu e s and
S uc u es 175 (2016) 100–122. doi:10.1016/j.comps uc.2016.07.004.
42. P. Chand asheka , A. Ga g, Ve ex-cen oid ini e olume scheme on e ahe-
d al g ids o conse a ion laws, Compu e s & Ma hema ics wi h Applica ions
65 (1) (2013) 58–74.
43. I. Gonz´alez, A. Nase i, J. Chi a, J. Rigola, C. D. P´e ez-Sega a, An enhanced
ini e olume based sol e o he moelas ic ma e ials in luid-s uc u e cou-
pled p oblems, in: 6 h Eu opean Con e ence on Compu a ional Mechanics
(ECCM-ECFD 2018), ECCOMAS, Glasgow, UK, June 2018.
44. U. K¨u le , W. A. Wall, Fixed-poin luid-s uc u e in e ac ion sol e s
wi h dynamic elaxa ion, Compu a ional Mechanics 43 (2008) 61–72.
doi:10.1007/s00466-008-0255-5.
45. J. F. Ge beau, M. Vid ascu, A quasi-New on algo i hm based on a
educed model o luid-s uc u e in e ac ion p oblems in blood lows,
ESAIM: Ma hema ical Modelling and Nume ical Analysis 37 (2003) 631–647.
doi:10.1051/m2an:2003049.
46. C. Michle , E. H. V. B ummelen, R. D. Bo s , An in e ace New on-K ylo
sol e o luid-s uc u e in e ac ion, In e na ional Jou nal o Nume ical
Me hods in Fluids 47 (10-11) (2005) 1189–1195. doi:10.1002/ ld.850.
47. K. Scheu ele, M. Mehl, Robus mul isecan quasi-New on a ian s o
pa allel luid-s uc u e simula ions–and o he mul iphysics applica ions,
SIAM Jou nal on Scien i ic Compu ing 39 (5) (2017) S404–S433.
doi:10.1137/16M1082020.
48. H.-J. Bunga z, F. Lindne , M. Mehl, B. Ueke mann, A plug-and-play cou-
pling app oach o pa allel mul i- ield simula ions, Compu a ional Mechanics
55 (6) (2015) 1119–1129. doi:10.1007/s00466-014-1113-2.
49. M. Mehl, B. Ueke mann, H. Bijl, D. Blom, B. Ga zhamme , A. Van Zuijlen,
Pa allel coupling nume ics o pa i ioned luid–s uc u e in e ac ion simula-
ions, Compu e s & Ma hema ics wi h Applica ions 71 (4) (2016) 869–891.
doi:10.1016/j.camwa.2015.12.025.
50. G. Oya zun, R. Bo ell, A. Go obe s, A. Oli a, MPI-CUDA spa se ma ix–
ec o mul iplica ion o he conjuga e g adien me hod wi h an app ox-
ima e in e se p econdi ione , Compu e s & Fluids 92 (2014) 244–252.
doi:10.1016/j.comp luid.2013.10.035.
36 Ali eza Nase i e al.
51. G. Oya zun, R. Bo ell, A. Go obe s, A. Oli a, Po able implemen a ion
model o CFD simula ions. applica ion o hyb id CPU/GPU supe compu -
e s, In e na ional Jou nal o Compu a ional Fluid Dynamics 31 (9) (2017)
396–411. doi:10.1080/10618562.2017.1390084.
52. G. Oya zun, R. Bo ell, A. Go obe s, F. Man o ani, A. Oli a, E i-
cien CFD code implemen a ion o he ARM-based Mon -Blanc a -
chi ec u e, Fu u e gene a ion compu e sys ems 79 (2018) 786–796.
doi:10.1016/j. u u e.2017.09.029.
53. G. Ka ypis, V. Kuma , A as and high quali y mul ile el scheme o pa i-
ioning i egula g aphs, SIAM Jou nal on Scien i ic Compu ing 20 (1) (1998)
359–392. doi:10.1137/S1064827595287997.
54. The HDF G oup, Hie a chical Da a Fo ma , e sion 5,
h p://www.hd g oup.o g/HDF5/ (1997-2019).
55. A. To oun e oush, N. Eb ahimi Pou , J. Sch ode , S. Rolle , M. Mehl,
A new load balancing app oach o coupled mul i-physics simula ions, in:
In P oceedings o IEEE In e na ional Pa allel and Dis ibu ed P ocess-
ing Symposium Wo kshops (IPDPSW), Rio de Janei o, B azil, May 2019.
doi:10.1109/IPDPSW.2019.00115.
56. A. Calo oiu, D. Beckinsale, C. W. Ea l, T. Hoe le , I. Ka lin, M. Schulz,
F. Wol , Fas mul i-pa ame e pe o mance modeling, in: 2016 IEEE In e -
na ional Con e ence on Clus e Compu ing (CLUSTER), IEEE, 2016, pp.
172–181.
57. Leibniz Supe compu ing Cen e o he Ba a ian Academy o Sciences and
Humani ies , h ps://www.l z.de.
58. L. Fo maggia, J. F. Ge beau, F. Nobile, A. Qua e oni, On he coupling o
3D and 1D Na ie -S okes equa ions o low p oblems in complian essels,
Compu e Me hods in Applied Mechanics and Enginee ing 191 (6-7) (2001)
561–582. doi:10.1016/S0045-7825(01)00302-4.
59. M. A. Fe n´andez, M. Moubachi , A New on me hod using exac Jacobians o
sol ing luid–s uc u e coupling, Compu e s and S uc u es 83 (2005) 127–
142. doi:10.1016/j.comps uc.2004.04.021.
60. A. Eken, M. Sahin, A pa allel monoli hic algo i hm o he nume ical simula-
ion o la ge-scale luid s uc u e in e ac ion p oblems, In e na ional Jou nal
o Nume ical Me hods in Fluids 80 (12) (2016) 687–714. doi:10.1002/ ld.4169.
61. A. Lozo skiy, M. A. Olshanskii, Y. V. Vassile ski, Analysis and assessmen
o a monoli hic FSI ini e elemen me hod, Compu e s & Fluids 179 (2019)
277–288. doi:10.1016/j.comp luid.2018.11.004.
62. 2nd CFD challenge p edic ing pa ien -speci ic hemody-
namics a es and s ess h ough an ao ic coa c a ion,
h p://www. ascula model.o g/miccai2013/ (2013).
63. M. A. Fe n´andez, M. Landajuela, M. Vid ascu, Fully decoupled ime-ma ching
schemes o incomp essible luid/ hin-walled s uc u e in e ac ion, Jou nal o
Compu a ional Physics 297 (2015) 156–181. doi:10.1016/j.jcp.2015.05.009.
64. N. Wes e ho , J.-W. Lankhaa , B. E. Wes e ho , The a e ial Windkessel,
Medical & biological enginee ing & compu ing 47 (2) (2009) 131–141.
doi:10.1007/s11517-008-0359-2.
65. S. Pan , B. Fab `eges, J.-F. Ge beau, I. Vignon-Clemen el, A me hodolog-
ical pa adigm o pa ien -speci ic mul i-scale c d simula ions: om clinical
measu emen s o pa ame e es ima es o indi idual analysis, In e na ional
jou nal o nume ical me hods in biomedical enginee ing 30 (12) (2014) 1614–
1648. doi:10.1002/cnm.2692.

Ti le Supp essed Due o Excessi e Leng h 37
66. ˇ
Z. Tuko i´c, A. Ka aˇc, P. Ca di , H. Jasak, A. I anko i´c, OpenFOAM ini e
olume sol e o luid-solid in e ac ion, T ansac ions o FAMENA 42 (3)
(2018) 1–31. doi:10.21278/TOF.42301.
67. J. Seo, D. E. Schia azzi, A. L. Ma sden, Pe o mance o p econdi ioned i -
e a i e linea sol e s o ca dio ascula simula ions in igid and de o mable
essels, Compu a ional Mechanics 64 (3) (2019) 717–739. doi:10.1007/s00466-
019-01678-3.
68. D. Jodlbaue , U. Lange , T. Wick, Pa allel block-p econdi ioned mono-
li hic sol e s o luid-s uc u e in e ac ion p oblems, In e na ional Jou -
nal o Nume ical Me hods in Enginee ing 117 (6) (2019) 623–643.
doi:10.1002/nme.5970.
69. D. Fo i, A. Qua e oni, S. Depa is, e al., A pa allel algo i hm o he so-
lu ion o la ge-scale noncon o ming luid-s uc u e in e ac ion p oblems in
hemodynamics, Jou nal o Compu a ional Ma hema ics 35 (3) (2017) 363–
380. doi:10.4208/jcm.1702-m2016-0630.