A scalable framework for the partitioned solution of fluid-structure interaction problems

UPCommons Po al del coneixemen obe de la UPC h p://upcommons.upc.edu/e-p in [s This is a pos -pee - e iew, p e-copyedi e sion o an a icle published in Compu a ional mechanics . The inal au hen ica ed e sion is a ailable online a : h ps://doi.o g/10.1007/s00466-020-01860-y. Compu a ional Mechanics manusc ip No. (will be inse ed by he edi o ) A Scalable F amewo k o he Pa i ioned Solu ion o Fluid-S uc u e In e ac ion P oblems Ali eza Nase i1, Amin To oun e oush2, Ignacio Gonz´alez1, Mi iam Mehl2, Ca los Da id P´e ez-Sega a1 1Hea and Mass T ans e Technological Cen e (CTTC) Uni e si a Poli ´ecnica de Ca alunya - Ba celonaTech (UPC) ESEIAAT, C/ Colom 11, 08222 Te assa (Ba celona), Spain 2Ins i u e o Pa allel and Dis ibu ed Sys ems (IPVS) Uni e si y o S u ga Uni e si ¨a ss aße 38, D-70569 S u ga , Ge many The da e o eceip and accep ance will be inse ed by he edi o Abs ac In his wo k, we p esen a scalable and e icien pa allel sol e o he pa i ioned solu ion o luid-s uc u e in e ac ion p oblems h ough mul i–code coupling. Two ins ances o an in–house pa allel so wa e, Te - moFluids, a e used o sol e he luid and he s uc u al sub–p oblems, cou- pled oge he on he in e ace ia he p eCICE coupling lib a y. Fo luid low, he A bi a y Lag angian-Eule ian o m o he Na ie -S okes equa ions is sol ed on an uns uc u ed con o ming g id using a second–o de ini e– olume disc e iza ion. A pa allel dynamic mesh me hod o uns uc u ed meshes is used o ack he mo ing bounda y. Fo he s uc u al p oblem, he nonlinea elas odynamics equa ions a e sol ed on an uns uc u ed g id using a second–o de ini e– olume me hod. A semi–implici FSI coupling me hod is used which seg ega es he luid p essu e e m and couples i s ongly o he s uc u e, while he emaining luid e ms and he geome - ical nonlinea i ies a e only loosely coupled. A obus and ad anced mul i– ec o quasi–New on me hod is used o he coupling i e a ions be ween he sol e s. Bo h he luid and he s uc u al sol e use dis ibu ed–memo y pa - allelism. The in a–sol e communica ion equi ed o da a upda e in he solu ion p ocess is ca ied ou using non–blocking poin – o–poin communi- ca o s. The in e –code communica ion is ully pa allel and poin – o–poin , a oiding any cen al communica ion uni . Inside each single–physics sol e , he load is balanced by di iding he compu a ional domain in o ai ly equal blocks o each p ocess. Addi ionally, a load balancing model is used a he in e –code le el o minimize he o e all idle ime o he p ocesses. Two 2 Ali eza Nase i e al. p ac ical es cases in he con ex o hemodynamics a e s udied, demon- s a ing he accu acy and compu a ional e iciency o he coupled sol e . S ong scalabili y es esul s show a pa allel e iciency o 83% on 10,080 CPU co es. Key wo ds Fluid-S uc u e In e ac ion; Pa i ioned Me hod; Mul i-Code Coupling; Scalabili y; High Pe o mance Compu ing 1 In oduc ion Nume ical me hods o sol e luid-s uc u e in e ac ion (FSI) p oblems can be b oadly di ided in o wo ca ego ies, monoli hic and pa i ioned. In a monoli hic app oach, he luid and he solid equa ions a e disc e ized and sol ed as a single la ge sys em, inhe en ly accoun ing o hei mu ual in- e ac ion (see e.g. [1,2]). In a pa i ioned app oach, on he o he hand, he FSI p oblem is di ided in o wo domains, o luid and solid. As a esul , hese me hods use sepa a e sol e s o luid and s uc u al sub-p oblems and adop a coupling echnique o accoun o he in e ac ion o he domains. One o he big ad an ages o he pa i ioned app oach is he possibili y o use he mos adap ed and well- alida ed nume ical me hods o each sub-p oblem. Mo eo e , i allows using p e iously de eloped and compu a- ionally op imized luid and s uc u al sol e codes, hus sa ing excessi e so wa e de elopmen e o [3,4]. Howe e , he pa i ioned app oach in o- duces a new challenge o he p oblem which is he coupling be ween he sepa a e sol e s. This challenge has wo di e en aspec s. The i s aspec conce ns he me hodology, i.e., coupling dis inc se s o (disc e ized) pa - ial di e en ial equa ions and ensu ing he physical equilib ium condi ions on he in e ace. The second aspec is ela ed o he implemen a ion, i.e., coupling wo pa allel codes wi h di e en modules and s uc u es, wi h he aim o achie ing an e icien and scalable o e all so wa e. Mode n scien i ic and enginee ing p oblems a e o en e y complex and equi e a huge compu a ional e o . The e o e, any simula ion so wa e mus be able o e icien ly un on massi ely pa allel compu e s. The pa al- lel e iciency o a sol e is c ucial in o de o be able o use he a ailable esou ces adequa ely and pe o m a complex calcula ion. Du ing he ecen yea s, e icien pa allel codes ha e been de eloped o many single-physics p oblems, pa icula ly luid and s uc u e sys ems. While he monoli hic app oach o sol e FSI p oblems equi es de eloping a new sol e and imple- men ing a so wa e, ollowing a pa i ioned app oach c ea es he oppo u- ni y o exploi he p e iously-de eloped e icien codes o FSI simula ions. Ne e heless, using e icien single-physics simula ion codes does no au o- ma ically gua an ee achie ing a good pa allel e iciency o a coupled mul i- physics simula ion. Mul i-code coupling in oduces se e al new challenges. One pa icula di icul y is he da a exchange be ween sepa a e codes which o en use di e en da a s uc u es and could e en be w i en in di e en Ti le Supp essed Due o Excessi e Leng h 3 languages. An e icien ly pa allel mechanism o da a exchange be ween he codes is c ucial o achie ing pa allel e iciency on he coupled amewo k. Mo eo e , by coupling wo pa allel codes, a new le el o load balancing is in oduced o he p oblem, as each code would be esponsible o a di e en amoun o calcula ions on a di e en numbe o CPUs. Recen e o s ha e been made o de elop e icien and scalable mul i- physics sol e s (pa icula ly o FSI p oblems) using ei he monoli hic [5– 7] o pa i ioned [8–10] app oaches. Monoli hic FSI sol e s a e na u ally mo e sui able o massi e pa alleliza ion, since he whole coupled p oblem is sol ed as a single sys em o equa ions using a single sol e . The e o e, his app oach does no ace wo main compu a ional challenges o he pa i- ioned app oach, i.e., pa allel communica ion be ween sepa a e sol e s and he in e -sol e load balancing. Ne e heless, achie ing a high scalabili y o monoli hic me hods is also a e y challenging ask, as FSI p oblems a e highly complex and nonlinea . In his wo k, we ocus on he pa i ioned app oach in o de o c ea e a scalable and e icien FSI sol e using exis ing single–physics sol e s. Loss o pa allel e iciency in mul i-code coupling is conside ed a main d awback o pa i ioned me hods. This is due o he challenging issues on da a s uc u ing, domain decomposi ion, pa allel da a communica ion and in e -sol e load balancing. Cajas e al. [9] p esen ed a pa allel pa i ioned sol e o FSI p oblems, based on mul i-code cou- pling. The single-physics sol e s we e wo ins ances o an in-house code, communica ing di ec ly ia MPI messages. The in e -code communica ion was pa allel and poin - o-poin while each sol e used a mas e -wo ke ap- p oach in e nally. An in e -code load balancing me hod was p oposed based on o e loading he a ailable co es in o de o minimize he idle ime. The coupled amewo k was shown o scale well o 1280 MPI p ocesses on 768 CPU co es [9]. Hewi e al. [10] de eloped a mul i-code coupled ame- wo k using open sou ce single-physics sol e s (OpenFOAM o luid and Pa aFEM o s uc u e). The communica ion be ween he sol e s was ca - ied ou sequen ially using a mas e ank. The sol e was shown o scale well on 1,536 co es o a coupled FSI p oblem. This appea s o be he high- es scalabili y epo ed o a pa i ioned FSI sol e in he li e a u e. Apa om sequen ial communica ion, a majo p oblem o he amewo k in [10] is ha i mus use he same numbe o co es o he luid and he solid sol e . This p oblem a ises om he complexi y o he wo-laye ed da a communi- ca ion s uc u e (inside each sol e and be ween hem), which is essen ial in a pa i ioned mul i-physics sol e . Using he same numbe o co es o he sol e s is a se ious d awback, as i e ec i ely ules ou any load balancing be ween he sol e s, g ea ly educing he compu a ional e iciency o he coupled amewo k. A compu a ionally e icien pa i ioned FSI sol e ha could scale on se e al housand p ocesso s is s ill missing in he li e a u e. Towa ds ha goal, we ocus on he common issues o pa allel pa i ioned sol e s and p opose e ec i e solu ions o o e come hem. In ou mul i-code coupled sol e , he communica ion be ween he sepa- a e codes is managed by using a communica ion lib a y. The communica- 4 Ali eza Nase i e al. ion lib a y ecei es he da a om each code ia an adap e and con ains unc ions o acili a e he exchange o da a. I also accele a es he cou- pling i e a ions be ween he sol e s in he s ongly-coupled con igu a ion. Adap e s a e used o connec he sol e s and adjus he da a s uc u e while ans e ing da a om one sol e , h ough he communica ion lib a y, o he o he sol e . This app oach leads o a obus and powe ul scheme o cou- ple di e en codes o mul i-physics simula ions. An immedia e ad an age o his me hod is ha one o bo h o he single-physics sol e s could be eplaced by o he sol e s wi h ela i ely small changes in he code, in he bes case limi ed o he adap e . Examples o such communica ion lib a ies can be ound in [11,8,12,13]. Examples o mul i-physics simula ion so wa e using communica ion lib a ies can be ound in [8,14,13,15]. The e a e o he examples in he li e a u e whe e he da a exchange is handled di ec ly and he communica ion unc ions a e included in he single-physics codes hem- sel es. This app oach is shown o be e icien when wo ins ances o he same code a e coupled (e.g. [9]). Howe e , i is no as obus and powe ul as he i s app oach, especially in cases whe e wo di e en codes a e being coupled. Mo eo e , i does no allow o exchange one o bo h sol e s. E en hough in he cu en wo k we use wo ins ances o he same sol e o he single–physics sub-p oblems, we ollow he i s app oach (using a coupling lib a y) o c ea e a obus and lexible amewo k. A he coupling me hodology le el, pa i ioned me hods a e gene ally di ided in o explici (o loosely-coupled) and implici (o s ongly-coupled) schemes. Explici me hods sol e he luid and s uc u e equa ions only once pe ime s ep, using da a om he p e ious solu ion o he pa ne sol e . The e o e, explici me hods do no sa is y he exac equilib ium condi ions a he in e ace, which causes ins abili y issues in many FSI p oblems. The so-called added-mass ins abili y is pa icula ly s ong in FSI p oblems wi h incomp essible low, a slende in e ace, and simila densi ies o luid and solid [16,17]. Implici me hods, on he o he hand, en o ce he equilib ium condi ion a he in e ace h ough coupling i e a ions be ween luid and s uc u al sol e s. These me hods a e s able o p oblems wi h s ong added- mass e ec . Howe e , hei compu a ional cos is gene ally high due o he epe i i e solu ion o he go e ning equa ions a each ime s ep [3,4]. In he ecen ly in oduced semi-implici coupling app oach [18–20], he luid p essu e e m is seg ega ed and s ongly coupled o he s uc u e, while he emaining luid e ms a e only loosely coupled. S ong coupling o he luid p essu e and s uc u al de o ma ion elimina es he added-mass ins a- bili y issue, while loose coupling o he emaining luid e ms helps a oiding excessi e compu a ional cos [18,19]. In his wo k, wo ins ances o a pa allel in-house code, Te moFluids [21], a e used o sol e he luid and he s uc u e p oblem. Te moFluids is a o- bus gene al-pu pose so wa e o luid and s uc u e p oblems, using s a e- o - he-a me hods o u bulen low [22,23], mul iphase low [24,25], and complex he mal sys ems [26,27], wi h a high compu a ional e iciency and pa allel scalabili y [28–30]. I p esen s a conse a i e disc e iza ion o he Ti le Supp essed Due o Excessi e Leng h 5 go e ning equa ions on uns uc u ed g ids based on a ini e- olume me hod. I is also equipped wi h dynamic-mesh schemes o ack he mo ing bound- a y. The coupling o he codes is ca ied ou using he p eCICE coupling lib a y [12]. The p eCICE lib a y p o ides communica ion, da a mapping and equa ion coupling o su ace coupled mul i-physics applica ions in a modula manne . I o e s a ully pa allel poin - o-poin communica ion, ad anced quasi-New on i e a i e coupling schemes and a ious ad anced mapping me hods (bo h consis en and conse a i e). A semi-implici FSI coupling app oach p oposed in [19] is applied, which e ec i ely seg ega es he luid p essu e e m and couples i s ongly o he s uc u e. The e- maining luid e ms and he geome ical non-linea i ies a e ea ed explic- i ly, educing he compu a ional cos o he nume ical solu ion. P ac ical es cases in he con ex o biological low ( low inside de o mable essels) a e s udied and he scalabili y o he o e all amewo k is e alua ed. The emainde o his a icle is o ganized as ollows. In Sec ion 2, he go e ning equa ions o each sub-p oblem and he coupling condi ions a e p esen ed. Sec ion 3 desc ibes he p oposed nume ical me hods. Sec ion 4 p esen s he pa alleliza ion me hod o each single-physics sol e , as well as he in e -code communica ions and load balancing. Nume ical es s a e p esen ed in Sec ion 5, while Sec ion 6 summa izes and concludes he a icle. 2 Go e ning Equa ions In his sec ion, we p esen he luid and s uc u e go e ning equa ions and he coupling condi ions on hei common in e ace. The luid and he s uc- u al domain a e e e ed o as Ω ( )⊂R3×(0, T) and Ωs( )⊂R3×(0, T), espec i ely, whe e ∈(0, T ) deno es ime. The luid-s uc u e in e ace is he common bounda y o he domains, deno ed by Γ( ) = ∂Ω ( )∩∂Ωs( ). An A bi a y Lag angian-Eule ian (ALE) o mula ion oge he wi h a con- o ming mesh echnique is used o sol e he luid low in a mo ing domain. A Lag angian o mula ion is used o he s uc u al equa ions. The uns eady low o an incomp essible iscous luid is go e ned by he Na ie -S okes equa ions. An ALE o mula ion o hese equa ions in a mo ing domain is gi en by ∂u ∂ +c· ∇u=1 ρ ∇ · σ (1) ∇ · u= 0 (2) whe e uis he luid eloci y and ρ he luid densi y. Vec o cis he ALE con ec i e eloci y c=u−w, which is he luid eloci y ela i e o a domain mo ing wi h a eloci y w. The s ess enso σ o an incomp essible New onian luid is de ined as σ =−pI+µ (∇u+∇uT) (3) 6 Ali eza Nase i e al. whe e pis he luid p essu e, I he uni enso and µ is he dynamic is- cosi y o he luid. The s uc u al domain is go e ned by he conse a ion laws o mass and momen um, whose Lag angian o m is gi en by ρ0 s=ρsJ(4) ∂ ∂ ρs ∂d ∂ =∇ · σs(5) whe e supe sc ip 0 e e s o he e e ence ma e ial (unde o med) con igu- a ion o he body, ρsis he s uc u al densi y and dis he displacemen om he e e ence con igu a ion. The enso σsis he Cauchy s ess enso , which can be ela ed o he displacemen ield by he hype elas ic cons i u- i e model o Sain Venan -Ki chho σs=B 2J[2µs(B−I) + λs (B−I)] (6) whe e Bis he le Cauchy-G een de o ma ion enso B=F·FT, and µs and λsa e he Lam´e’s pa ame e s. The ma e ial de o ma ion enso Fis e alua ed as F=I+∇dand i s de e minan is deno ed by J= de (F). The physical equilib ium on he luid-solid common bounda y (kine- ma ic and dynamic equilib ium) cons i u es he coupling condi ions on he in e ace. Fo a non-slip ype in e ace hey ead uΓ=∂dΓ ∂ (7) σsnΓ=σ nΓ(8) a Γ, whe e nΓis he uni no mal ec o on he in e ace. 3 Nume ical Me hods In his sec ion we p esen he nume ical me hods o disc e iza ion and solu ion o he single-physics p oblems, as well as he coupling me hod. Th oughou he de elopmen o ou coupled amewo k and he single- physics sol e s, an emphasis was made on making he so wa e simple, modula and, as a as possible, ma ix- ee. Mo eo e , he luid sol e was mainly de eloped o u bulen low simula ions. The e o e, many aspec s o he disc e iza ion and he nume ical me hods co espond o he pa icula conside a ions o u bulen lows. Ti le Supp essed Due o Excessi e Leng h 7 3.1 Fluid Sol e Fo luid low, a ac ional-s ep p ojec ion me hod along wi h an explici ime ad ancemen is used o sol e he eloci y-p essu e coupling o he mo- men um equa ion. This leads o a h ee s ep solu ion o he luid go e ning equa ions om ime s ep n o n+ 1, wi h a ime inc emen o ∆ up=un−∆ [3 2(cn· ∇un−µ ρ ∆un)−1 2(cn−1· ∇un−1−µ ρ ∆un−1)] (9) ∆ ρ ∆pn+1 =∇ · up(10) un+1 =up−∆ ρ ∇pn+1 (11) in Ωn+1 , whe e upis a p edic ed eloci y ield which does no sa is y he in- comp essibili y condi ion (Eq. (2)). This in e media e eloci y ield is hen p ojec ed on o a di e gence- ee ield h ough he co ec ion a Eq. (11). An explici Adams-Bash o h me hod is used o he con ec i e and di u- si e e ms in Eq. (9). Using an explici me hod is pa icula ly p e e ed in u bulen low simula ions whe e small ime s eps a e indispensable. F om a compu a ional poin o iew, i a oids sol ing a nonlinea sys em o he luid eloci y ield. I also o e s ad an ages in pa alleliza ion as only he in o ma ion om he p e ious ime s ep is equi ed (only one episode o da a upda e be ween nodes is equi ed a each ime s ep). A ini e- olume me hod is used o he spa ial disc e iza ion o he luid equa ions on a colloca ed, uns uc u ed mesh wi h second-o de symme y- p ese ing schemes. Symme y-p ese ing schemes conse e he kine ic en- e gy o he low a he disc e e le el which is c ucially impo an in u bulen low simula ions [31,32]. A Jacobi–p econdi ioned conjuga e g adien sol e is used o sol e he Poisson equa ion o p essu e. Mo e de ails o he nu- me ical me hods o luid low equa ions can be ound in [33,32]. We use a con o ming mesh echnique o ack he mo ing bounda y, hus he luid mesh needs o mo e in o de o adap o he new loca ion o he in e ace. The ansla ed mesh and he e alua ed domain eloci y wmus sa is y he Geome ic Conse a ion Law (GCL) [34,35]. The GCL gua an ees ha no olume is los while mo ing he g id, and a cons an ield is p ese ed by he ALE scheme. Fo any con ol olume (CV) in he luid domain, he GCL is s a ed as ∂ ∂ −Zs w·dA= 0 (12) whe e and ss and o he olume and he bounda y su ace o he CV, espec i ely, and Ais he a ea ec o poin ing ou wa d. A pa allel mo ing mesh echnique based on he adial basis unc ion in e pola ion me hod [36] is used o mo e he luid g id in acco dance wi h 8 Ali eza Nase i e al. he new loca ion o he in e ace and de ine he disc e ized luid domain a he new ime s ep Ωn+1 . The me hod uses he known displacemen on he in e ace o e alua e an in e pola ed alue o he in e io e ices o he luid g id. A g ea ad an age o his me hod is ha i does no need he connec i i y o he mesh elemen s and can be applied o bo h s uc u ed and uns uc u ed g ids. Mo eo e , i only equi es sol ing a linea sys em o equa ions whose size is limi ed o he numbe o e ices on he luid-solid in e ace. A de ailed desc ip ion o he mo ing mesh me hod can be ound in [36,19]. A e he luid mesh is mo ed, he domain eloci y is e alua ed a he su aces o each con ol olume. We e alua e he su ace eloci ies based on he GCL law in o de o exac ly sa is y i . The ime a e o change o olume o a CV is equal o he sum o olumes swep by i s aces. In his wo k, we e alua e he domain eloci y w ace a each ace based on he olume swep by ha ace. Wi h a second-o de backwa d disc e iza ion, i eads wn+1 ace =3 2(δ A∆ n)n+1 −1 2(δ A∆ n)n(13) whe e Ais he su ace a ea, n he uni no mal ec o o he ace, ∆ he ime s ep, and δ is he olume swep by he ace a one ime s ep. A mo e de ailed desc ip ion o he e alua ion o he domain eloci y ield and he sa is ac ion o he geome ic conse a ion law can be ound in [20]. 3.2 Solid Sol e Aiming a low-induced de o ma ions and oscilla ions o he s uc u e, he solid equa ions a e sol ed using an implici ime in eg a ion, whe e bo h he ine ial and su ace o ces in Eq. (5) a e e alua ed a he cu en ime ins an n+1. A cell-cen e ed ini e- olume me hod wi h a o al Lag angian app oach is used o he spa ial disc e iza ion. The momen um equa ion is in eg a ed on he unde o med con igu a ion. A e applying he Gauss heo em on he s ess di e gence and ela ing he unde o med and cu en a ea ec o s wi h Nanson’s o mula, he momen um balance can be w i en as Z 0 ρ0 s ∂2d ∂ 2d 0=Zs0 σs(JF−Tn0)ds0(14) The accele a ion o he ine ial e m is compu ed acco ding o he ape- zoidal ule ∂2d ∂ 2=4 ∆ 2dn+1 −4 ∆ 2dn−4 ∆ ∂d ∂ n −∂2d ∂ 2n (15) The dependencies be ween di e en di ec ions o he displacemen and he geome ical and ma e ial non-linea i ies ound in he igh -hand side o Eq. (14) a e de e ed o he sou ce e m o he sys em. The e o e, ou e Ti le Supp essed Due o Excessi e Leng h 15 b a y [12]. In e –code communica ion ia p eCICE can be ei he based on MPI po s (MPI-2.0) o on lowe le el TCP/IP socke s. In he cu en wo k we ha e used TCP/IP socke s, because he MPI po s unc ionali y is miss- ing in he implemen ed MPI e sions on he supe compu e ha we used o ou es s (Supe MUC supe compu e a he Leibniz Supe compu ing Cen e o he Ba a ian Academy o Sciences). To es ablish communica- ion channels be ween he pa icipan s’ anks, he luid and he solid mesh pa i ions a e ini ially analyzed o ind logical connec ions be ween hem. Once he communica ion channels a e es ablished, da a a e exchanged in an asynch onous way o a oid unnecessa y blocking. Es ablishing he commu- nica ion channels is ca ied ou only once a he ini ializa ion s age. Fo he es o he un- ime, he same channels a e used o da a exchange. Since using a cen al communica ion ins ance can deg ade he scalabili y o ou amewo k, we use a ully pa allel poin - o-poin communica ion scheme, hus no cen al se e –like uni is used. This way, da a exchange happens locally be ween he connec ed anks. I is ob ious ha , in case o ha ing a big mesh o a high numbe o anks and using a cen al communica ion ins ance, a single mas e ank o se e would be a se e e bo leneck. Figu e 2 schema ically shows he pa allel s uc u e o he coupled ame- wo k and he di e en le els o communica ion. 4.3 Load Balancing Load imbalance can be a se ious sou ce o ine iciency in pa allel simula ion codes. Inside each single–physics sol e , he load is balanced by di iding he compu a ional domain in o ai ly equal blocks o each p ocess, as explained in Sec ion 4.1. Howe e , in a pa i ioned FSI simula ion, he e exis s a new le el o load balancing be ween he single–physics sol e s. In his wo k wi h a simul aneous execu ion o he sol e s (Jacobi- ype p oblem desc ibed in Sec ion 3.3), bo h sol e s mus inish an i e a ion and send he ou pu o he pa ne sol e be o e he nex i e a ion can s a . This means, in case he a ailable CPUs a e no dis ibu ed op imally among he sol e s, one sol e will be wai ing o he pa ne o inish i s own compu a ions. In con as o he load balancing wi hin a single–physics sol e , load balancing ac oss he codes is mo e di icul since we do no know a p io i he ela ion be ween he wo k pe cell o di e en sol e s. In his wo k, we ollow he app oach p oposed in [55] o add ess his issue. We i s model he sol e pe o mance agains he numbe o anks o each domain, and hen sol e an in ege op imiza ion p oblem o ind he app op ia e pai o ank numbe s o he domains ha minimizes he wai ing ime. Since analy ical modeling o he sol e s’ pe o mance is e y complex, we use an empi ical eg ession-based app oach ins ead, aiming o ind an app op ia e pe o mance model. We un msimula ions wi h di e en numbe o co es, yielding a se o da a poin s consis ing o pai s (n, n) mapping he numbe o anks n o he 16 Ali eza Nase i e al. Rank n-1 Rank n-2 Rank 2 Rank 1 Rank 0 Rank m-1 Rank 1 Rank 0 adap e adap e adap e adap e adap e adap e adap e adap e p eCICE Communica o Poin - o-Poin Communica ion In e -Code In a-Sol e Poin - o-Poin Communica ion In a-Sol e Poin - o-Poin Communica ion Fluid Sol e Solid Sol e MPI Comm Wo ld 1 MPI Comm Wo ld 2 SIMD wi hin each ank . Fig. 2 Pa alleliza ion model o he coupled amewo k and he di e en le els o communica ion. un- ime n(in his wo k, m=5 simula ions, each o only 3 ime s eps). In o de o ind a unc ion (n) o each sol e , which p edic s he un- ime o any n, we use he Pe o mance Model No mal Fo m (PMNF) [56], de ined in (Eq. (26)), as a basis o ou p edic ion model: (n) = q X k=1 ckniklogjk 2(n),(26) whe e nis he numbe o anks used by he sol e , qis he numbe o e ms used o he empi ical un- ime app oxima ion, ckis he weigh o each e m in he app oxima ion unc ion, and ikand jka e empi ical coe icien s. Ti le Supp essed Due o Excessi e Leng h 17 Calo oiu e al. [56] sugges , ha he sea ch space gi en by q= 2, ik∈ {±0 4,±1 4,...,±12 4}and jk∈ {0,±1,±2}is sui able o many applica ions. To ind he op imal model, we simply check all he combina ions wi hin he sea ch space, calcula e a c oss- alida ion-based loss o each combina ion ( he accumula i e e o on he alida ion da a se ) and pick he combina ion wi h he smalles loss. By applying a PMNF eg ession, we a e able o gene a e pe o mance models (n) o each sol e in ol ed in he simula ion. As men ioned be o e, ou goal is o ind an op imal assignmen o co es o each sol e o a limi ed o al numbe o a ailable co es P, such ha he o e all un- ime F(n , ns) is minimized (n and nsa e numbe o anks used by he luid and he solid sol e espec i ely). This can be exp essed by he ollowing op imiza ion p oblem: minimize n ,ns F(n , ns) wi h F(n , ns) = max( s(ns), (n )) subjec o n +ns≤P. (27) I he unc ions sand a e app oxima ed by he PMNF eg ession, his op imiza ion p oblem is a nonlinea , possibly non-con ex in ege op i- miza ion. We assume ha sand a e bo h mono onically dec easing, i.e., assigning mo e co es o a sol e ne e inc eases he un- ime. Wi h his, we can simpli y he cons ain o P=n +ns.(28) The op imiza ion p oblem can hen be sol ed by checking all possible alues o nsand n in o de o choose he pai ha minimizes he o al un- ime. This pai o ank coun s is hen used o di ide he a ailable p ocesso s be ween he luid and he s uc u al sol e o he FSI simula ion. Fo mo e de ails, please e e o [55]. 5 Nume ical Tes s Nume ical es s a e p o ided in his sec ion o demons a e he pa allel scal- abili y o he coupled amewo k in sol ing p ac ically ele an FSI p ob- lems. The es cases a e in he con ex o hemodynamics. The i s es case is a benchma k p oblem ep esen ing he p opaga ion o p essu e wa es in- side a 3D de o mable ube. The second es case is he blood low inside a pa ien -speci ic ao a conside ing he elas ic de o ma ion o he ao a wall. The scalabili y es s a e ca ied ou on he Supe MUC supe compu e a he Leibniz Supe compu ing Cen e o he Ba a ian Academy o Sciences and Humani ies in Ga ching [57]. Supe MUC consis s o 2.6GHz In el Xeon E5-2697- 3 (Haswell) p ocesso s. Each compu ing node con ains wo p o- cesso s wi h 14 co es pe p ocesso (28 co es pe node) and 64GB o RAM. The nodes a e connec ed ia Mellanox In iniband FDR14 in e connec . The GNU GCC compile was used o compile bo h he Te moFluids sol e and 18 Ali eza Nase i e al. he p eCICE lib a y. In addi ion, an In el MPI implemen a ion compa ible wi h he GCC compile was used o in a-sol e pa alleliza ion. 5.1 Tes case 1: 3D Flow Inside a De o mable Tube This benchma k p oblem was p oposed by [58] and s udied, among o he s, by [59,44,60,61]. The p oblem is a 3D incomp essible low inside a s aigh ube wi h a de o mable wall, mo i a ed by he ype o p oblems encoun e ed in hemodynamics. The ube has a leng h o l= 0.05m, an inne adius o R0= 0.005m, and a wall hickness o h= 0.001m. The luid densi y and iscosi y a e ρ = 1000kg/m3and µ = 0.003Pa ·s, espec i ely. The s uc u al densi y is ρs= 1200kg/m3, he Young modulus E= 3×105N/m2, and he Poisson a io ν= 0.3. The ube is clamped a bo h ends and he luid is ini ially a es . An o e p essu e o 1333.2Pa is applied a he ube inle du ing a pe iod o 0.003s and a cons an p essu e o 0Pa a e wa ds. The p essu e a he ou le is 0Pa du ing he whole simula ion. A Neumann bounda y condi ion is used o he luid eloci y a bo h he inle and he ou le bounda ies. The ou e su ace o he ube wall is ea ed as a ac ion- ee bounda y. An uns uc u ed e ahed al mesh is used o he luid domain. The solid domain mesh is cons uc ed by ex uding he luid mesh on he ou e bounda y (s uc u ed mesh). Th ee di e en mesh esolu ions a e used o he nume ical es s. The esolu ion o he mesh o he luid and he solid domain is p o ided in Table 1. Two meshes (M1 and M2 in Table 1) a e used o sol e he p oblem om = 0 un il = 0.02s wi h cons an ime s ep sizes o ∆ = 10−4and ∆ = 5 ×10−5s, espec i ely (which means doubling he esolu ion in each spa ial di ec ion and ime). Figu e 3 p esen s he adial displacemen a he mid-leng h o he ube du ing he simula ion ime. As seen in he igu e, he esul s ob ained by mesh M1 and M2 (and hei co esponding ime s ep) a e ai ly close. Small di e ences a e isible nea he peaks and he oughs. Resul s wi h he coa se mesh and he la ge ime s ep appea o be mo e damped, which is expec ed due o a la ge nume ical dissipa ion. Figu e 3 also con ains nume ical esul s om Eken e al. [60] and Lozo skiy e al. [61]. Al hough he ou se s o esul s ollow he same end, he esul s om [60,61] appea o be mo e damped, compa ed o he p esen solu ion. This could be due o he ex a nume ical dissipa ion in [60,61], as bo h wo ks use a i s -o de Eule disc e iza ion in ime wi h a ime s ep ∆ = 10−4s (compa ed o a second-o de ime disc e iza ion in he cu en wo k). Figu e 4 shows he eloci y ec o s inside he de o med ube a wo ins an s = 0.005 and = 0.01s. The colo con ou in he solid domain is he on Mises equi alen s ess. The de o ma ion o he wall is magni ied by a ac o o en o be be e isible. The igu e shows he p opaga ion o he p essu e wa e wi h a ini e eloci y inside he ube. Fo scalabili y es s, we ha e used a much ine g id (mesh M3 in Ta- ble 1). A comple e simula ion is no ca ied ou o hese es s, only he i s Ti le Supp essed Due o Excessi e Leng h 19 Table 1 Compu a ional g ids used o he de o mable ube es case ( es case 1). Mesh name No. o cells Fluid S uc u e M1 15K 10K M2 120K 80K M3 9M 6M -0.1 -0.05 0 0.05 0.1 0.15 0 0.005 0.01 0.015 0.02 mid-leng h adial displacemen (mm) ime (s) mesh M2, ∆ =5e-5 mesh M1, ∆ =1e-4 Eken e al. (2016) Lozo skiy e al. (2019) Fig. 3 Radial displacemen a he mid-leng h o he ube ( es case 1) o wo di e en mesh esolu ions and ime s ep sizes, compa ed o nume ical esul s om [60,61]. en ime s eps a e sol ed. The solu ion esul s a he en i e ield ( o bo h luid and solid domains) a e w i en in o h5 ou pu iles a e e y ime s ep. Figu e 5 shows he a e age un– ime pe ime s ep o di e en numbe s o CPU co es using mesh M3. This un– ime includes he ime o w i ing ou pu da a iles, bu does no include he ini ializa ion ime o he sol e s. The ideal (linea ) educ ion o he un- ime by inc easing he co e coun is also shown o compa ison. The numbe o co es in he ho izon al axis o he g aph indica es he o al sum o co es used o he luid and he solid sol e . The dis ibu ion o he a ailable co es be ween he sol e s is based on he load balancing model desc ibed in Sec ion 4.3, and i is p o ided in Table 2. As seen in he igu e, a e y good educ ion in he compu a ional ime is achie ed by inc easing he numbe o co es. Resul s show a e y good scalabili y o up o 1400 co es, wi h an almos ideal speed-up o up o 1120 co es. Figu e 6 shows he pa allel e iciency a each poin o he g aph. The pa allel e iciency o he s ong scalabili y on mco es is e alua ed as 20 Ali eza Nase i e al. Fig. 4 P opaga ion o he p essu e wa e inside he de o mable ube ( es case 1). Shown in he igu e a e luid eloci y ec o s inside he de o med domain and he on Mises equi alen s ess a he wall. Top: = 0.005s; Bo om: = 0.01s. De o ma ions a e magni ied by a ac o o en. e iciency = l×l m×m(29) whe e mis he un- ime on mco es, and lis he un- ime on he smalles numbe o co es, indica ed by l. I possible, he pa allel e iciency is measu ed agains he sequen ial un- ime (l= 1). Howe e , due o he limi a ion o memo y on a single co e, la ge p oblems canno be sol ed sequen ially. The e o e, he smalles numbe o co es which can be used o sol e he p oblem is used as he basis o e alua e he e iciency (l= 280 o his es case). As seen in he igu e, he pa allel e iciency is 96% o 1120 co es which indica es an almos ideal pa allel scalabili y. Fo 1400 co es he e iciency is s ill high, a 82%. Fo highe co e numbe s he pa allel e iciency deg ades. This limi appea s o be mos ly de e mined by he size o he mesh used o he es s. The size o he compu a ional g id ( o al o 15M cells o mesh M3) is ela i ely small o be di ided among o e a housand p ocesses. Di iding mesh M3 among 1400 co es means each p ocess owns oughly 10K Ti le Supp essed Due o Excessi e Leng h 21 cells, which is oo ew calcula ions and co esponds o a e y low a i hme ic in ensi y (compa ed o communica ion). By epea ing he scalabili y es s using ine g ids, a g ea e scalabili y can be achie ed on highe numbe s o co es. La ge compu a ional g ids a e used o he nex es case. Mo eo e , he ou pu da a w i ing could limi he scalabili y as i essen ially con ains sequen ial s eps. We ha e included he ou pu esul w i ing ime in he scalabili y es s o he cu en es case, because i is a necessa y pa o any p ac ical simula ion. Table 2 Dis ibu ion o he CPU co es be ween he luid and he solid sol e based on he in e -code load balancing model, o es case 1 (de o mable ube) and mesh M3. To al No. o co es 280 420 560 700 840 980 1120 1260 1400 Fluid sol e 153 230 301 363 417 461 496 524 545 Solid sol e 127 190 259 337 423 519 624 736 855 280 420 560 700 840 980 1120 1260 1400 Numbe o co es 102 4 × 101 6 × 101 2 × 102 Run ime pe ime s ep [s] Tube mesh M3 Ideal scalabili y Fig. 5 S ong scalabili y es esul s o es case 1 (de o mable ube) and mesh M3: a e age un– ime pe ime s ep o di e en numbe s o co es. A e e y ime s ep, an a e age o oughly 17 coupling i e a ions we e e- qui ed o achie e con e gence on he coupled p oblem (using F SI = 10−5). This numbe emained ai ly cons an by inc easing he numbe o p o- cesses, as seen in Table 3. This shows ha he o e all me hodology o he coupled FSI p oblem is ma hema ically scalable. Table 3 also con ains he da a o he ini ializa ion ime o he coupled amewo k. The epo ed ime 22 Ali eza Nase i e al. 280 420 560 700 840 980 1120 1260 1400 Numbe o co es 85 90 95 100 105 E iciency [%] Ideal E iciency E iciency Fig. 6 S ong scalabili y es esul s o es case 1 (de o mable ube) and mesh M3: pa allel e iciency a di e en co e coun s. includes he ini ializa ion imes o he single–physics sol e s, as well as he ime o es ablishing he in e –code communica ion channels and ini ializing he coupling lib a y modules. I does no include he p e-p ocessing s eps such as mesh gene a ion o pa i ioning using he METIS lib a y. The da a in Table 3 show ha he ini ializa ion ime emains ai ly cons an and only sligh ly inc eases when inc easing he numbe o p ocesses. Table 3 S ong scalabili y o es case 1 (de o mable ube) and mesh M3. Resul s in he able a e he a e age numbe o coupling i e a ions pe ime s ep, and he o al ini ializa ion ime o he coupled amewo k. No. o co es 280 420 560 700 840 980 1120 1260 1400 coupling i e a ions (a e age) 17.4 17.6 16.8 17.4 16.8 16.8 17.2 17.4 17.2 ini ializa ion ime [s] 329.6 335.8 339.2 345.6 351.7 353.9 359.2 365.4 370.6 5.2 Tes case 2: Pa ien –Speci ic Ao a The second es case is a simula ion o blood low inside a pa ien -speci ic ao a wi h a mild ho acic ao ic coa c a ion. The 3D geome y o he ao a is ob ained by con as agen magne ic esonance angiog aphy, p o ided by he 2nd CFD challenge o he STACOM 2013 con e ence [62]. Figu e 7 shows he p o ided geome y and he loca ion o he bounda ies. The hickness Ti le Supp essed Due o Excessi e Leng h 23 o he ao a wall and i s mechanical p ope ies we e no p o ided in he challenge da a. The e o e, we assume alues in he ypical physiological ange. A uni o m hickness o h= 2mm is assumed o he wall along wi h densi y ρs= 1200kg/m3, Young modulus E= 3 ×105N/m2, and Poisson a io ν= 0.3. The densi y and he iscosi y o he blood a e conside ed o be ρ = 1000kg/m3and µ = 0.004Pa ·s. These alues o he wall hickness and he p ope ies o he wall and blood a e simila o he alues used in [63]. Fig. 7 Tes case 2, 3D geome y o a pa ien -speci ic ao a p o ided in [62], and he loca ion o he bounda ies. Fo he inle bounda y o he ao a, a Di ichle bounda y condi ion is used o he eloci y, using measu ed physiological low a e da a p o ided in [62] o he es condi ion. A Neumann bounda y condi ion is used o he luid p essu e a he inle . Fo he ou le bounda ies, explici RCR Windkessel bounda y condi ions [64] a e used o model he e ec o he es o he ascula ne wo k. The Windkessel pa ame e s a e chosen as hose epo ed in [65]. Fo he solid, a ze o–displacemen (clamped) bounda y condi ion is se a he inle and a he ou le s, while a ac ion- ee bounda y condi ion is used on he ou e su ace o he wall. The objec i e o he cu en wo k is no a deep s udy o he biophysical phenomena ea u ed in his es case, bu a he o show he capabili y o he de eloped amewo k o sol e such a complex p oblem on a massi ely pa allel con igu a ion. Ne e heless, he ob ained luid eloci y and p es- 24 Ali eza Nase i e al. su e ields, as well as he s uc u al displacemen s we e seen o be in he easonable physiological ange. Figu e 8 shows he solu ion a wo ins an s, = 0.05s and = 0.1s. The igu e con ains he eloci y ec o plo in- side he de o med ao ic wall. The colo con ou s in he s uc u al domain co espond o he on Mises equi alen s ess. Fig. 8 Tes case 2, luid eloci y ec o s inside he de o med ao ic wall and he on Mises equi alen s ess a he wall. Le : a = 0.05s; Righ : a = 0.1s. The s ong scalabili y o he de eloped amewo k is e alua ed using he compu a ional g ids M1 and M2 (Table 4). Bo h luid and solid g ids a e uns uc u ed e ahed al meshes. The i s en ime s eps a e sol ed o he scalabili y es s. Unlike he p e ious es case, he ou pu esul w i ing s ep is disabled, hus he un- imes only include he solu ion ime o he coupled p oblem. Figu e 9 shows he a e age un– ime pe ime s ep o di e en numbe s o co es o mesh M1, wi h and wi hou in e -code load balancing. Simila o he p e ious es case, he numbe o co es indica es he o al numbe o p ocesses o he luid and he solid sol e oge he . Fo he case wi hou load balancing, he a ailable co es a e di ided be ween he sol e s p opo ional o hei espec i e mesh size. Fo he load balancing case, he di ision o he co es is based on he load balancing model in Sec ion 4.3, and i is p o ided in Table 5. As seen in Figu e 9, a e y good educ ion in compu a ional ime is achie ed by inc easing he numbe o co es, up Ti le Supp essed Due o Excessi e Leng h 31 load balancing is conside ed, i.e., inside each sol e and ac oss he codes. The a ailable p ocesso s a e di ided be ween he wo sol e s based on a load balancing model ha minimizes he o al idle ime o he p ocesses. Bo h he luid and he s uc u al sol e disc e ize and sol e he co e- sponding go e ning equa ions on uns uc u ed 3D meshes. A semi–implici FSI coupling me hod is used, in which he luid p essu e e m is seg ega ed and s ongly coupled o he s uc u e, while he emaining e ms a e only loosely coupled. An e icien mul i– ec o quasi–New on me hod is used o sol e he coupled in e ace p oblem. Two nume ical es cases in he con ex o hemodynamics a e conside ed and he s ong scalabili y o he coupled amewo k is e alua ed. The i s es case is a benchma k FSI p oblem, sol ing an incomp essible low inside a de o mable ube. The simula ion esul s o his es case a e compa ed o o he nume ical esul s om he li e a u e in o de o e i y he accu- acy o he solu ion me hods and he o e all amewo k. Scalabili y es s a e ca ied ou using a compu a ional mesh consis ing o 9 million uns uc- u ed e ahed al cells o he luid and 6 million s uc u ed hexahed al cells o he solid. Tes esul s showed a pa allel e iciency o 82% on 1400 CPU co es. This limi appea s o be mos ly de e mined by he size o mesh, as o highe numbe o co es he a i hme ic in ensi y o he p oblem becomes oo small. The numbe o coupling i e a ions a each ime s ep emained nea ly cons an o di e en numbe o co es, which shows he de eloped me hods a e ma hema ically scalable. Mo eo e , he ini ializa ion ime o he ame- wo k emained ai ly cons an and inc eased only sligh ly o di e en co e coun s. The second es case sol es he blood low inside a pa ien –speci ic ao a. Two di e en meshes we e used o he scalabili y es s, consis ing o un- s uc u ed e ahed al cells o bo h he luid and he solid domain. Tes esul s using a mesh wi h 20 million cells o he luid and 9 million cells o he solid, showed a pa allel e iciency o 79% on 3920 CPU co es. Simila o he p e ious es case, his limi appea s o be de e mined by he mesh size and small a i hme ic in ensi y o highe co e numbe s. Mo eo e , we ha e e alua ed he e ec o ou in e -code load balancing model on he pe o - mance o he coupled sol e . Applying he load balancing me hod educed he un- ime by an a e age o 15%, almos uni o mly ac oss he di e en o al co e coun s. The scalabili y es s we e epea ed using a mesh wi h 95 million cells o he luid and 41 million cells o he solid. Resul s demon- s a ed a pa allel e iciency o 83% on 10,080 CPU co es. The numbe o FSI coupling i e a ions a each ime s ep emained cons an o di e en co e coun s, demons a ing he ma hema ical scalabili y o he me hods. The ini ializa ion ime o he amewo k inc eased only mode a ely o up o 6720 co es, howe e , i g ew apidly o highe numbe o co es. Fu he - mo e, i was shown ha he ini ializa ion ime o he amewo k inc eases linea ly by inc easing he size o he mesh. 32 Ali eza Nase i e al. 7 Acknowledgemen This wo k was inancially suppo ed by –Minis e io de Econom´ıa y Compe i i idad, Sec e a ´ıa de Es ado de In- es igaci´on, Desa ollo e Inno aci´on, Spain (ENE2017-88697-R), –p io i y p og am 1648 - So wa e o Exascale Compu ing 214 (ExaFSA - Exascale Simula ion o Fluid-S uc u e-Acous ics In e ac ions) o he Ge man Resea ch Founda ion, –and a FI PhD schola ship by he Ag`encia de Ges i´o d’Aju s Uni e si a is i de Rece ca (AGAUR) o Gene ali a de Ca alunya (Spain). The pe o mance measu emen s we e ca ied ou on he Supe MUC supe - compu e a Leibniz Rechenzen um (LRZ) de Baye ischen Akademie de Wissenscha en. The au ho s wish o hank LRZ o he compu ing ime and he echnical suppo . Re e ences 1. Y. Bazile s, V. M. Calo, Y. Zhang, T. J. R. Hughes, Isogeome ic luid- s uc u e in e ac ion analysis wi h applica ions o a e ial blood low, Compu- a ional Mechanics 38 (4-5) (2006) 310–322. doi:10.1007/s00466-006-0084-3. 2. K. Takizawa, Y. Bazile s, T. E. Tezduya , Space– ime and ALE–VMS ech- niques o pa ien -speci ic ca dio ascula luid–s uc u e in e ac ion modeling, A chi es o Compu a ional Me hods in Enginee ing 19 (2) (2012) 171–225. doi:10.1007/s11831-012-9071-3. 3. J. Deg oo e, Pa i ioned simula ion o luid-s uc u e in e ac ion, A chi es o Compu a ional Me hods in Enginee ing 20 (2013) 185–238. doi:10.1007/s11831-013-9085-5. 4. G. Hou, J. Wang, A. Lay on, Nume ical me hods o luid-s uc u e in e - ac ion - A e iew, Communica ions in Compu a ional Physics 12 (2) (2012) 337–377. doi:10.4208/cicp.291210.290411s. 5. F. Kong, X.-C. Cai, A scalable nonlinea luid–s uc u e in e ac ion sol e based on a schwa z p econdi ione wi h isogeome ic uns uc u ed coa se spaces in 3d, Jou nal o Compu a ional Physics 340 (2017) 498–518. doi:10.1016/j.jcp.2017.03.043. 6. S. Depa is, D. Fo i, G. G andpe in, A. Qua e oni, FaCSI: A block pa allel p econdi ione o luid–s uc u e in e ac ion in hemodynamics, Jou nal o Compu a ional Physics 327 (2016) 700–718. doi:10.1016/j.jcp.2016.10.005. 7. F. Kong, V. Khey e s, E. Finol, X.-C. Cai, Simula ion o uns eady blood lows in a pa ien -speci ic complian pulmona y a e y wi h a highly pa al- lel monoli hically coupled luid-s uc u e in e ac ion algo i hm, In e na ional jou nal o nume ical me hods in biomedical enginee ing 35 (7) (2019) e3208. doi:10.1002/cnm.3208. 8. S. Ka aoka, S. Minami, H. Kawai, T. Yamada, S. Yoshimu a, A pa allel i e a i e pa i ioned coupling analysis sys em o la ge-scale acous ic luid– s uc u e in e ac ions, Compu a ional Mechanics 53 (6) (2014) 1299–1310. doi:10.1007/s00466-013-0973-1. Ti le Supp essed Due o Excessi e Leng h 33 9. J. Cajas, G. Houzeaux, M. V´azquez, M. Ga cia, E. Casoni, H. Calme , A. A - igues, R. Bo ell, O. Lehmkuhl, D. Pas ana, e al., Fluid-s uc u e in e ac- ion based on hpc mul icode coupling, SIAM Jou nal on Scien i ic Compu ing 40 (6) (2018) C677–C703. doi:10.1137/17M1138868. 10. S. Hewi , L. Ma ge s, A. Re ell, P. Pankaj, F. Le e o- Flo encio, OpenFPCI: A pa allel luid–s uc u e in e ac ion ame- wo k, Compu e Physics Communica ions 244 (2019) 469–482. doi:h ps://doi.o g/10.1016/j.cpc.2019.05.016. 11. J. La son, R. Jacob, E. Ong, The model coupling oolki : a new o an90 oolki o building mul iphysics pa allel coupled models, The In e na ional Jou nal o High Pe o mance Compu ing Applica ions 19 (3) (2005) 277–292. doi:10.1177/1094342005056115. 12. H.-J. Bunga z, F. Lindne , B. Ga zhamme , M. Mehl, K. Scheu ele, A. Shukae , B. Ueke mann, p eCICE–a ully pa allel lib a y o mul i- physics su ace coupling, Compu e s & Fluids 141 (2016) 250–258. doi:10.1016/j.comp luid.2016.04.003. 13. D. Thomas, M. L. Ce quaglia, R. Boman, T. D. Economon, J. J. Alonso, G. Dimi iadis, V. E. Te apon, CUPyDO - An in eg a ed Py hon en i on- men o coupled luid-s uc u e simula ions, Ad ances in Enginee ing So - wa e 128 (2019) 69–85. 14. H.-J. Bunga z, F. Lindne , M. Mehl, K. Scheu ele, A. Shukae , B. Ueke - mann, Pa i ioned luid–s uc u e–acous ics in e ac ion on dis ibu ed da a: Coupling ia p ecice, in: H.-J. Bunga z, P. Neumann, W. E. Nagel (Eds.), So wa e o Exascale Compu ing - SPPEXA 2013-2015, Sp inge In e na- ional Publishing, Cham, 2016, pp. 239–266. doi:10.1007/978-3-319-40528-5- 11. 15. M. L. Ce quaglia, D. Thomas, R. Boman, V. Te apon, J.-P. Pon ho , A ully pa i ioned lag angian amewo k o si p oblems cha ac e ized by ee su aces, la ge solid de o ma ions and displacemen s, and s ong added-mass e ec s, Compu e Me hods in Applied Mechanics and Enginee ing 348 (2019) 409–442. doi:10.1016/j.cma.2019.01.021. 16. P. Causin, J. F. Ge beau, F. Nobile, Added-mass e ec in he de- sign o pa i ioned algo i hms o luid-s uc u e p oblems, Compu e Me hods in Applied Mechanics and Enginee ing 194 (2005) 4506–4527. doi:10.1016/j.cma.2004.12.005. 17. C. F¨o s e , W. A. Wall, E. Ramm, A i icial added mass ins abili ies in se- quen ial s agge ed coupling o nonlinea s uc u es and incomp essible iscous lows, Compu e Me hods in Applied Mechanics and Enginee ing 196 (2007) 1278–1293. doi:10.1016/j.cma.2006.09.002. 18. M. A. Fe n´andez, J.-F. Ge beau, C. G andmon , A p ojec ion semi-implici scheme o he coupling o an elas ic s uc u e wi h an incomp essible luid, In e na ional Jou nal o Nume ical Me hods in Enginee ing 69 (4) (2007) 794–821. doi:10.1002/nme.1792. 19. A. Nase i, O. Lehmkuhl, I. Gonzalez, E. Ba ons, C. D. P´e ez-Sega a, A. Oli a, A semi-implici coupling echnique o luid–s uc u e in e ac ion p oblems wi h s ong added-mass e ec , Jou nal o Fluids and S uc u es 80 (2018) 94–112. doi:10.1016/j.j luids uc s.2018.03.012. 20. A. Nase i, I. Gonzalez, A. Amani, C. D. P´e ez-Sega a, A. Oli a, A second–o de ime accu a e semi–implici me hod o luid–s uc u e in- e ac ion p oblems, Jou nal o Fluids and S uc u es 86 (2019) 135–155. doi:10.1016/j.j luids uc s.2019.02.007. 34 Ali eza Nase i e al. 21. Te mo Fluids S.L., Te mo luids, h p://www. e mo luids.com (2020). 22. I. Rod iguez, R. Bo ell, O. Lehmkuhl, C. D. Pe ez Sega a, A. Oli a, Di ec nume ical simula ion o he low o e a sphe e a Re = 3700, Jou nal o Fluid Mechanics 679 (2011) 263–287. doi:10.1017/j m.2011.136. 23. I. Rod ´ıguez, O. Lehmkuhl, J. Chi a, R. Bo ell, A. Oli a, On he low pas a ci cula cylinde om c i ical o supe -c i ical eynolds numbe s: Wake opol- ogy and o ex shedding, In e na ional Jou nal o Hea and Fluid Flow 55 (2015) 91 – 103. doi:10.1016/j.ijhea luid low.2015.05.009. 24. N. Balc´aza , L. Jo e, O. Lehmkuhl, J. Cas o, J. Rigola, A ini e- olume/le el-se me hod o simula ing wo-phase lows on uns uc- u ed g ids, In e na ional jou nal o mul iphase low 64 (2014) 55–72. doi:10.1016/j.ijmul iphase low.2014.04.008. 25. E. Gu i´e ez, F. Fa e, N. Balcaza , A. Amani, J. Rigola, Nume ical app oach o s udy bubbles and d ops e ol ing h ough complex geome ies by using a le el se –mo ing mesh–imme sed bounda y me hod, Chemical Enginee ing Jou nal 349 (2018) 662–682. doi:10.1016/j.cej.2018.05.110. 26. P. Galione, O. Lehmkuhl, J. Rigola, A. Oli a, Fixed-g id nume ical model- ing o mel ing and solidi ica ion using a iable he mo-physical p ope ies– applica ion o he mel ing o n-oc adecane inside a sphe ical capsule, In e na ional Jou nal o Hea and Mass T ans e 86 (2015) 721–743. doi:10.1016/j.ijhea mass ans e .2015.03.033. 27. E. Ba ons, C. Olie , E. Gu ie ez, A. Nase i, C. D. P´e ez-Sega a, A ini e olume me hod o sol e he os g ow h using dynamic meshes, In e na ional Jou nal o Hea and Mass T ans e 124 (2018) 615–628. doi:10.1016/j.ijhea mass ans e .2018.03.104. 28. G. Colome , R. Bo ell, F. X. T ias, I. Rod ´ıguez, Pa allel algo i hms o Sn anspo sweeps on uns uc u ed meshes, Jou nal o Compu a ional Physics 232 (1) (2013) 118–135. doi:10.1016/j.jcp.2012.07.009. 29. L. Jo e, R. Bo ell, O. Lehmkuhl, A. Oli a, Pa allel load balancing s a egy o olume-o - luid me hods on 3-d uns uc u ed meshes, Jou nal o Compu- a ional Physics 282 (2015) 269–288. doi:10.1016/j.jcp.2014.11.009. 30. R. Bo ell, J. Chi a, O. Lehmkuhl, G. Oya zun, I. Rod ´ıguez, A. Oli a, Op imising he Te mo luids CFD code o pe ascale simula ions, In e na- ional Jou nal o Compu a ional Fluid Dynamics 30 (6) (2016) 425–430. doi:10.1080/10618562.2016.1221503. 31. R. Ve s appen, A. Veldman, Symme y-p ese ing disc e iza ion o u - bulen low, Jou nal o Compu a ional Physics 187 (1) (2003) 343–368. doi:10.1016/S0021-9991(03)00126-8. 32. F. X. T ias, O. Lehmkuhl, A. Oli a, C. D. P´e ez-Sega a, R. Ve s appen, Symme y-p ese ing disc e iza ion o Na ie -S okes equa ions on colloca ed uns uc u ed g ids, Jou nal o Compu a ional Physics 258 (2014) 246–267. doi:10.1016/j.jcp.2013.10.031. 33. L. Jo e, O. Lehmkuhl, J. Ven osa, F. X. T ias, A. Oli a, Conse a ion p op- e ies o uns uc u ed ini e- olume mesh schemes o he na ie -s okes equa- ions, Nume ical Hea T ans e , Pa B: Fundamen als 65 (1) (2014) 53–79. doi:10.1080/10407790.2013.836335. 34. P. Thomas, C. Lomba d, Geome ic conse a ion law and i s applica ion o low compu a ions on mo ing g ids, AIAA jou nal 17 (10) (1979) 1030–1037. doi:10.2514/3.61273. 35. M. Lesoinne, C. Fa ha , Geome ic conse a ion laws o low p oblems wi h mo ing bounda ies and de o mable meshes, and hei impac on ae oelas- Ti le Supp essed Due o Excessi e Leng h 35 ic compu a ions, Compu e me hods in applied mechanics and enginee ing 134 (1-2) (1996) 71–90. doi:10.1016/0045-7825(96)01028-6. 36. O. Es uch, O. Lehmkuhl, R. Bo ell, C. D. P. Sega a, A. Oli a, A pa allel adial basis unc ion in e pola ion me hod o uns uc u ed dynamic meshes, Compu e s and Fluids 80 (2013) 44–54. doi:10.1016/j.comp luid.2012.06.015. 37. P. Ca di , I. Demi dˇzi´c, Thi y yea s o he ini e olume me hod o solid mechanics, a Xi p ep in a Xi :1810.02105 (2018). 38. H. Jasak, E o analysis and es ima ion o he ini e olume me hod wi h applica ions o luid lows, Ph.D. hesis, Impe ial College London (Uni e si y o London) (1996). 39. A. J. Macleod, Accele a ion o ec o sequences by mul i-dimensional ∆2 me hods, Communica ions in Applied Nume ical Me hods 2 (4) (1986) 385– 392. 40. ˇ Z. Tuko i´c, A. I anko i´c, A. Ka aˇc, Fini e- olume s ess analysis in mul i- ma e ial linea elas ic body, In e na ional Jou nal o Nume ical Me hods in Enginee ing 93 (4) (2013) 400–419. 41. P. Ca di , Tuko i´c, H. Jasak, A. I anko i´c, A block-coupled Fini e Volume me hodology o linea elas ici y and uns uc u ed meshes, Compu e s and S uc u es 175 (2016) 100–122. doi:10.1016/j.comps uc.2016.07.004. 42. P. Chand asheka , A. Ga g, Ve ex-cen oid ini e olume scheme on e ahe- d al g ids o conse a ion laws, Compu e s & Ma hema ics wi h Applica ions 65 (1) (2013) 58–74. 43. I. Gonz´alez, A. Nase i, J. Chi a, J. Rigola, C. D. P´e ez-Sega a, An enhanced ini e olume based sol e o he moelas ic ma e ials in luid-s uc u e cou- pled p oblems, in: 6 h Eu opean Con e ence on Compu a ional Mechanics (ECCM-ECFD 2018), ECCOMAS, Glasgow, UK, June 2018. 44. U. K¨u le , W. A. Wall, Fixed-poin luid-s uc u e in e ac ion sol e s wi h dynamic elaxa ion, Compu a ional Mechanics 43 (2008) 61–72. doi:10.1007/s00466-008-0255-5. 45. J. F. Ge beau, M. Vid ascu, A quasi-New on algo i hm based on a educed model o luid-s uc u e in e ac ion p oblems in blood lows, ESAIM: Ma hema ical Modelling and Nume ical Analysis 37 (2003) 631–647. doi:10.1051/m2an:2003049. 46. C. Michle , E. H. V. B ummelen, R. D. Bo s , An in e ace New on-K ylo sol e o luid-s uc u e in e ac ion, In e na ional Jou nal o Nume ical Me hods in Fluids 47 (10-11) (2005) 1189–1195. doi:10.1002/ ld.850. 47. K. Scheu ele, M. Mehl, Robus mul isecan quasi-New on a ian s o pa allel luid-s uc u e simula ions–and o he mul iphysics applica ions, SIAM Jou nal on Scien i ic Compu ing 39 (5) (2017) S404–S433. doi:10.1137/16M1082020. 48. H.-J. Bunga z, F. Lindne , M. Mehl, B. Ueke mann, A plug-and-play cou- pling app oach o pa allel mul i- ield simula ions, Compu a ional Mechanics 55 (6) (2015) 1119–1129. doi:10.1007/s00466-014-1113-2. 49. M. Mehl, B. Ueke mann, H. Bijl, D. Blom, B. Ga zhamme , A. Van Zuijlen, Pa allel coupling nume ics o pa i ioned luid–s uc u e in e ac ion simula- ions, Compu e s & Ma hema ics wi h Applica ions 71 (4) (2016) 869–891. doi:10.1016/j.camwa.2015.12.025. 50. G. Oya zun, R. Bo ell, A. Go obe s, A. Oli a, MPI-CUDA spa se ma ix– ec o mul iplica ion o he conjuga e g adien me hod wi h an app ox- ima e in e se p econdi ione , Compu e s & Fluids 92 (2014) 244–252. doi:10.1016/j.comp luid.2013.10.035. 36 Ali eza Nase i e al. 51. G. Oya zun, R. Bo ell, A. Go obe s, A. Oli a, Po able implemen a ion model o CFD simula ions. applica ion o hyb id CPU/GPU supe compu - e s, In e na ional Jou nal o Compu a ional Fluid Dynamics 31 (9) (2017) 396–411. doi:10.1080/10618562.2017.1390084. 52. G. Oya zun, R. Bo ell, A. Go obe s, F. Man o ani, A. Oli a, E i- cien CFD code implemen a ion o he ARM-based Mon -Blanc a - chi ec u e, Fu u e gene a ion compu e sys ems 79 (2018) 786–796. doi:10.1016/j. u u e.2017.09.029. 53. G. Ka ypis, V. Kuma , A as and high quali y mul ile el scheme o pa i- ioning i egula g aphs, SIAM Jou nal on Scien i ic Compu ing 20 (1) (1998) 359–392. doi:10.1137/S1064827595287997. 54. The HDF G oup, Hie a chical Da a Fo ma , e sion 5, h p://www.hd g oup.o g/HDF5/ (1997-2019). 55. A. To oun e oush, N. Eb ahimi Pou , J. Sch ode , S. Rolle , M. Mehl, A new load balancing app oach o coupled mul i-physics simula ions, in: In P oceedings o IEEE In e na ional Pa allel and Dis ibu ed P ocess- ing Symposium Wo kshops (IPDPSW), Rio de Janei o, B azil, May 2019. doi:10.1109/IPDPSW.2019.00115. 56. A. Calo oiu, D. Beckinsale, C. W. Ea l, T. Hoe le , I. Ka lin, M. Schulz, F. Wol , Fas mul i-pa ame e pe o mance modeling, in: 2016 IEEE In e - na ional Con e ence on Clus e Compu ing (CLUSTER), IEEE, 2016, pp. 172–181. 57. Leibniz Supe compu ing Cen e o he Ba a ian Academy o Sciences and Humani ies , h ps://www.l z.de. 58. L. Fo maggia, J. F. Ge beau, F. Nobile, A. Qua e oni, On he coupling o 3D and 1D Na ie -S okes equa ions o low p oblems in complian essels, Compu e Me hods in Applied Mechanics and Enginee ing 191 (6-7) (2001) 561–582. doi:10.1016/S0045-7825(01)00302-4. 59. M. A. Fe n´andez, M. Moubachi , A New on me hod using exac Jacobians o sol ing luid–s uc u e coupling, Compu e s and S uc u es 83 (2005) 127– 142. doi:10.1016/j.comps uc.2004.04.021. 60. A. Eken, M. Sahin, A pa allel monoli hic algo i hm o he nume ical simula- ion o la ge-scale luid s uc u e in e ac ion p oblems, In e na ional Jou nal o Nume ical Me hods in Fluids 80 (12) (2016) 687–714. doi:10.1002/ ld.4169. 61. A. Lozo skiy, M. A. Olshanskii, Y. V. Vassile ski, Analysis and assessmen o a monoli hic FSI ini e elemen me hod, Compu e s & Fluids 179 (2019) 277–288. doi:10.1016/j.comp luid.2018.11.004. 62. 2nd CFD challenge p edic ing pa ien -speci ic hemody- namics a es and s ess h ough an ao ic coa c a ion, h p://www. ascula model.o g/miccai2013/ (2013). 63. M. A. Fe n´andez, M. Landajuela, M. Vid ascu, Fully decoupled ime-ma ching schemes o incomp essible luid/ hin-walled s uc u e in e ac ion, Jou nal o Compu a ional Physics 297 (2015) 156–181. doi:10.1016/j.jcp.2015.05.009. 64. N. Wes e ho , J.-W. Lankhaa , B. E. Wes e ho , The a e ial Windkessel, Medical & biological enginee ing & compu ing 47 (2) (2009) 131–141. doi:10.1007/s11517-008-0359-2. 65. S. Pan , B. Fab `eges, J.-F. Ge beau, I. Vignon-Clemen el, A me hodolog- ical pa adigm o pa ien -speci ic mul i-scale c d simula ions: om clinical measu emen s o pa ame e es ima es o indi idual analysis, In e na ional jou nal o nume ical me hods in biomedical enginee ing 30 (12) (2014) 1614– 1648. doi:10.1002/cnm.2692. Ti le Supp essed Due o Excessi e Leng h 37 66. ˇ Z. Tuko i´c, A. Ka aˇc, P. Ca di , H. Jasak, A. I anko i´c, OpenFOAM ini e olume sol e o luid-solid in e ac ion, T ansac ions o FAMENA 42 (3) (2018) 1–31. doi:10.21278/TOF.42301. 67. J. Seo, D. E. Schia azzi, A. L. Ma sden, Pe o mance o p econdi ioned i - e a i e linea sol e s o ca dio ascula simula ions in igid and de o mable essels, Compu a ional Mechanics 64 (3) (2019) 717–739. doi:10.1007/s00466- 019-01678-3. 68. D. Jodlbaue , U. Lange , T. Wick, Pa allel block-p econdi ioned mono- li hic sol e s o luid-s uc u e in e ac ion p oblems, In e na ional Jou - nal o Nume ical Me hods in Enginee ing 117 (6) (2019) 623–643. doi:10.1002/nme.5970. 69. D. Fo i, A. Qua e oni, S. Depa is, e al., A pa allel algo i hm o he so- lu ion o la ge-scale noncon o ming luid-s uc u e in e ac ion p oblems in hemodynamics, Jou nal o Compu a ional Ma hema ics 35 (3) (2017) 363– 380. doi:10.4208/jcm.1702-m2016-0630.