Kemmine , Robin; Lange, Jannick; Kempkes, Jens Pe e ; Tie ney, Ke in; Weiß,
Dimi i
A icle — Published Ve sion
Con igu ing Mixed-In ege P og amming Sol e s o
La ge-Scale Ins ances
Ope a ions Resea ch Fo um
P o ided in Coope a ion wi h:
Sp inge Na u e
Sugges ed Ci a ion: Kemmine , Robin; Lange, Jannick; Kempkes, Jens Pe e ; Tie ney, Ke in; Weiß,
Dimi i (2024) : Con igu ing Mixed-In ege P og amming Sol e s o La ge-Scale Ins ances,
Ope a ions Resea ch Fo um, ISSN 2662-2556, Sp inge In e na ional Publishing, Cham, Vol. 5, Iss. 2,
h ps://doi.o g/10.1007/s43069-024-00327-7
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/316971
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h p://c ea i ecommons.o g/licenses/by/4.0/
h ps://doi.o g/10.1007/s43069-024-00327-7
RESEARCH
Con igu ing Mixed-In ege P og amming Sol e s
o La ge-Scale Ins ances
Robin Kemmine 1·Jannick Lange1·Jens Pe e Kempkes1·Ke in Tie ney2·
Dimi i Weiß2
Recei ed: 6 Ap il 2023 / Accep ed: 30 Ap il 2024 / Published online: 30 May 2024
Abs ac
Algo i hm con igu a ion echniques au oma ically sea ch o pa ame e s o sol e s
and algo i hms ha p o ide minimal un ime o maximal solu ion quali y on speci ied
ins ance se s. Mixed-in ege p og amming (MIP) sol e s pose a pa icula challenge
o algo i hm con igu a o s due o he di icul y o inding op imal, o e en easible,
solu ions on he la ge-scale p oblems commonly ound in p ac ice. We in oduce
he OPTANO Algo i hm Tune (OAT) o ind con igu a ions o MIP sol e s and
o he op imiza ion algo i hms. We p esen and e alua e se e al c i ical componen s o
OAT o sol ing MIPs in pa icula and show ha OAT can ind con igu a ions ha
signi ican ly imp o e he solu ion ime o MIPs on wo di e en da ase s.
Keywo ds Algo i hm con igu a ion ·Mixed-in ege p og amming ·La ge-scale
p oblem ins ances
BKe in Tie ney
ke[email p o ec ed]
Robin Kemmine
[email p o ec ed]
Jannick Lange
[email p o ec ed]
Jens Pe e Kempkes
jens.pe e [email p o ec ed]
Dimi i Weiß
[email p o ec ed]
1OPTANO GmbH, Technologiepa k 18, Pade bo n 33100, NRW, Ge many
2Decision and Ope a ion Technologies, Biele eld Uni e si y, Uni e si ä ss aße 25, Biele eld 33615,
NRW, Ge many
123
Ope a ions Resea ch Fo um (2024) 5: 48
© The Au ho (s) 2024
1 In oduc ion
The pe o mance o algo i hms and sol e s a ies g ea ly depending on he se ings o
he pa ame e s con olling he beha io o he app oach [1]. In pa icula , pa ame e
se ings ha wo k well o a pa icula da ase o ins ances may wo k poo ly on a
di e en da ase , especially in e ms o special p oblem s uc u es o ins ance sizes.
To ensu e good pe o mance o an algo i hm, ei he in e ms o un ime o solu ion
quali y, i is c i ical ha algo i hm pa ame e s be con igu ed o uned o he ypes o
ins ances he algo i hm is expec ed o sol e in p ac ice.
Sea ching o high-quali y pa ame e se ings by hand is a ime-consuming
endea o , hence se e al ools ha e been de eloped o au oma ically de e mine good
pa ame e se ings o a sol e o algo i hm gi en a da ase o ins ances. These ools
use a a ie y o me hods anging om ac ional ac o ial design [2], local sea ch [3,
4], gene ic algo i hms [5–7], Bayesian op imiza ion [8,9], and acing [10] (see [1] o
a ull o e iew). Mos algo i hm con igu a o s suppo con igu ing o one o bo h o
he ollowing se ings: (1) minimiza ion o a ge algo i hm un ime o (2) maximiza-
ion o solu ion quali y. Some a ge algo i hms, such as mixed-in ege p og amming
sol e s, equi e a mix u e o con igu ing o un ime and solu ion quali y o ind high
quali y solu ions [6] o e ec i ely une hei pa ame e s o a gi en da ase .
Mixed-in ege p og amming (MIP) sol e s can ackle a wide ange o p oblem
ypes and hus ough o be con igu ed o he ins ance se hey a e mean o sol e.
Indeed, wi h his in mind, IBM CPLEX, Gu obi, and FICO Xp ess, h ee o he mos
well-known gene al MIP sol e s, ha e buil in pa ame e uning capabili ies [11–
13]. Mo eo e , MIP sol e s ha e been a ocus o he algo i hm con igu a ion (AC)
communi y o some ime, wi h ea ly esul s p o iding speed-ups o up o 52x on he
CPLEX sol e [14] and ecen esul s showing he e a e s ill pe o mance gains o be
had in uning hese app oaches [15,16].
Mos o he successes o AC sol e s on MIP ha e in ol ed small-scale p oblems;
howe e , in indus y, p oblems wi h ens o housands o e en millions o a iables
mus be sol ed on a egula basis. These ex emely la ge p oblems pos a challenge o
con igu a o s. On he one hand, when uning o un ime, many ins ances will likely
no inish in he gi en imeou , leading o was ed execu ions and poo pe o mance
o he con igu a o . On he o he hand, when uning o solu ion quali y, many MIP
uns may no ind easible solu ions, meaning a mechanism o compa ing hese ailed
execu ions is equi ed o p o ide he con igu a o wi h a sea ch ajec o y.
This pape in oduces he OPTANO Algo i hm Tune (OAT), a gene al algo i hm
con igu a o ha has a special ocus on add essing la ge-scale MIP ins ances. The
con ibu ions a e as ollows:
•We desc ibe OAT, an AC ool based on he GGA algo i hm.
•We in es iga e a dominance acing mechanism o sho en con igu a ion imes on
MIP wi hou sac i icing o e all pe o mance.
•We u he show on a eal wo ld p oblem ha con igu ing smalle copies o la ge
ins ances (i.e., ins ances o educed size, bu simila s uc u es o la ge ins ances)
is e ec i e o inding good con igu a ions o he la ge ins ances.
123
48 Page 2 o 14 Ope a ions Resea ch Fo um (2024) 5: 48
We make OAT eely a ailable unde he MIT license a h ps://gi hub.com/
OPTANO/op ano.algo i hm. une .
This pape is o ganized as ollows: We discuss he cu en s a e-o - he-a o con-
igu ing MIP sol e s in Sec ion2. In Sec ion3, we desc ibe OAT, which o ms he
expe imen al basis o his wo k ollowed by he ex ensions o OAT speci ically o
con igu ing MIP sol e s. We e alua e he ex ensions compu a ionally in Sec ion4and
show ha OAT can ind high-quali y con igu a ions o a la ge-scale, eal-wo ld MIP
da ase . Finally, we discuss u u e wo k and conclude in Sec ion 5.
2 Rela ed Wo k and Backg ound In o ma ion
We p o ide a gene al o e iew o AC, including o line and eal ime AC, and discuss
i s applica ion o con igu e MIP sol e s. Fo u he de ails abou AC and ela ed
p oblem se ings, we e e in e es ed eade s o [1].
2.1 O line Au oma ed AC
We i s o malize he o line AC p oblem and adop he no a ion in [3]. The goal o
AC is o op imize he pe o mance o a pa ame e ized algo i hm A. To achie e his,
he con igu a o sea ches o high-quali y pa ame e con igu a ions θin he space o
all possible con igu a ions o A. The quali y o a con igu a ion is measu ed by a
pe o mance me ic mon a se o p oblem ins ances ⊆ˆ
, whe e ˆ
ep esen s he
ull dis ibu ion o p oblem ins ances and he sample he AC app oach is p o ided,
such ha m:ˆ
×→R. The gene al p ocess o algo i hm con igu a ion is depic ed
in Fig.1.
O line AC aims a inding a high quali y con igu a ion θ∗ ha pe o ms well o e
any possible se d awn om ˆ
. To his end, a se o p oblem ins ances , called
he aining se , is d awn om ˆ
is p o ided o he AC me hod ha is ep esen a i e
o ˆ
. The con igu a ion space is sea ched o high quali y con igu a ions θion he
aining se , whe e he aim is o minimize π∈ˆ
m(π, θ) in he un ime scena io,
whe eas in he solu ion quali y scena io, his e m is maximized.
Fig. 1 The in o ma ion low o o line au oma ed AC
123
Page 3 o 14 48Ope a ions Resea ch Fo um (2024) 5: 48
Se e al well-known app oaches ha e been de eloped o o line AC using bo h
model-based (i.e., machine lea ned models o p edic /e alua e con igu a ions) and
non-model-based app oaches. Pa amILS [3], a non-model-based app oach, employs
an i e a ed local sea ch combined wi h an adap i e capping mechanism o a oid was -
ing CPU ime on poo ly pe o ming con igu a ions. The AC me hod on which ou
app oach in his pape is based, GGA [5,7], uses a gene ic algo i hm wi h a ou namen -
based acing mechanism, while i ace [10] also uses acing, bu in a s a is ical ashion.
GPS [4] exploi s pa ame e con igu a ion landscape s uc u es and examine spa ame-
e s in a semi-independen way. Model-based con igu a o s include SMAC [8], which
is based on a Bayesian op imiza ion pa adigm ha uses a andom o es o p edic he
pe o mance me ic o a gi en con igu a ion, and GGA++ [6], which uses a andom
o es wi h a modi ied ee building mechanism o p edic con igu a ion pe o mance.
2.2 Algo i hm Con igu a ion o MIP
The AC communi y has long a ge ed he MIP se ing due o he long un imes o
sol ing MIP ins ances and he indus ial ele ance o MIPs. Se e al comme cial MIP
sol e s include con igu a ion p ocedu es, such as CPLEX [11], Gu obi [12], and FICO
Xp ess [13], al hough we no e ha hese ha e no been shown o be mo e e ec i e
han any AC me hod in he li e a u e.
Pa amILS is used o con igu e he sol e s CPLEX, Gu obi, and LpSol e in [14],
esul ing in signi ican speedups on se en di e en ins ance se s. Se e al MIP se -
ings a e included in he AClib [17], allowing de elope s o AC me hods o es on
s anda d benchma ks. AC me hods ha e also been used o con igu e MIPs in ins ance-
speci ic se ings, i.e., a speci ic con igu a ion θis assigned o each ins ance in ˆ
Pi, e.g.,
in [18] using he Hyd a me hod [19] and in [20] using he ins ance-speci ic algo i hm
con igu a ion (ISAC) app oach. We u he no e ha online/dynamic app oaches o
con igu ing MIP pa ame e s exis , e.g., DASH [21] (see also dynamic AC (DAC), [22]).
3 OPTANO Algo i hm Tune
OAT is a gene al algo i hm con igu a o dis ibu ed as a.NET nuge package ha can
be used as a s andalone con igu a o o in eg a ed di ec ly in o sol e s o algo i hms
w i en in.NET. The goal o OAT is o p o ide a con igu a o ha has s a e-o - he-a
pe o mance combined wi h he eliabili y expec ed o so wa e unning in p oduc-
ion. While OAT is o iginally based on GGA [5] and GGA++ [6], i has since been
ex ended o include sea ch s a egies based on JADE [23] and ac i e CMA-ES [24].
OAT is inhe en ly dis ibu ed and can un i s a ge algo i hm in pa allel ac oss mul-
iple machines o educe he o e all wall-clock ime o he con igu a ion p ocess. In
addi ion, OAT suppo s con igu ing in mul iple sessions, allowing he con igu a ion
p ocess o be es a ed should i be in e up ed by, e.g., a sys em ailu e o eaching
a esou ce limi . Finally, OAT includes nume ous ideas om he li e a u e, including
pa ame e ee cus omiza ion ( om GGA), non-nume ic e alua ion me ics (GGA++),
and adap i e capping s a egies (Pa amILS). We i s desc ibe how OAT wo ks and
123
48 Page 4 o 14 Ope a ions Resea ch Fo um (2024) 5: 48
desc ibe how i dis ibu es jobs ac oss co es, which di e s om p e ious dis ibu ed
e sions o GGA. Then, we in oduce i s MIP-speci ic enhancemen s, namely he new
e alua ion me ic and sho -ci cui domina ion ule.
3.1 OAT’s Con igu a ion P ocess
OAT is based on he gene ic algo i hm-based GGA and GGA++ con igu a o s om
a me hodological s andpoin , howe e no an enginee ing one. The unc ion o
OAT consis s o h ee phases: (1) ini ializa ion, (2) he main loop, and (3) con e -
gence/ e mina ion. The main loop i e a es un il OAT ei he eaches he maximum
numbe o gene a ions (as speci ied by he use ), a maximum numbe o e alua ions
o he a ge algo i hm, o uns ou o ime. Figu e2p o ides an o e iew o he
unc ion o OAT, and we e e eade s o [5] and [6] o u he de ails.
Fig. 2 O e iew o he GGA app oach [5]usedinOAT
123
Page 5 o 14 48Ope a ions Resea ch Fo um (2024) 5: 48
Ini ializa ion Conside ing he p e iously in oduced o maliza ion o algo i hm con-
igu a ion, OAT needs he ollowing ou inpu s o s a i s sea ch. Fi s , i needs a lis
o ins ances, I, ha will be in es iga ed, po en ially associa ed wi h andom seeds.
Second, OAT mus be old how o in oke he a ge algo i hm, ei he on he command
line o h ough an in e ace in o o he .NET code. Thi d, he a ge algo i hm pa am-
e e s o be con igu ed mus be speci ied. OAT akes a s uc u ed iew o pa ame e s
as in GGA, accep ing a pa ame e ee de ining ela ions be ween pa ame e s (see
pa 1(c) o Fig.2). Fo example, Gu obi [12] con ains se e al pa ame e s ela ing o
he p esol e ha can be adjus ed o change i s beha io . Ano he pa ame e u ns he
p esol e on and o , meaning ha he pa ame e s ela ing o he p esol e depend on
he pa ame e o u n i on and o . This in o ma ion is used du ing sea ch o gene a e
new con igu a ions. Thus, he dependen pa ame e s a e placed lowe below he p e-
sol e on/o pa ame e , and he ecombina ion p ocedu e akes his in o accoun when
c ea ing new indi iduals. Finally, OAT’s own in e nal pa ame e s can be changed om
hei de aul alues, ela ing o how i s sea ch s a egy unc ions.1
Gi en he inpu s ou lined abo e, OAT ini ializes a popula ion consis ing o he
de aul con igu a ion(s) and andomly gene a ed con igu a ions, pa i ioned in o wo
g oups, ep esen ing he compe i i e (C) con igu a ions ha will be un on he a ge
algo i hm, and non-compe i i e (N) con igu a ions, which ac as a di e si y s o e.
Main Loop This phase consis s o up o n gene a ions, in which con igu a ions om
he C popula ion a e assessed in aces and he winne s a e ecombined wi h non-
compe i i e con igu a ions. A he beginning o each gene a ion, a subse o ins ances
a e sampled om he ins ance pool. This subse linea ly inc eases wi h each gene a ion
un il ei he all ins ances a e used o a use -speci ied maximum alue is eached. All
con igu a ions o he compe i i e popula ion mus be e alua ed on he cu en ly ac i e
subse o ins ances. We no e ha some con igu a ions may ha e al eady been e alua ed
on some o he ins ances in p e ious gene a ions; hese con igu a ions need no be
e alua ed on he same ins ances again. I he size o he compe i i e popula ion is
la ge han he numbe o a ailable CPUs,2 he con igu a ions a e spli in o mini-
ou namen s equal o he numbe o a ailable CPUs. In he pu e un ime se ing,
mini- ou namen s a e execu ed un il a ixed pe cen age o he con igu a ions ha e
sol ed all ins ances; in he case o Fig.2, only one con igu a ion can win he ace. The
es o he con igu a ions a e subsequen ly e mina ed when hey ha e used he same
amoun o CPU ime as he winning con igu a ion(s). The mechanism by which OAT
dis ibu es mini- ou namen s is desc ibed in mo e de ail below. In he case o MIP, we
sligh ly modi y his p ocedu e and desc ibe his in Sec ions3.2 and 3.3.
A e all planned e alua ions o he cu en gene a ion a e comple ed, he pop-
ula ion is upda ed based on he pe o mance o he con igu a ions. Se e al op ions
a e a ailable o do his, such as he GGA c osso e mechanism in GGA, he gene ic
1We no e ha “ uning he une ” poses a signi ican challenge; hus, hese pa ame e s a e se o alues ha
ha e wo ked well o GGA and GGA++ in he pas .
2We assume he a ge algo i hm is single h eaded in ou desc ip ions; howe e , OAT also suppo s
con igu ing mul i h eaded a ge algo i hms.
123
48 Page 6 o 14 Ope a ions Resea ch Fo um (2024) 5: 48
enginee ing algo i hm o GGA++, as well as app oaches based on JADE and ac i e
CMA-ES. Figu e2shows he GGA c osso e mechanism in which he winne s o
he mini- ou namen s a e ecombined wi h andomly chosen membe s o he non-
compe i i e popula ion. The c osso e p ocedu e cons uc s a new con igu a ion by
andomly choosing componen s om he wo pa en s. We e e o [5] o he ull
de ails o his algo i hm and o he subsequen mu a ion ope a o . A speci ied pe -
cen age o he popula ion is eplaced e e y gene a ion (usually one hi d) h ough
he ecombina ion p ocedu e in he hope o gene a ing high-quali y con igu a ions.
Model-based ecombina ion is also possible in OAT using he GGA++ ecombina ion
s a egy, see [6] o de ails.
Te mina ion The main loop o OAT uns un il one o h ee condi ions is eached. The
i s condi ion is whe he he maximum numbe o gene a ions is achie ed (usually
75 o 100). The second condi ion is whe he a maximum numbe o e alua ions o
he a ge algo i hm is exceeded. Finally, he hi d condi ion is whe he he maximum
wall-clock ime o he con igu a o is exceeded. No e ha GGA suppo s a con e -
gence c i e ion ha checks whe he he popula ion is imp o ing o no , bu his is no
implemen ed ye in OAT.
Inc easing Mini-Tou namen CPU U iliza ion The mini- ou namen s as desc ibed
abo e mus be e icien ly dis ibu ed ac oss he a ailable CPU esou ces. A key engi-
nee ing ad ancemen o OAT o e p e ious GGA con igu a o s is ha i a emp s
o maximally u ilize a ailable CPU esou ces. While OAT uns mini- ou namen s o
ace compe i i e con igu a ions, i dis ibu es mini- ou namen s ac oss mul iple nodes
acco ding o a p io i y queue o con igu a ion-ins ance-seed uples ha mus s ill be
un, leading o less was ed CPU capaci y han, e.g., GGA and GGA++. OAT p io i izes
con igu a ions ha i belie es a e likely o inish i s so ha he inishing ime can be
used in he sho -ci cui e alua ion o o he con igu a ions acco ding o he o mula
p io i y(c)=100 imeou s(c)
|Ig|+10 unning(c)
|Ig|+ un ime(c)
κ|Ig|,
whe e cis a con igu a ion being e alua ed in he cu en gene a ion, g, imeou s(c)
p o ides he numbe o imeou s he con igu a ion chas had in he cu en gene a ion
so a , Igis he ins ance subse being conside ed in he cu en gene a ion, unning(c)
desc ibes he numbe o ins ance-seed pai s cis cu en ly unning on, un ime(c)
gi es he o al un ime o cso a in he cu en gene a ion, and κis he imeou as
p e iously de ined.
The p oposed mechanism uns con igu a ion-ins ance-seed uples wi h a low alue.
The in ui ion is ha a low p io i y sco e co esponds i s o con igu a ions wi h a low
numbe o imeou s, ollowing ha con igu a ions ha ha e no ye seen much CPU
ime a e a o ed, and inally, he o al un ime expended should be a low pe cen age
o he o al CPU ime allo men o he con igu a ion. In his way, con igu a ions a e
p e e ed ha a e likely o inish he mini- ou namen s i s , allowing us o domina e
poo -pe o me s be o e hey was e CPU esou ces (see Sec ion 3.3).
123
Page 7 o 14 48Ope a ions Resea ch Fo um (2024) 5: 48
3.2 MIP E alua ion Me ic
The e alua ion me ic ells OAT how o in e p e and agg ega e he pe o mance o he
a ge algo i hm on a subse o he aining ins ances. One o he main conside a ions
when de eloping a un ime e alua ion me ic is how o deal wi h imeou s. While
many con igu a o s simply use he a e age pe o mance o a con igu a ion o e a se
o ins ances, his does no signi ican ly discou age imeou s om occu ing. Hence,
many con igu a o s also suppo he so-called PAR10 sco e, which ex ends he a e age
by mul iplying imeou s by a ac o o 10.
While PAR10 e ec i ely penalizes imeou s, when an ins ance se con ains many
di icul ins ances, i o en does no o e e ec i e sea ch guidance. To imp o e on
his, in he g ay-box con igu a ion schemes [25] and [16], we analyze in e media e
ou pu o he a ge algo i hm o assis in anking o o he wise sco ing imeou s. In
he case o CPPL, ies be ween con igu a ions ha do no inish a e b oken using he
quali y o he easible solu ion ound (i one was ound).
In con as o eal ime con igu a ion, whe e only a single ins ance is sol ed pe
i e a ion, in o line con igu a ion, b eaking imes is somewha mo e complica ed. Espe-
cially in he i s ew i e a ions o con igu a ion, imeou s a e e y likely as he sea ch
p ocess has no ye iden i ied good con igu a ions. Hence, i is c i ical o ha e an e ec-
i e mechanism o compa ing con igu a ions e en i none ind op imal solu ions o
he ins ances being sol ed. Thus, o minimize he un ime o sol ing MIPs, we p opose
he ollowing simple anking scheme. Assume we a e gi en wo con igu a ions Aand
B ha a e un on nins ances and he ollowing ules a e applied in o de :
1. I A inds mo e easible solu ions han B,Ais be e .
2. O he wise, i Ahas less imeou s han B,Ais be e .
3. O he wise, i Ahas a lowe a e age MIP gap o e he imeou uns han B,A
is be e .
4. O he wise, i Ahas a lowe a e age un ime han B,Ais be e .
Since he un ime is a loa ing poin alue, and he e is gene ally some noise in i s
measu emen , his anking is all bu gua an eed o e u n a o al o de o e he a ailable
con igu a ions. No e ha he ocus o he anking is on easibili y and no op imali y.
The eason o his is ha companies sol ing MIPs in p ac ice would much a he ha e
easible solu ions o all o he ins ances hey a e in es iga ing han op imal solu ions
on a ew and no solu ion a all on he es . Howe e , while ou mo i a ion o hese
ule se is a p ac ical one, we show la e ha he e is also a compu a ional bene i o he
ules, as hese ules help guide he con igu a o ’s sea ch owa ds a eas o he sea ch
space wi h con igu a ions e ec i e a inding op imal solu ions.
3.3 Dominance Racing
Running MIPs is compu a ionally expensi e; hus, i we de ec ha a pa icula con-
igu a ion is domina ed, we can s op unning i and use he a ailable esou ces o un
some hing else. GGA and GGA++ accomplish his in he a e age o PAR10 un ime
se ing h ough hei acing mechanism, which ensu es ha con igu a ions ha a e
domina ed a e s opped be o e was ing CPU esou ces. Howe e , when using a ank-
123
48 Page 8 o 14 Ope a ions Resea ch Fo um (2024) 5: 48
