This is a sel -a chi ed e sion o an o iginal a icle. This e sion
may di e om he o iginal in pagina ion and ypog aphic de ails.
Au ho (s):
Ti le:
Yea :
Ve sion:
Copy igh :
Righ s:
Righ s u l:
Please ci e he o iginal e sion:
CC BY 4.0
h ps://c ea i ecommons.o g/licenses/by/4.0/
T eed Gaussian P ocess Reg ession o Sol ing O line Da a-D i en Con inuous
Mul iobjec i e Op imiza ion P oblems
© MIT P ess 2023
Accep ed e sion (Final d a )
Mazumda , A anu; López-Ibáñez, Manuel; Chugh, Tinkle; Hakanen, Jussi;
Mie inen, Kaisa
Mazumda , A., López-Ibáñez, M., Chugh, T., Hakanen, J., & Mie inen, K. (2023). T eed Gaussian
P ocess Reg ession o Sol ing O line Da a-D i en Con inuous Mul iobjec i e Op imiza ion
P oblems. E olu iona y Compu a ion, 31(4), 375-399. h ps://doi.o g/10.1162/e co_a_00329
2023
329
T eed Gaussian P ocess Reg ession o Sol ing
O line Da a-D i en Con inuous Mul iobjec i e
Op imiza ion P oblems
A anu Mazumda [email p o ec ed]
Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland
Manuel López-Ibáñez manuel.lopez-ibanez@manches e .ac.uk
Alliance Manches e Business School, Uni e si y o Manches e , UK
Tinkle Chugh .chugh@exe e .ac.uk
Depa men o Compu e Science, Uni e si y o Exe e , UK
Jussi Hakanen [email p o ec ed]
Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland
Kaisa Mie inen [email p o ec ed]
Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland
h ps://doi.o g/10.1162/e co_a_00329
Abs ac
Fo o line da a-d i en mul iobjec i e op imiza ion p oblems (MOPs), no new da a is
a ailable du ing he op imiza ion p ocess. App oxima ion models (o su oga es) a e
i s buil using he p o ided o line da a, and an op imize , o example, a mul iob-
jec i e e olu iona y algo i hm, can hen be u ilized o ind Pa e o op imal solu ions o
he p oblem wi h su oga es as objec i e unc ions. In con as o online da a-d i en
MOPs, hese su oga es canno be upda ed wi h new da a and, hence, he app oxi-
ma ion accu acy canno be imp o ed by conside ing new da a du ing he op imiza-
ion p ocess. Gaussian p ocess eg ession (GPR) models a e widely used as su oga es
because o hei abili y o p o ide unce ain y in o ma ion. Howe e , building GPRs
becomes compu a ionally expensi e when he size o he da ase is la ge. Using spa se
GPRs educes he compu a ional cos o building he su oga es. Howe e , spa se GPRs
a e no ailo ed o sol e o line da a-d i en MOPs, whe e good accu acy o he su o-
ga es is needed nea Pa e o op imal solu ions. T eed GPR (TGPR-MO) su oga es o
o line da a-d i en MOPs wi h con inuous decision a iables a e p oposed in his pa-
pe . The p oposed su oga es i s spli he decision space in o sub egions using e-
g ession ees and build GPRs sequen ially in egions close o Pa e o op imal solu ions
in he decision space o accu a ely app oxima e adeo s be ween he objec i e unc-
ions. TGPR-MO su oga es a e compu a ionally inexpensi e because GPRs a e buil
only in a smalle egion o he decision space u ilizing a subse o he da a. The TGPR-
MO su oga es we e es ed on dis ance-based isualizable p oblems wi h a ious da a
sizes, sampling s a egies, numbe s o objec i e unc ions, and decision a iables. Ex-
pe imen al esul s showed ha he TGPR-MO su oga es a e compu a ionally cheape
and can handle da ase s o la ge size. Fu he mo e, TGPR-MO su oga es p oduced
solu ions close o Pa e o op imal solu ions compa ed o ull GPRs and spa se GPRs.
Keywo ds
Gaussian p ocesses, K iging, eg ession ees, me amodelling, su oga e, Pa e o
op imali y.
Manusc ip ecei ed: 13 Ap il 2022; e ised: 6 No embe 2022, 23 Feb ua y 2023, 23 Ma ch 2023; accep ed: 31
Ma ch 2023.
© 2023 Massachuse s Ins i u e o Technology.
Published unde a C ea i e Commons
A ibu ion 4.0 In e na ional (CC BY 4.0) license. E olu iona y Compu a ion xx(x): 1–25
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
1 In oduc ion
A mul iobjec i e op imiza ion p oblem (MOP) consis s o wo o mo e con lic ing ob-
jec i e unc ions ha a e op imized simul aneously. The solu ions o an MOP a e called
Pa e o op imal when he alue o any objec i e unc ion canno be u he imp o ed
wi hou deg ading some o he o he s; hus, adeo s exis among he objec i e unc-
ions. No all eal-wo ld MOPs ha e analy ical unc ions o simula ion models a ail-
able. Ins ead, he MOP may need o be o mula ed by u ilizing he da a acqui ed om
a phenomenon, o example, eal-li e p ocesses, physical expe imen s, and senso s. To
sol e his MOP, i s he unde lying objec i e unc ions a e app oxima ed by i ing su o-
ga es (also known as me amodels) on he da a a ailable. Nex , a mul iobjec i e e o-
lu iona y algo i hm (MOEA) can be used o app oxima e he se o Pa e o op imal
solu ions by conside ing he su oga es as objec i e unc ions.
Da a-d i en op imiza ion p oblems a e classi ied in o online and o line ones (Jin
e al., 2019). Fo online da a-d i en op imiza ion, new da a can be acqui ed du ing he
op imiza ion p ocess, o example, by conduc ing u he expe imen s o simula ions
o upda e he su oga es (Shah ia i e al., 2016). In con as , o o line da a-d i en op i-
miza ion p oblems, no new da a can be acqui ed, and he op imiza ion has o p oceed by
using only he exis ing da a (Wang e al., 2019). Ideally, he unde lying objec i e alues
o he solu ions should be be e han he objec i e alues in he p o ided da ase . All
he a ailable da a should be u ilized o build su oga es wi h good app oxima ion while
sol ing o line da a-d i en MOPs. Howe e , when he size o he da a se is la ge, o ex-
ample, in Wang e al. (2016) and Wang and Jin (2020), building ce ain su oga es, such
as Gaussian p ocesses, can become compu a ionally expensi e. Because he su oga es
canno be upda ed wi h new da a while sol ing o line da a-d i en MOPs, he app ox-
ima ion accu acy o he su oga es di ec ly in luences he closeness o he solu ions o
he Pa e o op imal se . Mos o he p e ious wo ks on o line da a-d i en op imiza ion,
such as Wang e al. (2019, 2016), Wang and Jin (2020), and Yang e al. (2020), conside ed
di e en su oga e modelling echniques o sol ing o line da a-d i en op imiza ion
p oblems. Howe e , hese wo ks did no conside he adeo s be ween he unde lying
objec i e unc ions while building he su oga es. O he de eloped app oaches, o ex-
ample, Mazumda e al. (2019, 2022), Raha e al. (2018), and Hughes (2001), elied on he
unce ain y in he p edic ion o Gaussian p ocesses o imp o e he quali y o solu ions.
The p e ious wo ks on o line da a-d i en op imiza ion dealing wi h la ge da ase s, o
example, Wang e al. (2016) and Wang and Jin (2020) did no quan i y he unce ain y,
which is a c i ical componen in sol ing o line da a-d i en MOPs, as shown in Mazum-
da e al. (2019, 2020, 2022). This pape p oposes an app oach o build compu a ionally
cheap su oga es wi h a high app oxima ion accu acy a he egion a ound he Pa e o
op imal se . The su oga es use a combina ion o eg ession ees and Gaussian p ocess
eg ession models and p o ide unce ain y in he p edic ion. The p oposed su oga es
a e ailo ed owa d sol ing o line da a-d i en MOPs and a e capable o handling la ge
da ase s ( es ed o a maximum size o 50,000).
To s udy o line da a-d i en MOPs, da a is sampled om benchma k p oblems o
which he Pa e o op imal se is known. Knowledge o he Pa e o op imal se enables
quali y compa isons o he solu ions ob ained by di e en op imiza ion app oaches
and su oga es. While sol ing an o line da a-d i en MOP, gene ally wo ypes o in-
dica o s a e used o quan i y he quali y o he solu ions (Mazumda e al., 2019, 2020).
The app oxima ion accu acy is measu ed in oo mean squa ed e o (RMSE) be ween
he app oxima ed objec i e alues o he su oga es and he unde lying objec i e al-
ues. In addi ion, he hype olume (Zi zle and Thiele, 1998) indica o is used o measu e
2E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
how close he app oxima ed Pa e o on (a e e alua ing wi h he unde lying objec i e
unc ions) is o he Pa e o on o he unde lying MOP.
K iging, o Gaussian p ocess eg ession (GPR), (Fo es e e al., 2008) is a popu-
la choice o su oga e (Rasmussen and Williams, 2006) o sol ing o line p oblems
because i also p o ides in o ma ion abou he unce ain y in he app oxima ion. P e i-
ous wo ks (Hughes, 2001; Raha e al., 2018; Mazumda e al., 2019, 2022) ha e shown
ha u ilizing he unce ain y p edic ion om GPR su oga es p oduces solu ions wi h
imp o ed accu acy and hype olume. The unce ain y p edic ion om GPR makes i
an a ac i e choice o su oga es o sol ing o line da a-d i en MOPs. Mos o he ap-
p oaches o sol ing o line da a-d i en MOPs build a global GPR su oga e o each
objec i e using all he p o ided da a. Howe e , ull GPRs ha e high compu a ional and
memo y complexi y when he size o he da ase (o sample size) is la ge. An al e na-
i e o ull GPRs is o use spa se GPRs (Snelson and Ghah amani, 2005; Ti sias, 2009),
which build app oxima ion models wi h a small subse o da a called suppo o induc-
ing poin s. In Ti sias (2009), a a ia ional me hod was p oposed o selec he inducing
poin s by g adien descen o g eedy sea ch algo i hm. Howe e , his me hod becomes
compu a ionally expensi e when he da ase is la ge. On he o he hand, using ewe
poin s o building he su oga es educes he app oxima ion accu acy o he su oga es
compa ed o ull GPRs (Ti sias, 2009). O he wo ks, such as Emme ich e al. (2006), p o-
posed building GPR using only K-nea es neighbo samples om he poin ha is o be
p edic ed.
While pe o ming mul iobjec i e op imiza ion using su oga es, i is desi able o
ob ain a good app oxima ion accu acy a he egion a ound he Pa e o se , which is
e e ed o as he ade-o egion in his pape . The accu acy o he su oga es in o he e-
gions o he decision space (excluding he ade-o egion) is no o u mos impo ance.
A possible app oach o educe he compu a ional cos is o build local GPR su oga es
o he unde lying objec i e unc ions exclusi ely in he ade-o egion. Using local
GPRs educes he o e all compu a ional cos o building GPRs while p o iding a good
app oxima ion accu acy a he ade-o egion. P e ious wo ks (Kim e al., 2005; Das
and S i as a a, 2010; an S ein e al., 2015) pa i ioned he decision space in o egions
and i ed GPRs in each egion. Pa i ioning he decision space is ela i ely inexpensi e,
and each GPR u ilizes smalle se s o da a. This concep was ex ended by G amacy and
Lee (2008) o i GPRs in each egion o lea nodes pa i ioned by a Bayesian eed model
p e iously p oposed by Chipman e al. (1998). In Assael e al. (2014), eed GPRs we e
p oposed o deal wi h cases whe e he noise is di e en o di e en samples. All he
p e ious wo ks build GPRs in all he egions o he decision space. These ypes o GPRs
become compu a ionally expensi e, depending on he numbe o egions and amoun
o da a in each egion. Mo eo e , hese su oga es do no conside he ade-o egion
o he o line da a-d i en MOP du ing he building p ocess.
This pape p oposes eed GPR su oga es o mul iobjec i e op imiza ion (TGPR-
MO) which ha e a high accu acy a ound he ade-o egion and a e ailo ed o sol ing
o line da a-d i en MOPs wi h con inuous decision a iables. Fi s , compu a ionally in-
expensi e eg ession ee su oga es a e buil using he p o ided da ase . The eg ession
ee su oga es p o ide a less accu a e app oxima ion o he unde lying objec i e unc-
ions (compa ed o GPRs) (Loh, 2011) and spli he decision space in o egions. A egion
is de ined as he pa o he decision space ha is enclosed by he lea node o he i ed
eg ession ees. The app oxima ion accu acy o he p edic ion by he lea node is he
accu acy o he egion. Nex , an MOEA is execu ed conside ing he eg ession ees as
objec i e unc ions. The solu ions o he MOEA a e no e y accu a e bu p o ide he
E olu iona y Compu a ion Volume xx, Numbe x 3
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
app oxima e loca ion o he ade-o egion. A e a ce ain numbe o gene a ions o
he MOEA, he accu acy o he ees’ p edic ion is imp o ed by building GPRs a lea
nodes wi hin he ade-o egion. The GPRs buil imp o e he accu acy o he solu ions
in he egion o he decision space co esponding o he lea node hey eplace. The inal
su oga es consis o eg ession ees wi h GPRs a a ew lea nodes p o iding accu a e
app oxima ions exclusi ely in he ade-o egion.
The TGPR-MO su oga es we e es ed on se e al ins ances o dis ance-based i-
sualizable es p oblems (DBMOPP) wi h di e en numbe s o decision a iables and
objec i e unc ions. Va ious sizes o he ini ial da ase wi h di e en sampling s a e-
gies we e used in he es s. Nume ical expe imen s showed ha TGPR-MO su oga es
signi ican ly educed he building ime o su oga es when using la ge size da a while
sol ing o line da a-d i en MOPs. TGPR-MO su oga es also p oduced solu ions wi h
a be e hype olume compa ed o spa se GPR su oga es.
The es o he pape is a anged as ollows. The backg ound o o line da a-d i en
MOPs, GPRs, and eg ession ees is gi en in Sec ion 2. The p oposed TGPR-MO su o-
ga es a e de ailed in Sec ion 3. The expe imen al esul s consis ing o a compa ison o
TGPR-MO su oga es wi h o he su oga es a e p esen ed in Sec ion 4. Sec ion 5 con-
cludes and discusses he u u e esea ch pe spec i es.
2 Backg ound
Fo an o line da a-d i en MOP, he s a ing poin o sol ing he p oblem is (p ecol-
lec ed) da a. In his pape , we assume ha he a ailable da a is he ou pu o a p ocess
o phenomenon. The p ocess o gene a ing he da a is e e ed o as he unde lying ob-
jec i e unc ions o an MOP. The unde lying MOP ha has o be sol ed is conside ed o
be o he ollowing o m:
minimize ( 1(x),...,
K(x))
subjec o x∈,(1)
whe e K≥2 is he numbe o objec i e unc ions and is he easible egion in he
decision space n. Fo a easible decision ec o x(consis ing o ndecision a iables),
he co esponding objec i e ec o is (x)=( 1(x),...,
K(x)), whe e 1(x),...,
K(x)
a e he objec i e ( unc ion) alues.
A solu ion x∈domina es ano he solu ion x∈i i(x)≤ i(x) o all i=
1,...,K and i(x)<
i(x) o a leas one i=1,...,K. I a solu ion o an MOP is no
domina ed by any o he easible solu ions, i is called nondomina ed. Sol ing an MOP
using a mul iobjec i e op imiza ion algo i hm, o example, an MOEA ypically p o-
duces a se o mu ually nondomina ed solu ions. The solu ions o he op imiza ion
p oblem de ined in Equa ion (1) ha a e nondomina ed in he whole se a e called
Pa e o op imal solu ions.
A gene ic way o sol e an o line da a-d i en MOP wi h an MOEA as he op imize
is shown in Figu e 1. As explained in Jin e al. (2019) and Wang e al. (2019), he solu ion
p ocess can be di ided in o h ee componen s: (a) da a collec ion, (b) o mula ing he
MOP and su oga e building, and (c) op imiza ion wi h su oga es.
The op imiza ion p ocess s a s wi h an o line da ase consis ing o Nsamples. A
sample consis s o a decision ec o xand i s co esponding objec i e ec o (x)asa
uple o wo ma ices: (X, Y ) whe e X∈
N×nand Y∈
N×K. Each ow in Xand Yis
a decision ec o and i s co esponding objec i e ec o , espec i ely. Nex , su oga es
a e buil using all o a subse o he da a. The su oga es a e conside ed as objec i e
4E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023

329
T eed GPR o O line Da a-D i en MOPs
Figu e 1: Flowcha o a gene ic o line da a-d i en mul iobjec i e op imiza ion
app oach.
unc ions by an MOEA o sol e he o line da a-d i en MOP. Fo simplici y o desc ibing
a su oga e model o a single objec i e i,le yi∈
N×1be he ec o o objec i e alues
i(x) o all decision ec o s x∈X.
2.1 Gaussian P ocess Reg ession
In his wo k, a su oga e model is buil o each objec i e unc ion. Fo simplici y in
e minologies, yis conside ed he e ins ead o yi o in oduce he Gaussian p ocess e-
g ession (GPR) and eg ession ees. GPRs ha e been widely used in su oga e-assis ed
op imiza ion and ime-se ies analysis (Jin e al., 2019; Chugh e al., 2019). The majo
ad an age o using a GPR is i s abili y o p o ide he dis ibu ion abou i s p edic ion
(o he unce ain y). A GPR is a mul i a ia e no mal dis ibu ion wi h a mean μand a
co a iance ma ix C:
y∼N(μ,C).(2)
Fo simplici y in calcula ions, a mean o ze o is conside ed wi hou loss o gene ali y.
The co a iance ma ix Cuses a co a iance (o ke nel) unc ion o de ine co ela ion be-
ween wo samples, xand x. In his wo k, we use he Ma é n 5/2 ke nel because i
does no make he co a iance unc ions un ealis ically smoo h compa ed o o he ke -
nels such as he Gaussian (o RBF) ke nel. The e o e, i is mo e app op ia e o sol ing
p ac ical op imisa ion p oblems (Snoek e al., 2012). The ke nel is de ined as:
κ(x,x,)=σ2
⎛
⎝1+√5
n

j=1
j+5
3
n

j=1
2
j⎞
⎠exp ⎛
⎝−√5
n

j=1
j⎞
⎠+σ2
δxx
whe e j=|xj−x
j|
ljand xjand x
ja e he j h componen s o he decision ec o o x
and x,=(σ ,l
1,...,l
n,σ
) is he se o pa ame e s in he GPR model and δxxis he
K onecke del a unc ion. The no a ion |xj−x
j| ep esen s he Euclidean dis ance be-
ween xjand x
j. The pa ame e s σ ,ljand σ ep esen he ampli ude, leng h scale o
he j h a iable, and noise in he da a, espec i ely. Fo mo e de ails on he signi icance
o hese pa ame e s, see Rasmussen and Williams (2006).
To build a GPR model, he pa ame e s men ioned abo e can be es ima ed by max-
imizing he ma ginal likelihood unc ion:
p(y|X, )=1
√|2πC|exp −1
2yC−1y.(3)
A e es ima ing he pa ame e s, he model de ined in Equa ion (2) can be used o he
pos e io p edic i e dis ibu ion a a new decision ec o x∗. The GPR model p o ides
a pos e io p edic i e dis ibu ion which is also Gaussian:
p(y∗|x∗,X,y,)=Nκ(x∗,X,)C−1y,κ(x∗,x∗,)−κ(x∗,X,)C−1κ(X, x∗,).(4)
E olu iona y Compu a ion Volume xx, Numbe x 5
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
The pos e io mean in he equa ion abo e is κ(x∗,X,)C−1y, and he a iance ep e-
sen ing he unce ain y is κ(x∗,x∗,)−κ(x∗,X,)C−1κ(X, x∗,).
2.2 Reg ession T ees
A eg ession ee ecu si ely pa i ions he decision space such ha he p o ided sam-
ples wi h simila objec i e unc ion alues a e g ouped oge he (Loh, 2011). Each
node φo he ee con ains da a Qφ=(Xφ,yφ)wi hNφsamples, whe e each ow o
Xφ∈
Nφ×nis a decision ec o xiand each ow o yφ∈
Nφ×1is i s co esponding objec-
i e alue yi,i=1,...,Nφ.Nodeφis spli in o nodes φle and φ igh acco ding o pa am-
e e θ=(j, ), which speci ies a j h decision a iable (1 ≤j≤n) and a h eshold ,by
pa i ioning Qφin o wo disjoin subse s Qφle
θ={(Xφle ,yφle )|(xi,y
i)∈Qφ∧xij ≤ }
and Qφ igh
θ=Qφ Qφle
θ. Tha is, spli pa ame e θpa i ions he samples ( ows) in Qφ
acco ding o he alue o a iable xjo each decision ec o x∈Xφ.
Gi en a node φand a spli pa ame e θ, he quali y o he spli o he o al loss is
calcula ed as:
G(Qφ,θ)=Nφle
NφH(Qφle
θ)+Nφ igh
NφH(Qφ igh
θ),(5)
whe e Nφle and Nφ igh a e he numbe o samples in Qφle
θand Qφ igh
θ, espec i ely, and
H(Qφ) is a loss unc ion. Fo ins ance, he loss unc ion o mean-squa ed e o is:
H(Qφ)=1
Nφ
yi∈Qφ
(yi−¯yφ)2,(6)
whe e ¯yφis he mean objec i e alue in Qφ. An op imal spli θ∗o a node φcan be
ound by minimizing he loss unc ion de ined in Equa ion (5) using a single objec i e
op imiza ion algo i hm. In he p oposed TGPR-MO su oga es, he spli ing p ocess is
ecu sed o bo h Qφle
θ∗and Qφ igh
θ∗un il spli ing a node would p oduce a lea node wi h
less han a p ede ined minimum numbe o samples, Nmin.
Fo p edic ing any gi en decision a iable alue, he eg ession ee is a e sed o
he espec i e lea node. The p edic ion o he ee a he lea node l, which con ains
aining subse Qlwi h Nlnumbe o samples, is ¯yl=1
Nlyi∈Qlyi.
3 T eed GPRs o Mul iobjec i e Op imiza ion (TGPR-MO)
The ime complexi y o a ull GPR is cubic, which is polynomial. A sui able al e na i e
o educe he compu a ional cos is o spli he da ase and build GPRs exclusi ely in he
ade-o egion. The p oposed TGPR-MO su oga es a e based on his modelling ap-
p oach. The ollowing subsec ions desc ibe he building p ocess o he p oposed TGPR-
MO su oga es. A de ailed accu acy and complexi y analysis and compa ison wi h ull
GPRs and spa se GPRs a e also p o ided.
3.1 Building he TGPR-MO Su oga es
In a gene alized eed GPR su oga e, i s a eg ession ee model is buil using he p o-
ided da a as desc ibed in Sec ion 2.2. The spli ing a he nodes is done by minimizing
he o al loss unc ion (Equa ion 5). The subse o da a a he l h lea node is Ql, whe e
l=1,...,Land Lis he o al numbe o lea nodes in he eg ession ee buil . A GPR
is i ed a e e y lea node o he eg ession ee using he da a Ql=(Xl,yl). The GPRs
a e buil in a simila ashion as desc ibed in Sec ion 2.1 by maximizing he ma ginal
6E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
likelihood a he lea node l, as de ined in Equa ion (3). The pos e io p edic i e dis i-
bu ion o he l h GPR is py∗|x∗,X
l,yl,, as de ined in Equa ion (4). Building GPRs
wi h smalle subse s o da a educes he o e all cos o building su oga es compa ed
o building a GPR wi h he en i e da ase . Howe e , building GPRs a all he lea nodes
becomes expensi e when he e a e oo many o hem, and he da a subse a each lea
node is la ge. Mo eo e , o sol ing an o line da a-d i en MOP, an accu a e app oxi-
ma ion o he global landscape o he unde lying objec i e unc ions is no equi ed. A
good app oxima ion o he local landscape nea he ade-o egion is su icien .
While pe o ming mul iobjec i e op imiza ion using su oga es, he app oxima-
ion accu acy in he ade-o egion is c ucial and di ec ly in luences he quali y o he
app oxima ed Pa e o op imal solu ions. Hence, o ob ain be e quali y solu ions, i is
desi able o ha e accu a e app oxima ions in he ade-o egion. While building he
eg ession ees, he spli s a each node di ide he decision space. The lea nodes o
he eg ession ees a e he smalles egions he decision space is spli in o. In addi ion,
he ini ial app oxima ion o eg ession ee su oga es ( hough no highly accu a e) p o-
ides in o ma ion abou he ade-o s be ween he objec i e unc ions. A e building
he eg ession ees wi h all he p o ided da a, an MOEA is un conside ing hem as
objec i e unc ions. The solu ions ound by he MOEA a e no accu a e, bu hey p o-
ide an app oxima ion o he Pa e o on . The accu acy o he ade-o egion is he
app oxima ion accu acy o he egion enclosed by he lea nodes ha p edic s he ap-
p oxima ed Pa e o on . La e , local GPRs a e buil exclusi ely in he lea nodes ep e-
sen ing he ade-o egion and achie e an accu a e app oxima ion only in he neigh-
bo hood o he Pa e o se . Nex , he algo i hm o build he eed GPR su oga es o
mul iobjec i e op imiza ion (TGPR-MO) ailo ed o sol e o line da a-d i en MOPs is
in oduced.
Algo i hm 1 s a s wi h a da ase con aining Nsamples wi h Kobjec i e unc ions
and ndecision a iables. Fi s , K eg ession ees (one pe objec i e unc ion) a e buil
wi h all he p o ided da a. Jus he pa ame e Nmin is adjus ed, and he dep h o he
ees is no con olled. A e building he eg ession ees, he e can be a maximum o
N
Nmin lea nodes. Ini ially, he e a e no GPRs p esen a he lea nodes, and he p edic ions
a e om he eg ession ees (i.e., ¯yl). Nex , a popula ion is ini ialized and an MOEA is
execu ed ha i e a i ely builds GPRs a speci ic lea nodes. Each i e a ion consis s o
unning he MOEA o Gmax gene a ions ( ha is a p ede ined pa ame e ) and building
a GPR a one lea node in e e y ee. In e e y gene a ion, o sp ing indi iduals a e
p oduced using c osso e and mu a ion ope a o s and e alua ed using he eed GPRs.
Selec ion o indi iduals is hen pe o med using MOEA-speci ic selec ion c i e ion, and
he p ocess is epea ed o Gmax gene a ions wi hin an i e a ion. The inne wo kings o
he p oposed algo i hm a e exempli ied in wo-dimensional decision spaces in Sec ion
1 and Figu e 1 o he supplemen a y ma e ial.
A e he MOEA comple es Gmax gene a ions, he app oxima ion e o s o he solu-
ions ( o each objec i e) a e compa ed. The compa ison is done by i s calcula ing he
loss unc ion alue, he e he mean squa ed e o , as de ined in Equa ion (6), o he lea
node p edic ing he objec i e alues o he solu ions ound by he MOEA. Fo he eed
GPR su oga e co esponding o he j h objec i e, le lj
1,...,lj
sdeno e he lea nodes
con aining he ssolu ions ound by he MOEA and le H(Qlj
1),...,H(Qlj
s) deno e he
loss unc ion alue o hose lea nodes, hen one GRP is buil a he lea node i∗wi h he
maximum loss alue, ha is, i∗=a g maxi=1,...,sH(Qlj
i), using i s subse o samples Qlj
i∗.
I mul iple solu ions all wi hin he same lea node, he loss alue is calcula ed once and
only one GPR is buil o ha lea node. The p ocess o building GPRs is epea ed o
E olu iona y Compu a ion Volume xx, Numbe x 7
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
8E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
o he op imiza ion p ocess. In addi ion, he complexi y o hese p oblems can be con-
olled by he ea u es (e.g., a ying densi y, numbe o local on s, dominance esis-
ance egions, numbe s o objec i e unc ions, and decision a iables). These ad an-
ages p o ide mo e con ol o e he p oblems and make hem adjus able o e lec he
needs o eal-wo ld op imiza ion p oblems. These ea u es o DBMOPP es p oblems
p o e o be mo e ad an ageous compa ed o DTLZ (Deb e al., 2005) benchma k p ob-
lems. A de ailed desc ip ion o he expe imen se ings, esul s, and in-dep h analysis
a e p o ided in he ollowing subsec ions.
4.1 Expe imen al Se up
All he app oaches o sol ing he o line da a-d i en MOP we e coded in Py hon u i-
lizing he DESDEO amewo k (Misi ano e al., 2021) (h ps://desdeo.i .jyu. i).1The
expe imen s we e execu ed on one node o an HPC clus e equipped wi h AMD Rome
CPUs, each node ha ing 128 co es unning a 2.6 GHz, wi h 256 GiB o memo y. Each
indi idual un was execu ed on one CPU co e and alloca ed a maximum memo y us-
age o 2 GiB. The eg ession ee was buil using he sklea n Py hon package (Ped egosa
e al., 2011). Fo building he GPRs a he lea nodes, he GPy (GPy, 2012) Py hon package
was used.
4.1.1 Benchma k P oblems
Fou DBMOPP p oblems P1–4, as shown in Table 1, we e used. The p oblem ins ances
and da a we e gene a ed by he code p o ided by Fieldsend e al. (2019). All combi-
na ions o numbe s o objec i e unc ions (K∈{3,5,7}) and numbe s o decision a i-
ables (n∈{2,5,7,10}) we e used o he es s. The o al numbe o p oblem ins ances
es ed was 48, and e e y p oblem ins ance ep esen ed a di e en ype o p oblem
cha ac e is ic.
4.1.2 Da ase s
Fo gene a ing he da a, La in hype cube sampling (LHS) and mul i a ia e no mal sam-
pling (MVNS) (Fo es e e al., 2008) we e used. In MVNS sampling, he objec i e unc-
ions we e conside ed independen wi h mean a he mid-poin o he decision space,
ha is, ze o o DBMOPP p oblems. The a iance o he sampling dis ibu ion was se o
0.1 o all he objec i e unc ions. Using MVNS sampling es s he abili y o he su oga e
models and op imiza ion algo i hm o handle biased (o skewed) da ase s (Mazumda
e al., 2022). Sample sizes o ini ial da a (N∈{2000,10000,50000}) we e chosen o he
es s. A o al o 288 cases we e es ed, and 31 se s o da a wi h a andom seed we e
gene a ed o each es case. Each o hese da ase s was he s a ing poin o he h ee
di e en su oga e models ha we e es ed. These indi idual uns we e independen ,
and he esul s we e used o compa e he su oga es’ pe o mances s a is ically.
4.1.3 Se ings o TGPR-MO Su oga es
The MOEA o building TGPR-MO su oga es was RVEA (Cheng e al., 2016), a
decomposi ion-based MOEA (Zhang and Li, 2007; Deb and Jain, 2014). Decomposi ion-
based MOEAs ha e shown o be e ec i e in sol ing o line da a-d i en MOPs wi h
mo e han h ee objec i e unc ions. Howe e , he su oga e is no limi ed o RVEA and
one can use any MOEA o choice. The pa ame e s o RVEA we e kep he same as sug-
ges ed by Cheng e al. (2016). The pa ame e Gmax was se o 50. The maximum numbe
o i e a ions was se o Imax =N
Nmin =N
10n. Such a alue o Imax was chosen conside ing
1Sou ce code a ailable a h ps://gi hub.com/indus ial-op imiza ion-g oup/T eedGP_MOEA
E olu iona y Compu a ion Volume xx, Numbe x 15
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023

329
A. Mazumda e al.
one GPR is buil a a lea node o all he ees in each i e a ion (as he maximum numbe
o lea nodes is N
Nmin ). The loss unc ion used o building he ees was MSE as men ioned
in Equa ion (6) wi h Nmin =10n. These pa ame e s we e chosen because su icien de-
sign poin s a e necessa y o building GPRs a he lea es. Based on ecommenda ions
om he GPR li e a u e (Chapman e al., 1994; Jones e al., 1998), a leas 10npoin s a e
equi ed a each lea node. The ke nel used o building he GPRs a he lea nodes was
Má e n 5/2 wi h au oma ic ele ance de e mina ion enabled.
4.1.4 O he Su oga es Tes ed
The p oposed TGPR-MO su oga es we e compa ed wi h spa se GPR and ull GPR
su oga es. The same GPy package and ke nel we e used o building bo h o hese
su oga es as o he TGPR-MO su oga es. Fo spa se GPR su oga es, he numbe o
induc ion poin s was se o M=10n. He e, he o e all p ocess o sol ing an o line da a-
d i en MOP wi h a speci ic su oga e is e e ed o as an app oach o simplici y. I should
be no ed ha andom o es was no conside ed in he es s as he ocus o his pape is
on GPR su oga es.
4.1.5 Pa ame e Se ings o MOEA ( o Sol ing he O line Da a-D i en MOP)
Fo sol ing he o line da a-d i en MOP wi h su oga es as objec i e unc ions, RVEA
wi h he same de aul pa ame e se ings was used. The e mina ion c i e ion o RVEA
was 1,000 gene a ions, which was su icien o all he app oaches es ed o con e ge. To
gene a e a uni o m Pa e o on , he e e ence ec o s a e ea anged o adap ed a e a
ce ain numbe o gene a ions in RVEA. The e e ence ec o adap a ion a e was se o
once e e y 100 gene a ions.
4.1.6 Pe o mance Indica o s
The quali y o he solu ions ob ained by he MOEA was measu ed in e ms o hei
hype olume (HV) a e e alua ing hem wi h he unde lying objec i e unc ions o
he es p oblems. Fo compu ing he HV indica o , he e e ence poin o y∗=(y∗
1,y∗
2,
...,y∗
K) in he objec i e space was chosen, such ha i is domina ed by all he solu ions.
In his pape , he e e ence poin used was y∗=(2√K,2√K,...,2√K) because his
poin is always domina ed in DBMOPP p oblems. The mul i a ia e RMSE o he ob-
jec i e alues o he solu ions ob ained by he MOEA wi h hei espec i e unde lying
objec i e alues was used o measu e he accu acy. The mul i a ia e RMSE is he Eu-
clidean dis ance be ween he app oxima ed and e alua ed unde lying objec i e unc-
ion alues o he solu ions and is gi en by 1
ss
i=1K
j=1(ˆ
j,i − j,i )2, whe e sis he
numbe o solu ions, ˆ
j,i and j,i a e he app oxima ed and he unde lying objec i e
alue, espec i ely, o he i h solu ion and j h objec i e. Finally, he compu a ional
cos o he a ious me hods was measu ed as he ime aken in seconds o build he
su oga es.
In his pape , mul i a ia e RMSE is e e ed o as RMSE o simplici y. The hype -
olume and RMSE indica o s a e used he e o benchma k he pe o mance o TGPR-
MO su oga es o sol ing an o line da a-d i en MOP. E alua ing he solu ions wi h
he unde lying objec i e unc ions in an o line da a-d i en MOP may no be possible
in eal li e.
4.2 Resul s and Discussions
While unning he expe imen s, i was obse ed ha building ull GPR su oga es wi h
10,000 and 50,000 sample sizes consis en ly ga e ou -o -memo y e o s. The s o age
16 E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
complexi y o ull GPRs is O(KN2), and each elemen o he a ay is a 64-bi loa . Hence,
he memo y equi emen o h ee objec i e unc ions and sample sizes o 10,000 and
50,000 is 2.3 GiB and 56 GiB, espec i ely. Thus, building GPR su oga es using all he
p o ided o line da a will become almos impossible wi h eadily a ailable compu ing
esou ces when he sample size is la ge. Hence, he esul s o ull GPR su oga es a e
no included o sample sizes o 10,000 and 50,000.
A pai wise Wilcoxon wo- ailed signi icance es was conduc ed o compa e he
pe o mance o he di e en app oaches. The calcula ed p- alues we e Bon e onni co -
ec ed, and α=0.05 was conside ed o ejec ing he null hypo hesis (an app oach is
no signi ican ly be e o wo se han ano he app oach). The median alues we e com-
pa ed o de e mine whe he an app oach is signi ican ly be e o wo se han ano he
one, p o ided ha he p- alue is less han α. A pai wise compa ison o he app oaches
wi h a sco ing sys em was used o anking he app oaches. An app oach is gi en a
sco e o +1 i i is signi ican ly be e han he o he app oach. A sco e o −1isgi en
o he app oach i i is signi ican ly wo se han he o he app oach. I he app oach is
no signi ican ly be e o wo se han he o he app oach, a sco e o ze o is gi en o
bo h app oaches. The sum o he sco es is used o anking all he app oaches (a highe
sco e gi es a be e ank) o he indica o being compa ed. A ank o “1” indica es ha
an app oach has pe o med signi ican ly be e han all o he app oaches. Equal anks
indica e ha hose app oaches a e no signi ican ly di e en in hei pe o mance.
The pe o mance o he app oaches is summa ized in Table 2. The anks o he ap-
p oaches o h ee di e en indica o s a e colo -coded in g een, yellow, and ed in he
o de o bes o wo s . I should be no ed ha hese ankings a e ca ego ized wi h e-
spec o he sample size and sampling s a egies. The o al numbe o ins ances is 96
(48 o LHS and MVNS each) o each sample size. Each ins ance consis s o he com-
bina ion o he a ious p oblem se ings, ha is, he numbe o samples (N), sampling
s a egy, numbe o objec i e unc ions (K), numbe o decision a iables (n), and p ob-
lem con igu a ion. The numbe o ins ances in which ull GPRs o spa se GPRs pe o m
be e , wo se, o no signi ican ly di e en compa ed o he p oposed TGPR-MO su o-
ga es is deno ed by “+,” “-,” and “≈,” espec i ely. Fo TGPR-MO su oga es, only he
numbe o ins ances whe e i pe o med signi ican ly be e han bo h ull GPRs and
spa se GPRs (o i anked he bes ) is shown.
The ows measu ing “Time” in Table 2 show ha he p oposed TGPR-MO su o-
ga es we e compu a ionally he cheapes compa ed o he ull GPR and spa se GPR o
di e en sample sizes and sampling s a egies. The numbe o ins ances TGPR-MO pe -
o med signi ican ly be e han he o he wo su oga es in each subca ego y is close o
48 ( ha was he o al numbe o ins ances in each subca ego y). The TGPR-MO su o-
ga es pe o med be e han spa se GPR su oga es in hype olume o all sample sizes
and sampling s a egies. Howe e , ull GPRs pe o med he bes in hype olume o a
sample size o 2,000. Fo 2,000 samples, spa se GPRs ou pe o med TGPR-MO su o-
ga es o bo h sampling s a egies in RMSE, and ull GPR pe o med he bes . The RMSE
o he solu ions o TGPR-MO su oga es was be e han spa se GPRs o sample sizes
o 10,000 and 50,000 o LHS sampling only, whe eas o MVNS sampling, he spa se
GPR sligh ly ou pe o med TGPR-MO su oga es o 10,000 and 50,000 samples.
Fo a smalle sample size, one may choose ull GPRs because hey gi e he bes pe -
o mance in hype olume and RMSE. Howe e , ull GPRs become almos impossible
o build due o hei high compu a ional cos when he sample size is la ge. One can
use TGPR-MO su oga es o la ge sample sizes because hey pe o m be e in hy-
pe olume and RMSE and excellen ly in compu a ion ime compa ed o spa se GPRs.
E olu iona y Compu a ion Volume xx, Numbe x 17
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
Table 2: Summa y o pai wise compa ison o he hype olume (HV), RMSE, and ime
(seconds) o he di e en app oaches. The numbe o ins ances in which ull GP and
spa se GP su oga es pe o m be e , wo se, and no signi ican ly di e en compa ed
o he TGP-MO su oga es is indica ed by “+,” “-,” and “≈,” espec i ely. The ull GP
uns ailed due o memo y o e low in he ins ances ma ked “—.” The app oaches’
pe o mance is anked by he colo code g een, yellow, and ed in he o de o bes o
wo s , espec i ely.
Su oga e ype
Sample Sampling Full GP Spa se GP TGP-MO
size s a egy Indica o +/-/≈+/-/≈+
2,000 LHS HV 9/9/30 4/38/6 9
RMSE 43/2/3 19/21/8 1
Time (s) 0/48/0 1/46/1 46
MVNS HV 17 / 14 / 17 14 / 28 / 6 12
RMSE 34/7/7 20 / 16 / 12 2
Time (s) 0/48/0 1/47/0 47
10,000 LHS HV — 4/33/11 33
RMSE — 19 / 23 / 6 23
Time (s) — 0/48/0 48
MVNS HV — 11 / 29 / 8 29
RMSE — 21 / 16 / 11 16
Time (s) — 0/48/0 48
50,000 LHS HV — 3/35/10 35
RMSE — 15 / 24 / 9 24
Time (s) — 0/48/0 48
MVNS HV — 10 / 29 / 9 29
RMSE — 23 / 19 / 6 19
Time (s) — 0/48/0 48
Howe e , o non-uni o m sampling s a egies, o example, MVNS, TGPR-MO su o-
ga es su e sligh ly in RMSE compa ed o spa se GPR su oga es. Spa se GPR su o-
ga es ha e be e RMSE han TGPR-MO su oga es because he a ia ional pa ame e s
a e selec ed by minimizing he KL di e gence. Thus, he inducing inpu s selec ed a e
no skewed e en i he p o ided da ase is, o example, in MVNS sampling.
The pe o mances o a ew selec ed ins ances a e shown in Table 3. The able shows
he median and he s anda d de ia ion o hype olume, RMSE, and ime aken o build
he su oga es o he h ee di e en app oaches o 31 uns. The ins ances shown in he
able we e chosen based on he maximum di e ence be ween he median hype olume
o he bes and he second bes pe o ming su oga es. One ins ance was selec ed om
each sample size, sampling s a egy, and numbe o objec i e unc ions. The igu es in
bold ep esen he bes pe o ming app oaches. I can be obse ed ha he p oposed
TGPR-MO su oga es pe o med he bes in building ime in he ins ances shown. I
can also be obse ed ha TGPR-MO su oga es p oduce solu ions wi h an imp o ed
hype olume and RMSE o sample sizes o 10,000 and 50,000 compa ed o spa se GPR
su oga es. I can be obse ed ha he building imes o TGPR-MO su oga es a e less
18 E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
Table 3: Compa ison o selec ed es ins ances showing he median hype olume, RMSE, and ime and hei s anda d de ia ion (in
i alics) o he 31 es uns. The bes pe o ming app oaches a e shown in bold.
Hype olume RMSE Time (s)
Sample Sampling Full Spa se Full Spa se Full Spa se
size s a egy P oblem K n GPR GPR TGPR-MO GPR GPR TGPR-MO GPR GPR TGPR-MO
2,000 LHS P3 3 5 7.54E+01 5.49E+01 7.48E+01 4.18E-02 1.04E+00 2.67E-01 1.36E+02 8.64E+01 1.40E+01
4.02E+00 9.35E+00 2.73E+00 2.74E-01 5.72E-01 1.38E-01 1.85E+01 2.16E+01 1.20E+01
P4 5 10 1.46E+04 1.27E+04 1.64E+04 1.75E+00 1.79E+00 9.33E-01 6.16E+02 5.51E+02 2.62E+01
6.73E+02 1.23E+03 5.59E+02 3.69E-01 4.38E-01 1.66E-01 1.40E+02 1.43E+02 1.12E+01
P3 7 5 9.21E+06 6.55E+06 9.23E+06 5.11E-02 2.16E+00 1.66E-01 3.20E+02 2.10E+02 2.83E+01
1.00E+05 1.24E+06 9.32E+04 8.26E-02 9.70E-01 1.11E-01 5.38E+01 3.46E+01 1.53E+01
MVNS P3 3 10 5.71E+01 5.65E+01 6.27E+01 1.08E+00 1.52E+00 8.55E-01 3.25E+02 3.56E+02 1.31E+01
4.14E+00 6.35E+00 5.10E+00 4.65E-01 6.09E-01 1.99E-01 6.76E+01 1.19E+02 2.45E+00
P2 5 10 1.26E+04 1.06E+04 1.30E+04 1.86E+00 2.76E+00 2.04E+00 5.34E+02 5.51E+02 1.39E+01
2.08E+03 2.10E+03 1.20E+03 1.95E-01 1.59E-01 2.78E-01 1.01E+02 7.64E+01 8.00E+00
P2 7 7 7.81E+06 6.42E+06 8.82E+06 7.80E-01 3.76E+00 9.80E-01 3.57E+02 4.36E+02 1.27E+01
5.36E+05 2.62E+05 5.30E+05 2.07E-01 8.55E-01 2.38E-01 4.40E+01 6.32E+01 1.69E+01
10,000 LHS P3 3 5 — 6.26E+01 7.58E+01 — 8.95E-01 1.50E-01 — 1.45E+03 2.49E+01
9.78E+00 5.93E-01 4.85E-01 5.05E-02 2.22E+02 8.80E+00
P4 5 10 — 1.27E+04 1.65E+04 — 2.10E+00 7.09E-01 — 6.52E+03 5.93E+01
6.09E+02 8.26E+02 2.22E-01 2.28E-01 1.24E+03 2.11E+01
P1 7 10 — 6.71E+06 8.59E+06 — 1.56E+00 9.73E-01 — 9.37E+03 3.78E+01
1.90E+05 2.61E+05 1.84E-01 8.82E-02 1.38E+03 1.32E+01
MVNS P3 3 10 — 4.97E+01 6.27E+01 — 1.68E+00 9.09E-01 — 4.12E+03 1.42E+01
5.40E+00 3.31E+00 3.40E-01 1.52E-01 6.69E+02 1.52E+00
P3 5 5 — 1.23E+04 1.75E+04 — 2.32E+00 1.83E-01 — 2.38E+03 4.73E+01
2.50E+03 3.15E+02 1.10E+00 1.22E-01 4.65E+02 1.19E+01
P3 7 5 — 6.55E+06 9.24E+06 — 2.91E+00 1.22E-01 — 3.51E+03 8.94E+01
1.22E+06 1.01E+05 1.34E+00 1.42E-01 6.54E+02 3.02E+01
E olu iona y Compu a ion Volume xx, Numbe x 19
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
Table 3: Con inued.
Hype olume RMSE Time (s)
Sample Sampling Full Spa se Full Spa se Full Spa se
size s a egy P oblem K n GPR GPR TGPR-MO GPR GPR TGPR-MO GPR GPR TGPR-MO
50,000 LHS P3 3 10 — 6.04E+01 7.30E+01 — 8.29E-01 3.91E-01 — 2.16E+04 3.16E+01
7.28E+00 2.62E+00 6.08E-01 8.63E-02 1.77E+03 9.25E+00
P3 5 10 — 1.45E+04 1.65E+04 — 8.64E-01 6.18E-01 — 3.59E+04 3.90E+01
4.10E+02 4.04E+02 2.10E-01 1.20E-01 6.29E+03 1.10E+01
P1 7 10 — 7.49E+06 8.65E+06 — 9.20E-01 7.44E-01 — 5.05E+04 8.77E+01
1.12E+05 1.65E+05 9.23E-02 8.59E-02 9.41E+03 3.44E+01
MVNS P3 3 5 — 5.27E+01 7.59E+01 — 1.77E+00 9.73E-02 — 8.19E+03 5.29E+01
1.14E+01 4.41E-01 9.11E-01 5.04E-02 6.35E+02 2.07E+01
P3 5 5 — 1.25E+04 1.76E+04 — 2.73E+00 1.43E-01 — 1.34E+04 8.68E+01
2.04E+03 3.03E+02 1.13E+00 1.49E-01 7.96E+02 4.14E+01
P3 7 5 — 6.55E+06 9.26E+06 — 3.75E+00 9.81E-02 — 1.87E+04 1.91E+02
1.05E+06 1.18E+05 1.45E+00 1.36E-01 2.24E+03 7.59E+01
20 E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023

329
T eed GPR o O line Da a-D i en MOPs
han hose o spa se GPR by an o de o abou 102and 103 o samples size o 10,000
and 50,000 espec i ely. The building ime also inc eases wi h he numbe o objec i e
unc ions o spa se GPR su oga es.
The o al numbe o samples u ilized o build TGPR-MO su oga es wi h he num-
be o i e a ions o six di e en p oblem ins ances wi h sample sizes o 2,000, 10,000,
and 50,000 is shown in Figu e 6. The solid line shows he mean numbe o samples used,
and he shaded egion deno es he 95% con idence in e al o he uns o each eed
GPR su oga e ( o each objec i e). The plo s ha e been ex ended in he i e a ion axis
o he maximum allowed i e a ions, Imax =N
10n. The i e a ion axis is b oken when Imax
is la ge o educe he wid h o he plo s. I can be obse ed ha he numbe o samples
u ilized con e ges be o e he maximum i e a ions and a ies wi h he objec i e. Ce ain
p oblem ins ances equi ed mo e samples du ing he building p ocess han o he s, es-
pecially due o he numbe o decision a iables and he cha ac e is ics o he p oblem.
I he ade-o egion is loca ed in a smalle egion o he decision space, he building
p ocess con e ges quickly and he e o e consumes ewe samples.
The quan i y o da a a ailable a he ade-o egion a ec s he accu acy and hy-
pe olume o he solu ions ob ained ha is a gene al challenge while sol ing o line
da a-d i en MOPs. The p oposed TGPR-MO su oga es spli he decision space in o
sub egions and app oxima e he unde lying objec i e unc ions a he ade-o egion.
I is bes sui able when he Pa e o se is loca ed in a smalle egion o he decision space.
When he Pa e o se is in a la ge egion, he p edic ion o TGPR-MO su oga es is om
mul iple lea node GPRs and has discon inui ies nea he spli s o he eg ession ees.
The e o e he app oxima ed Pa e o on has some discon inui ies compa ed o spa se
GPRs o ull GPRs. Fu he es s wi h he DTLZ (Deb e al., 2005) benchma k p oblems
a e gi en in he supplemen a y ma e ial.
5 Conclusions
This pape p oposed ailo ed su oga e models ha can be buil wi h a lowe compu a-
ional cos compa ed o spa se GPRs and ull GPRs o sol ing o line da a-d i en MOPs
when dealing wi h la ge da ase s. The p oposed TGPR-MO su oga es pe o med sig-
ni ican ly be e han spa se GPR su oga es in hype olume, RMSE, and compu a ion
ime o mos o he ins ances o he DBMOPP p oblems. The ull GPR su oga es ailed
o comple e he uns o la ge sample sizes due o memo y es ic ions. Thus, i can
be concluded ha he p oposed TGPR-MO su oga es a e bes sui ed o sol ing o line
da a-d i en MOPs wi h a la ge size da ase .
A GPR a a lea node o a ee app oxima es ce ain egions o he decision space.
This ea u e can be exploi ed while sol ing o line da a-d i en MOPs wi h p e e ences
om he decision make in an a p io i o in e ac i e ashion. The p o ided p e e ences
o objec i e unc ions can be u ilized o build GPRs a he lea nodes, and only he
solu ions ha ollow he p e e ences can be app oxima ed. De eloping an in e ac i e
amewo k u ilizing he TGPR-MO su oga es will be one o he u u e wo ks.
U ilizing he co ela ion be ween he objec i e unc ions using mul i- a ge eg es-
sion ees (Osojnik e al., 2018) and mul i-ou pu GPRs (Bo chani e al., 2015) is an in-
e es ing u u e esea ch di ec ion. Tes s and compa isons o he unce ain y p edic ion
p o ided by he p oposed TGPR-MO su oga es will be conduc ed o sol ing o line
da a-d i en MOPs. Selec ing a sui able ke nel o a gi en da ase is an ac i e esea ch
opic. In eg a ing au oma ic ke nel selec ion in o he TGPR-MO su oga es will be qui e
bene icial. One o he majo d awbacks o he TGPR-MO su oga es is he discon inui y
be ween he lea node GPRs. Tackling discon inui y in he p edic ion ( an S ein e al.,
E olu iona y Compu a ion Volume xx, Numbe x 21
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
Figu e 6: The median and 95% con idence in e al o he o al numbe o samples u i-
lized (o he sum o he samples a ailable a he lea nodes ha a e conside ed o build-
ing GPRs) wi h i e a ions o h ee di e en DBMOPP p oblem ins ances as displayed
acco ding o he ollowing o ma (N, sampling s a egy, p oblem, K,n). The i e a ion
axis is ex ended o he maximum possible i e a ion N
10n. The i e a ion axes o plo s 6c–6
a e b oken o accommoda e he plo s.
22 E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
T eed GPR o O line Da a-D i en MOPs
2015; Wang e al., 2017) a he pa i ion o he decision space (as p o ided by he ee)
will be a u u e ask. Fu he imp o emen s can be made in he way he ees pa i ion
he decision space. Ins ead o using a adi ional loss unc ion, a nonlinea ade-o
c i e ion can be o mula ed o spli he nodes. Such spli ing c i e ia will enable he
TGPR-MO su oga es o app oxima e he ade-o egion wi h ewe samples. Tes ing
he TGPR-MO su oga es o sol ing eal-li e o line da a-d i en MOPs will be a u u e
ask.
Acknowledgmen s
This esea ch was pa ly suppo ed by he Academy o Finland (g an numbe 311877
and 322221) and is ela ed o he hema ic esea ch a ea DEMO (Decision Analy ics u i-
lizing Causal Models and Mul iobjec i e Op imiza ion, h p://www.jyu. i/demo) o
he Uni e si y o Jy äskylä. The nume ical expe imen s we e pe o med on Mah i su-
pe compu e p o ided by CSC (h ps://www.csc. i).
Re e ences
Assael, J.-A. M., Wang, Z., Shah ia i, B., and de F ei as, N. (2014). He e oscedas ic eed Bayesian
op imisa ion. CoRR. Re ie ed om a Xi :1410.7172.
Bo chani, H., Va ando, G., Bielza, C., and La añaga, P. (2015). A su ey on mul i-ou pu eg es-
sion. WIREs Da a Mining and Knowledge Disco e y, 5(5):216–233. 10.1002/widm.1157
Chapman, W., Welch, W., Bowman, K., Sacks, J., and Walsh, J. (1994). A c ic sea ice a iabili y:
Model sensi i i ies and a mul idecadal simula ion. Jou nal o Geophysical Resea ch: Oceans,
99(C1):919–935. 10.1029/93JC02564
Cheng, R., Jin, Y., Olho e , M., and Sendho , B. (2016). A e e ence ec o guided e olu iona y
algo i hm o many-objec i e op imiza ion. IEEE T ansac ions on E olu iona y Compu a ion,
20(5):773–791. 10.1109/TEVC.2016.2519378
Chipman, H. A., Geo ge, E. I., and McCulloch, R. E. (1998). Bayesian CART model sea ch. Jou nal
o he Ame ican S a is ical Associa ion, 93(443):935–948. 10.1080/01621459.1998.10473750
Chugh, T., Sindhya, K., Hakanen, J., and Mie inen, K. (2019). A su ey on handling compu a-
ionally expensi e mul iobjec i e op imiza ion p oblems wi h e olu iona y algo i hms. So
Compu ing, 23:3137–3166. 10.1007/s00500-017-2965-0
Das, K., and S i as a a, A. N. (2010). Block-GP: Scalable Gaussian p ocess eg ession o mul i-
modal da a. In 2010 IEEE In e na ional Con e ence on Da a Mining, pp. 791–796.
Deb, K., and Jain, H. (2014). An e olu iona y many-objec i e op imiza ion algo i hm using
e e ence-poin -based nondomina ed so ing app oach, Pa I: Sol ing p oblems wi h box
cons ain s. IEEE T ansac ions on E olu iona y Compu a ion, 18(4):577–601. 10.1109/TEVC
.2013.2281535
Deb, K., Thiele, L., Laumanns, M., and Zi zle , E. (2005). Scalable es p oblems o e olu iona y
mul iobjec i e op imiza ion. In A. Ab aham, L. Jain, and R. Goldbe g (Eds.), E olu iona y
mul iobjec i e op imiza ion: Theo e ical ad ances and applica ions, pp. 105–145. Sp inge .
Emme ich, M., Giannakoglou, K., and Naujoks, B. (2006). Single- and mul iobjec i e e olu iona y
op imiza ion assis ed by Gaussian andom ield me amodels. IEEE T ansac ions on E olu ion-
a y Compu a ion, 10(4):421–439. 10.1109/TEVC.2005.859463
Fieldsend, J. E., Chugh, T., Allmendinge , R., and Mie inen, K. (2019). A ea u e ich dis ance-
based many-objec i e isualisable es p oblem gene a o . In P oceedings o he Gene ic and
E olu iona y Compu a ion Con e ence, pp. 541–549.
E olu iona y Compu a ion Volume xx, Numbe x 23
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023
329
A. Mazumda e al.
Fo es e , A., Sobes e , A., and Keane, A. (2008). Enginee ing design ia su oga e modelling. John
Wiley & Sons.
GPy (2012). GPy: A Gaussian p ocess amewo k in py hon. h p://gi hub.com/She ieldML/
GPy.
G amacy, R. B., and Lee, H.K.H. (2008). Bayesian eed Gaussian p ocess models wi h an applica-
ion o compu e modeling. Jou nal o he Ame ican S a is ical Associa ion, 103(483):1119–1130.
10.1198/016214508000000689
Hughes, E. J. (2001). E olu iona y mul i-objec i e anking wi h unce ain y and noise. In P oceed-
ings o E olu iona y Mul i-C i e ion Op imiza ion, pp. 329–343.
Jin, Y., Wang, H., Chugh, T., Guo, D., and Mie inen, K. (2019). Da a-d i en e olu iona y op imiza-
ion: An o e iew and case s udies. IEEE T ansac ions on E olu iona y Compu a ion, 23(3):442–
458. 10.1109/TEVC.2018.2869001
Jones, D. R., Schonlau, M., and Welch, W. J. (1998). E icien global op imiza ion o expensi e
black-box unc ions. Jou nal o Global Op imiza ion, 13:455–492. 10.1023/A:1008306431147
Kim, H.-M., Mallick, B. K., and Holmes, C. C. (2005). Analyzing nons a iona y spa ial da a using
piecewise Gaussian p ocesses. Jou nal o he Ame ican S a is ical Associa ion, 100(470):653–668.
10.1198/016214504000002014
Loh, W.-Y. (2011). Classi ica ion and eg ession ees. WIREs Da a Mining and Knowledge Disco e y,
1(1):14–23. 10.1002/widm.8
Mazumda , A., Chugh, T., Hakanen, J., and Mie inen, K. (2020). An in e ac i e amewo k o o -
line da a-d i en mul iobjec i e op imiza ion. In P oceedings o Bioinspi ed Op imiza ion Me h-
ods and Thei Applica ions, pp. 97–109.
Mazumda , A., Chugh, T., Hakanen, J., and Mie inen, K. (2022). P obabilis ic selec ion ap-
p oaches in decomposi ion-based e olu iona y algo i hms o o line da a-d i en mul-
iobjec i e op imiza ion. IEEE T ansac ions on E olu iona y Compu a ion, 26(5):1182–1191.
10.1109/TEVC.2022.3154231
Mazumda , A., Chugh, T., Mie inen, K., and López-Ibáñez, M. (2019). On dealing wi h unce ain-
ies om K iging models in o line da a-d i en e olu iona y mul iobjec i e op imiza ion. In
P oceedings o E olu iona y Mul i-C i e ion Op imiza ion, pp. 463–474.
Misi ano, G., Saini, B. S., A sa , B., Sha azipu , B., and Mie inen, K. (2021). DESDEO: The mod-
ula and open sou ce amewo k o in e ac i e mul iobjec i e op imiza ion. IEEE Access,
9:148277–148295. 10.1109/ACCESS.2021.3123825
Osojnik, A., Pano , P., and Dže oski, S. (2018). T ee-based me hods o online mul i-
a ge eg ession. Jou nal o In elligen In o ma ion Sys ems, 50:315–339. 10.1007/s10844-017
-0462-7
Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., Blondel, M., e
al. (2011). Sciki -lea n: Machine lea ning in Py hon. Jou nal o Machine Lea ning Resea ch,
12:2825–2830.
Raha , A.A.M., Wang, C., E e son, R. M., and Fieldsend, J. E. (2018). Da a-d i en mul i-objec i e
op imisa ion o coal- i ed boile combus ion sys ems. Applied Ene gy, 229:446–458. 10.1016/
j.apene gy.2018.07.101
Rasmussen, C. E., and Williams, C.K.I. (2006). Gaussian p ocesses o machine lea ning. MIT P ess.
Shah ia i, B., Swe sky, K., Wang, Z., Adams, R. P., and de F ei as, N. (2016). Taking he human
ou o he loop: A e iew o Bayesian op imiza ion. P oceedings o he IEEE, 104(1):148–175.
10.1109/JPROC.2015.2494218
24 E olu iona y Compu a ion Volume xx, Numbe x
Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023