Treed Gaussian Process Regression for Solving Offline Data-Driven Continuous Multiobjective Optimization Problems

This is a sel -a chi ed e sion o an o iginal a icle. This e sion may di e om he o iginal in pagina ion and ypog aphic de ails. Au ho (s): Ti le: Yea : Ve sion: Copy igh : Righ s: Righ s u l: Please ci e he o iginal e sion: CC BY 4.0 h ps://c ea i ecommons.o g/licenses/by/4.0/ T eed Gaussian P ocess Reg ession o Sol ing O line Da a-D i en Con inuous Mul iobjec i e Op imiza ion P oblems © MIT P ess 2023 Accep ed e sion (Final d a ) Mazumda , A anu; López-Ibáñez, Manuel; Chugh, Tinkle; Hakanen, Jussi; Mie inen, Kaisa Mazumda , A., López-Ibáñez, M., Chugh, T., Hakanen, J., & Mie inen, K. (2023). T eed Gaussian P ocess Reg ession o Sol ing O line Da a-D i en Con inuous Mul iobjec i e Op imiza ion P oblems. E olu iona y Compu a ion, 31(4), 375-399. h ps://doi.o g/10.1162/e co_a_00329 2023 329 T eed Gaussian P ocess Reg ession o Sol ing O line Da a-D i en Con inuous Mul iobjec i e Op imiza ion P oblems A anu Mazumda [email p o ec ed] Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland Manuel López-Ibáñez manuel.lopez-ibanez@manches e .ac.uk Alliance Manches e Business School, Uni e si y o Manches e , UK Tinkle Chugh .chugh@exe e .ac.uk Depa men o Compu e Science, Uni e si y o Exe e , UK Jussi Hakanen [email p o ec ed] Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland Kaisa Mie inen [email p o ec ed] Uni e si y o Jy askyla, Facul y o In o ma ion Technology, Finland h ps://doi.o g/10.1162/e co_a_00329 Abs ac Fo o line da a-d i en mul iobjec i e op imiza ion p oblems (MOPs), no new da a is a ailable du ing he op imiza ion p ocess. App oxima ion models (o su oga es) a e i s buil using he p o ided o line da a, and an op imize , o example, a mul iob- jec i e e olu iona y algo i hm, can hen be u ilized o ind Pa e o op imal solu ions o he p oblem wi h su oga es as objec i e unc ions. In con as o online da a-d i en MOPs, hese su oga es canno be upda ed wi h new da a and, hence, he app oxi- ma ion accu acy canno be imp o ed by conside ing new da a du ing he op imiza- ion p ocess. Gaussian p ocess eg ession (GPR) models a e widely used as su oga es because o hei abili y o p o ide unce ain y in o ma ion. Howe e , building GPRs becomes compu a ionally expensi e when he size o he da ase is la ge. Using spa se GPRs educes he compu a ional cos o building he su oga es. Howe e , spa se GPRs a e no ailo ed o sol e o line da a-d i en MOPs, whe e good accu acy o he su o- ga es is needed nea Pa e o op imal solu ions. T eed GPR (TGPR-MO) su oga es o o line da a-d i en MOPs wi h con inuous decision a iables a e p oposed in his pa- pe . The p oposed su oga es i s spli he decision space in o sub egions using e- g ession ees and build GPRs sequen ially in egions close o Pa e o op imal solu ions in he decision space o accu a ely app oxima e adeo s be ween he objec i e unc- ions. TGPR-MO su oga es a e compu a ionally inexpensi e because GPRs a e buil only in a smalle egion o he decision space u ilizing a subse o he da a. The TGPR- MO su oga es we e es ed on dis ance-based isualizable p oblems wi h a ious da a sizes, sampling s a egies, numbe s o objec i e unc ions, and decision a iables. Ex- pe imen al esul s showed ha he TGPR-MO su oga es a e compu a ionally cheape and can handle da ase s o la ge size. Fu he mo e, TGPR-MO su oga es p oduced solu ions close o Pa e o op imal solu ions compa ed o ull GPRs and spa se GPRs. Keywo ds Gaussian p ocesses, K iging, eg ession ees, me amodelling, su oga e, Pa e o op imali y. Manusc ip ecei ed: 13 Ap il 2022; e ised: 6 No embe 2022, 23 Feb ua y 2023, 23 Ma ch 2023; accep ed: 31 Ma ch 2023. © 2023 Massachuse s Ins i u e o Technology. Published unde a C ea i e Commons A ibu ion 4.0 In e na ional (CC BY 4.0) license. E olu iona y Compu a ion xx(x): 1–25 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. 1 In oduc ion A mul iobjec i e op imiza ion p oblem (MOP) consis s o wo o mo e con lic ing ob- jec i e unc ions ha a e op imized simul aneously. The solu ions o an MOP a e called Pa e o op imal when he alue o any objec i e unc ion canno be u he imp o ed wi hou deg ading some o he o he s; hus, adeo s exis among he objec i e unc- ions. No all eal-wo ld MOPs ha e analy ical unc ions o simula ion models a ail- able. Ins ead, he MOP may need o be o mula ed by u ilizing he da a acqui ed om a phenomenon, o example, eal-li e p ocesses, physical expe imen s, and senso s. To sol e his MOP, i s he unde lying objec i e unc ions a e app oxima ed by i ing su o- ga es (also known as me amodels) on he da a a ailable. Nex , a mul iobjec i e e o- lu iona y algo i hm (MOEA) can be used o app oxima e he se o Pa e o op imal solu ions by conside ing he su oga es as objec i e unc ions. Da a-d i en op imiza ion p oblems a e classi ied in o online and o line ones (Jin e al., 2019). Fo online da a-d i en op imiza ion, new da a can be acqui ed du ing he op imiza ion p ocess, o example, by conduc ing u he expe imen s o simula ions o upda e he su oga es (Shah ia i e al., 2016). In con as , o o line da a-d i en op i- miza ion p oblems, no new da a can be acqui ed, and he op imiza ion has o p oceed by using only he exis ing da a (Wang e al., 2019). Ideally, he unde lying objec i e alues o he solu ions should be be e han he objec i e alues in he p o ided da ase . All he a ailable da a should be u ilized o build su oga es wi h good app oxima ion while sol ing o line da a-d i en MOPs. Howe e , when he size o he da a se is la ge, o ex- ample, in Wang e al. (2016) and Wang and Jin (2020), building ce ain su oga es, such as Gaussian p ocesses, can become compu a ionally expensi e. Because he su oga es canno be upda ed wi h new da a while sol ing o line da a-d i en MOPs, he app ox- ima ion accu acy o he su oga es di ec ly in luences he closeness o he solu ions o he Pa e o op imal se . Mos o he p e ious wo ks on o line da a-d i en op imiza ion, such as Wang e al. (2019, 2016), Wang and Jin (2020), and Yang e al. (2020), conside ed di e en su oga e modelling echniques o sol ing o line da a-d i en op imiza ion p oblems. Howe e , hese wo ks did no conside he adeo s be ween he unde lying objec i e unc ions while building he su oga es. O he de eloped app oaches, o ex- ample, Mazumda e al. (2019, 2022), Raha e al. (2018), and Hughes (2001), elied on he unce ain y in he p edic ion o Gaussian p ocesses o imp o e he quali y o solu ions. The p e ious wo ks on o line da a-d i en op imiza ion dealing wi h la ge da ase s, o example, Wang e al. (2016) and Wang and Jin (2020) did no quan i y he unce ain y, which is a c i ical componen in sol ing o line da a-d i en MOPs, as shown in Mazum- da e al. (2019, 2020, 2022). This pape p oposes an app oach o build compu a ionally cheap su oga es wi h a high app oxima ion accu acy a he egion a ound he Pa e o op imal se . The su oga es use a combina ion o eg ession ees and Gaussian p ocess eg ession models and p o ide unce ain y in he p edic ion. The p oposed su oga es a e ailo ed owa d sol ing o line da a-d i en MOPs and a e capable o handling la ge da ase s ( es ed o a maximum size o 50,000). To s udy o line da a-d i en MOPs, da a is sampled om benchma k p oblems o which he Pa e o op imal se is known. Knowledge o he Pa e o op imal se enables quali y compa isons o he solu ions ob ained by di e en op imiza ion app oaches and su oga es. While sol ing an o line da a-d i en MOP, gene ally wo ypes o in- dica o s a e used o quan i y he quali y o he solu ions (Mazumda e al., 2019, 2020). The app oxima ion accu acy is measu ed in oo mean squa ed e o (RMSE) be ween he app oxima ed objec i e alues o he su oga es and he unde lying objec i e al- ues. In addi ion, he hype olume (Zi zle and Thiele, 1998) indica o is used o measu e 2E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs how close he app oxima ed Pa e o on (a e e alua ing wi h he unde lying objec i e unc ions) is o he Pa e o on o he unde lying MOP. K iging, o Gaussian p ocess eg ession (GPR), (Fo es e e al., 2008) is a popu- la choice o su oga e (Rasmussen and Williams, 2006) o sol ing o line p oblems because i also p o ides in o ma ion abou he unce ain y in he app oxima ion. P e i- ous wo ks (Hughes, 2001; Raha e al., 2018; Mazumda e al., 2019, 2022) ha e shown ha u ilizing he unce ain y p edic ion om GPR su oga es p oduces solu ions wi h imp o ed accu acy and hype olume. The unce ain y p edic ion om GPR makes i an a ac i e choice o su oga es o sol ing o line da a-d i en MOPs. Mos o he ap- p oaches o sol ing o line da a-d i en MOPs build a global GPR su oga e o each objec i e using all he p o ided da a. Howe e , ull GPRs ha e high compu a ional and memo y complexi y when he size o he da ase (o sample size) is la ge. An al e na- i e o ull GPRs is o use spa se GPRs (Snelson and Ghah amani, 2005; Ti sias, 2009), which build app oxima ion models wi h a small subse o da a called suppo o induc- ing poin s. In Ti sias (2009), a a ia ional me hod was p oposed o selec he inducing poin s by g adien descen o g eedy sea ch algo i hm. Howe e , his me hod becomes compu a ionally expensi e when he da ase is la ge. On he o he hand, using ewe poin s o building he su oga es educes he app oxima ion accu acy o he su oga es compa ed o ull GPRs (Ti sias, 2009). O he wo ks, such as Emme ich e al. (2006), p o- posed building GPR using only K-nea es neighbo samples om he poin ha is o be p edic ed. While pe o ming mul iobjec i e op imiza ion using su oga es, i is desi able o ob ain a good app oxima ion accu acy a he egion a ound he Pa e o se , which is e e ed o as he ade-o egion in his pape . The accu acy o he su oga es in o he e- gions o he decision space (excluding he ade-o egion) is no o u mos impo ance. A possible app oach o educe he compu a ional cos is o build local GPR su oga es o he unde lying objec i e unc ions exclusi ely in he ade-o egion. Using local GPRs educes he o e all compu a ional cos o building GPRs while p o iding a good app oxima ion accu acy a he ade-o egion. P e ious wo ks (Kim e al., 2005; Das and S i as a a, 2010; an S ein e al., 2015) pa i ioned he decision space in o egions and i ed GPRs in each egion. Pa i ioning he decision space is ela i ely inexpensi e, and each GPR u ilizes smalle se s o da a. This concep was ex ended by G amacy and Lee (2008) o i GPRs in each egion o lea nodes pa i ioned by a Bayesian eed model p e iously p oposed by Chipman e al. (1998). In Assael e al. (2014), eed GPRs we e p oposed o deal wi h cases whe e he noise is di e en o di e en samples. All he p e ious wo ks build GPRs in all he egions o he decision space. These ypes o GPRs become compu a ionally expensi e, depending on he numbe o egions and amoun o da a in each egion. Mo eo e , hese su oga es do no conside he ade-o egion o he o line da a-d i en MOP du ing he building p ocess. This pape p oposes eed GPR su oga es o mul iobjec i e op imiza ion (TGPR- MO) which ha e a high accu acy a ound he ade-o egion and a e ailo ed o sol ing o line da a-d i en MOPs wi h con inuous decision a iables. Fi s , compu a ionally in- expensi e eg ession ee su oga es a e buil using he p o ided da ase . The eg ession ee su oga es p o ide a less accu a e app oxima ion o he unde lying objec i e unc- ions (compa ed o GPRs) (Loh, 2011) and spli he decision space in o egions. A egion is de ined as he pa o he decision space ha is enclosed by he lea node o he i ed eg ession ees. The app oxima ion accu acy o he p edic ion by he lea node is he accu acy o he egion. Nex , an MOEA is execu ed conside ing he eg ession ees as objec i e unc ions. The solu ions o he MOEA a e no e y accu a e bu p o ide he E olu iona y Compu a ion Volume xx, Numbe x 3 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. app oxima e loca ion o he ade-o egion. A e a ce ain numbe o gene a ions o he MOEA, he accu acy o he ees’ p edic ion is imp o ed by building GPRs a lea nodes wi hin he ade-o egion. The GPRs buil imp o e he accu acy o he solu ions in he egion o he decision space co esponding o he lea node hey eplace. The inal su oga es consis o eg ession ees wi h GPRs a a ew lea nodes p o iding accu a e app oxima ions exclusi ely in he ade-o egion. The TGPR-MO su oga es we e es ed on se e al ins ances o dis ance-based i- sualizable es p oblems (DBMOPP) wi h di e en numbe s o decision a iables and objec i e unc ions. Va ious sizes o he ini ial da ase wi h di e en sampling s a e- gies we e used in he es s. Nume ical expe imen s showed ha TGPR-MO su oga es signi ican ly educed he building ime o su oga es when using la ge size da a while sol ing o line da a-d i en MOPs. TGPR-MO su oga es also p oduced solu ions wi h a be e hype olume compa ed o spa se GPR su oga es. The es o he pape is a anged as ollows. The backg ound o o line da a-d i en MOPs, GPRs, and eg ession ees is gi en in Sec ion 2. The p oposed TGPR-MO su o- ga es a e de ailed in Sec ion 3. The expe imen al esul s consis ing o a compa ison o TGPR-MO su oga es wi h o he su oga es a e p esen ed in Sec ion 4. Sec ion 5 con- cludes and discusses he u u e esea ch pe spec i es. 2 Backg ound Fo an o line da a-d i en MOP, he s a ing poin o sol ing he p oblem is (p ecol- lec ed) da a. In his pape , we assume ha he a ailable da a is he ou pu o a p ocess o phenomenon. The p ocess o gene a ing he da a is e e ed o as he unde lying ob- jec i e unc ions o an MOP. The unde lying MOP ha has o be sol ed is conside ed o be o he ollowing o m: minimize ( 1(x),..., K(x)) subjec o x∈,(1) whe e K≥2 is he numbe o objec i e unc ions and is he easible egion in he decision space n. Fo a easible decision ec o x(consis ing o ndecision a iables), he co esponding objec i e ec o is (x)=( 1(x),..., K(x)), whe e 1(x),..., K(x) a e he objec i e ( unc ion) alues. A solu ion x∈domina es ano he solu ion x∈i i(x)≤ i(x) o all i= 1,...,K and i(x)< i(x) o a leas one i=1,...,K. I a solu ion o an MOP is no domina ed by any o he easible solu ions, i is called nondomina ed. Sol ing an MOP using a mul iobjec i e op imiza ion algo i hm, o example, an MOEA ypically p o- duces a se o mu ually nondomina ed solu ions. The solu ions o he op imiza ion p oblem de ined in Equa ion (1) ha a e nondomina ed in he whole se a e called Pa e o op imal solu ions. A gene ic way o sol e an o line da a-d i en MOP wi h an MOEA as he op imize is shown in Figu e 1. As explained in Jin e al. (2019) and Wang e al. (2019), he solu ion p ocess can be di ided in o h ee componen s: (a) da a collec ion, (b) o mula ing he MOP and su oga e building, and (c) op imiza ion wi h su oga es. The op imiza ion p ocess s a s wi h an o line da ase consis ing o Nsamples. A sample consis s o a decision ec o xand i s co esponding objec i e ec o (x)asa uple o wo ma ices: (X, Y ) whe e X∈ N×nand Y∈ N×K. Each ow in Xand Yis a decision ec o and i s co esponding objec i e ec o , espec i ely. Nex , su oga es a e buil using all o a subse o he da a. The su oga es a e conside ed as objec i e 4E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs Figu e 1: Flowcha o a gene ic o line da a-d i en mul iobjec i e op imiza ion app oach. unc ions by an MOEA o sol e he o line da a-d i en MOP. Fo simplici y o desc ibing a su oga e model o a single objec i e i,le yi∈ N×1be he ec o o objec i e alues i(x) o all decision ec o s x∈X. 2.1 Gaussian P ocess Reg ession In his wo k, a su oga e model is buil o each objec i e unc ion. Fo simplici y in e minologies, yis conside ed he e ins ead o yi o in oduce he Gaussian p ocess e- g ession (GPR) and eg ession ees. GPRs ha e been widely used in su oga e-assis ed op imiza ion and ime-se ies analysis (Jin e al., 2019; Chugh e al., 2019). The majo ad an age o using a GPR is i s abili y o p o ide he dis ibu ion abou i s p edic ion (o he unce ain y). A GPR is a mul i a ia e no mal dis ibu ion wi h a mean μand a co a iance ma ix C: y∼N(μ,C).(2) Fo simplici y in calcula ions, a mean o ze o is conside ed wi hou loss o gene ali y. The co a iance ma ix Cuses a co a iance (o ke nel) unc ion o de ine co ela ion be- ween wo samples, xand x. In his wo k, we use he Ma é n 5/2 ke nel because i does no make he co a iance unc ions un ealis ically smoo h compa ed o o he ke - nels such as he Gaussian (o RBF) ke nel. The e o e, i is mo e app op ia e o sol ing p ac ical op imisa ion p oblems (Snoek e al., 2012). The ke nel is de ined as: κ(x,x,)=σ2 ⎛ ⎝1+√5 n  j=1 j+5 3 n  j=1 2 j⎞ ⎠exp ⎛ ⎝−√5 n  j=1 j⎞ ⎠+σ2 δxx whe e j=|xj−x j| ljand xjand x ja e he j h componen s o he decision ec o o x and x,=(σ ,l 1,...,l n,σ ) is he se o pa ame e s in he GPR model and δxxis he K onecke del a unc ion. The no a ion |xj−x j| ep esen s he Euclidean dis ance be- ween xjand x j. The pa ame e s σ ,ljand σ ep esen he ampli ude, leng h scale o he j h a iable, and noise in he da a, espec i ely. Fo mo e de ails on he signi icance o hese pa ame e s, see Rasmussen and Williams (2006). To build a GPR model, he pa ame e s men ioned abo e can be es ima ed by max- imizing he ma ginal likelihood unc ion: p(y|X, )=1 √|2πC|exp −1 2yC−1y.(3) A e es ima ing he pa ame e s, he model de ined in Equa ion (2) can be used o he pos e io p edic i e dis ibu ion a a new decision ec o x∗. The GPR model p o ides a pos e io p edic i e dis ibu ion which is also Gaussian: p(y∗|x∗,X,y,)=Nκ(x∗,X,)C−1y,κ(x∗,x∗,)−κ(x∗,X,)C−1κ(X, x∗,).(4) E olu iona y Compu a ion Volume xx, Numbe x 5 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. The pos e io mean in he equa ion abo e is κ(x∗,X,)C−1y, and he a iance ep e- sen ing he unce ain y is κ(x∗,x∗,)−κ(x∗,X,)C−1κ(X, x∗,). 2.2 Reg ession T ees A eg ession ee ecu si ely pa i ions he decision space such ha he p o ided sam- ples wi h simila objec i e unc ion alues a e g ouped oge he (Loh, 2011). Each node φo he ee con ains da a Qφ=(Xφ,yφ)wi hNφsamples, whe e each ow o Xφ∈ Nφ×nis a decision ec o xiand each ow o yφ∈ Nφ×1is i s co esponding objec- i e alue yi,i=1,...,Nφ.Nodeφis spli in o nodes φle and φ igh acco ding o pa am- e e θ=(j, ), which speci ies a j h decision a iable (1 ≤j≤n) and a h eshold ,by pa i ioning Qφin o wo disjoin subse s Qφle θ={(Xφle ,yφle )|(xi,y i)∈Qφ∧xij ≤ } and Qφ igh θ=Qφ Qφle θ. Tha is, spli pa ame e θpa i ions he samples ( ows) in Qφ acco ding o he alue o a iable xjo each decision ec o x∈Xφ. Gi en a node φand a spli pa ame e θ, he quali y o he spli o he o al loss is calcula ed as: G(Qφ,θ)=Nφle NφH(Qφle θ)+Nφ igh NφH(Qφ igh θ),(5) whe e Nφle and Nφ igh a e he numbe o samples in Qφle θand Qφ igh θ, espec i ely, and H(Qφ) is a loss unc ion. Fo ins ance, he loss unc ion o mean-squa ed e o is: H(Qφ)=1 Nφ yi∈Qφ (yi−¯yφ)2,(6) whe e ¯yφis he mean objec i e alue in Qφ. An op imal spli θ∗o a node φcan be ound by minimizing he loss unc ion de ined in Equa ion (5) using a single objec i e op imiza ion algo i hm. In he p oposed TGPR-MO su oga es, he spli ing p ocess is ecu sed o bo h Qφle θ∗and Qφ igh θ∗un il spli ing a node would p oduce a lea node wi h less han a p ede ined minimum numbe o samples, Nmin. Fo p edic ing any gi en decision a iable alue, he eg ession ee is a e sed o he espec i e lea node. The p edic ion o he ee a he lea node l, which con ains aining subse Qlwi h Nlnumbe o samples, is ¯yl=1 Nlyi∈Qlyi. 3 T eed GPRs o Mul iobjec i e Op imiza ion (TGPR-MO) The ime complexi y o a ull GPR is cubic, which is polynomial. A sui able al e na i e o educe he compu a ional cos is o spli he da ase and build GPRs exclusi ely in he ade-o egion. The p oposed TGPR-MO su oga es a e based on his modelling ap- p oach. The ollowing subsec ions desc ibe he building p ocess o he p oposed TGPR- MO su oga es. A de ailed accu acy and complexi y analysis and compa ison wi h ull GPRs and spa se GPRs a e also p o ided. 3.1 Building he TGPR-MO Su oga es In a gene alized eed GPR su oga e, i s a eg ession ee model is buil using he p o- ided da a as desc ibed in Sec ion 2.2. The spli ing a he nodes is done by minimizing he o al loss unc ion (Equa ion 5). The subse o da a a he l h lea node is Ql, whe e l=1,...,Land Lis he o al numbe o lea nodes in he eg ession ee buil . A GPR is i ed a e e y lea node o he eg ession ee using he da a Ql=(Xl,yl). The GPRs a e buil in a simila ashion as desc ibed in Sec ion 2.1 by maximizing he ma ginal 6E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs likelihood a he lea node l, as de ined in Equa ion (3). The pos e io p edic i e dis i- bu ion o he l h GPR is py∗|x∗,X l,yl,, as de ined in Equa ion (4). Building GPRs wi h smalle subse s o da a educes he o e all cos o building su oga es compa ed o building a GPR wi h he en i e da ase . Howe e , building GPRs a all he lea nodes becomes expensi e when he e a e oo many o hem, and he da a subse a each lea node is la ge. Mo eo e , o sol ing an o line da a-d i en MOP, an accu a e app oxi- ma ion o he global landscape o he unde lying objec i e unc ions is no equi ed. A good app oxima ion o he local landscape nea he ade-o egion is su icien . While pe o ming mul iobjec i e op imiza ion using su oga es, he app oxima- ion accu acy in he ade-o egion is c ucial and di ec ly in luences he quali y o he app oxima ed Pa e o op imal solu ions. Hence, o ob ain be e quali y solu ions, i is desi able o ha e accu a e app oxima ions in he ade-o egion. While building he eg ession ees, he spli s a each node di ide he decision space. The lea nodes o he eg ession ees a e he smalles egions he decision space is spli in o. In addi ion, he ini ial app oxima ion o eg ession ee su oga es ( hough no highly accu a e) p o- ides in o ma ion abou he ade-o s be ween he objec i e unc ions. A e building he eg ession ees wi h all he p o ided da a, an MOEA is un conside ing hem as objec i e unc ions. The solu ions ound by he MOEA a e no accu a e, bu hey p o- ide an app oxima ion o he Pa e o on . The accu acy o he ade-o egion is he app oxima ion accu acy o he egion enclosed by he lea nodes ha p edic s he ap- p oxima ed Pa e o on . La e , local GPRs a e buil exclusi ely in he lea nodes ep e- sen ing he ade-o egion and achie e an accu a e app oxima ion only in he neigh- bo hood o he Pa e o se . Nex , he algo i hm o build he eed GPR su oga es o mul iobjec i e op imiza ion (TGPR-MO) ailo ed o sol e o line da a-d i en MOPs is in oduced. Algo i hm 1 s a s wi h a da ase con aining Nsamples wi h Kobjec i e unc ions and ndecision a iables. Fi s , K eg ession ees (one pe objec i e unc ion) a e buil wi h all he p o ided da a. Jus he pa ame e Nmin is adjus ed, and he dep h o he ees is no con olled. A e building he eg ession ees, he e can be a maximum o N Nmin lea nodes. Ini ially, he e a e no GPRs p esen a he lea nodes, and he p edic ions a e om he eg ession ees (i.e., ¯yl). Nex , a popula ion is ini ialized and an MOEA is execu ed ha i e a i ely builds GPRs a speci ic lea nodes. Each i e a ion consis s o unning he MOEA o Gmax gene a ions ( ha is a p ede ined pa ame e ) and building a GPR a one lea node in e e y ee. In e e y gene a ion, o sp ing indi iduals a e p oduced using c osso e and mu a ion ope a o s and e alua ed using he eed GPRs. Selec ion o indi iduals is hen pe o med using MOEA-speci ic selec ion c i e ion, and he p ocess is epea ed o Gmax gene a ions wi hin an i e a ion. The inne wo kings o he p oposed algo i hm a e exempli ied in wo-dimensional decision spaces in Sec ion 1 and Figu e 1 o he supplemen a y ma e ial. A e he MOEA comple es Gmax gene a ions, he app oxima ion e o s o he solu- ions ( o each objec i e) a e compa ed. The compa ison is done by i s calcula ing he loss unc ion alue, he e he mean squa ed e o , as de ined in Equa ion (6), o he lea node p edic ing he objec i e alues o he solu ions ound by he MOEA. Fo he eed GPR su oga e co esponding o he j h objec i e, le lj 1,...,lj sdeno e he lea nodes con aining he ssolu ions ound by he MOEA and le H(Qlj 1),...,H(Qlj s) deno e he loss unc ion alue o hose lea nodes, hen one GRP is buil a he lea node i∗wi h he maximum loss alue, ha is, i∗=a g maxi=1,...,sH(Qlj i), using i s subse o samples Qlj i∗. I mul iple solu ions all wi hin he same lea node, he loss alue is calcula ed once and only one GPR is buil o ha lea node. The p ocess o building GPRs is epea ed o E olu iona y Compu a ion Volume xx, Numbe x 7 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. 8E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs o he op imiza ion p ocess. In addi ion, he complexi y o hese p oblems can be con- olled by he ea u es (e.g., a ying densi y, numbe o local on s, dominance esis- ance egions, numbe s o objec i e unc ions, and decision a iables). These ad an- ages p o ide mo e con ol o e he p oblems and make hem adjus able o e lec he needs o eal-wo ld op imiza ion p oblems. These ea u es o DBMOPP es p oblems p o e o be mo e ad an ageous compa ed o DTLZ (Deb e al., 2005) benchma k p ob- lems. A de ailed desc ip ion o he expe imen se ings, esul s, and in-dep h analysis a e p o ided in he ollowing subsec ions. 4.1 Expe imen al Se up All he app oaches o sol ing he o line da a-d i en MOP we e coded in Py hon u i- lizing he DESDEO amewo k (Misi ano e al., 2021) (h ps://desdeo.i .jyu. i).1The expe imen s we e execu ed on one node o an HPC clus e equipped wi h AMD Rome CPUs, each node ha ing 128 co es unning a 2.6 GHz, wi h 256 GiB o memo y. Each indi idual un was execu ed on one CPU co e and alloca ed a maximum memo y us- age o 2 GiB. The eg ession ee was buil using he sklea n Py hon package (Ped egosa e al., 2011). Fo building he GPRs a he lea nodes, he GPy (GPy, 2012) Py hon package was used. 4.1.1 Benchma k P oblems Fou DBMOPP p oblems P1–4, as shown in Table 1, we e used. The p oblem ins ances and da a we e gene a ed by he code p o ided by Fieldsend e al. (2019). All combi- na ions o numbe s o objec i e unc ions (K∈{3,5,7}) and numbe s o decision a i- ables (n∈{2,5,7,10}) we e used o he es s. The o al numbe o p oblem ins ances es ed was 48, and e e y p oblem ins ance ep esen ed a di e en ype o p oblem cha ac e is ic. 4.1.2 Da ase s Fo gene a ing he da a, La in hype cube sampling (LHS) and mul i a ia e no mal sam- pling (MVNS) (Fo es e e al., 2008) we e used. In MVNS sampling, he objec i e unc- ions we e conside ed independen wi h mean a he mid-poin o he decision space, ha is, ze o o DBMOPP p oblems. The a iance o he sampling dis ibu ion was se o 0.1 o all he objec i e unc ions. Using MVNS sampling es s he abili y o he su oga e models and op imiza ion algo i hm o handle biased (o skewed) da ase s (Mazumda e al., 2022). Sample sizes o ini ial da a (N∈{2000,10000,50000}) we e chosen o he es s. A o al o 288 cases we e es ed, and 31 se s o da a wi h a andom seed we e gene a ed o each es case. Each o hese da ase s was he s a ing poin o he h ee di e en su oga e models ha we e es ed. These indi idual uns we e independen , and he esul s we e used o compa e he su oga es’ pe o mances s a is ically. 4.1.3 Se ings o TGPR-MO Su oga es The MOEA o building TGPR-MO su oga es was RVEA (Cheng e al., 2016), a decomposi ion-based MOEA (Zhang and Li, 2007; Deb and Jain, 2014). Decomposi ion- based MOEAs ha e shown o be e ec i e in sol ing o line da a-d i en MOPs wi h mo e han h ee objec i e unc ions. Howe e , he su oga e is no limi ed o RVEA and one can use any MOEA o choice. The pa ame e s o RVEA we e kep he same as sug- ges ed by Cheng e al. (2016). The pa ame e Gmax was se o 50. The maximum numbe o i e a ions was se o Imax =N Nmin =N 10n. Such a alue o Imax was chosen conside ing 1Sou ce code a ailable a h ps://gi hub.com/indus ial-op imiza ion-g oup/T eedGP_MOEA E olu iona y Compu a ion Volume xx, Numbe x 15 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. one GPR is buil a a lea node o all he ees in each i e a ion (as he maximum numbe o lea nodes is N Nmin ). The loss unc ion used o building he ees was MSE as men ioned in Equa ion (6) wi h Nmin =10n. These pa ame e s we e chosen because su icien de- sign poin s a e necessa y o building GPRs a he lea es. Based on ecommenda ions om he GPR li e a u e (Chapman e al., 1994; Jones e al., 1998), a leas 10npoin s a e equi ed a each lea node. The ke nel used o building he GPRs a he lea nodes was Má e n 5/2 wi h au oma ic ele ance de e mina ion enabled. 4.1.4 O he Su oga es Tes ed The p oposed TGPR-MO su oga es we e compa ed wi h spa se GPR and ull GPR su oga es. The same GPy package and ke nel we e used o building bo h o hese su oga es as o he TGPR-MO su oga es. Fo spa se GPR su oga es, he numbe o induc ion poin s was se o M=10n. He e, he o e all p ocess o sol ing an o line da a- d i en MOP wi h a speci ic su oga e is e e ed o as an app oach o simplici y. I should be no ed ha andom o es was no conside ed in he es s as he ocus o his pape is on GPR su oga es. 4.1.5 Pa ame e Se ings o MOEA ( o Sol ing he O line Da a-D i en MOP) Fo sol ing he o line da a-d i en MOP wi h su oga es as objec i e unc ions, RVEA wi h he same de aul pa ame e se ings was used. The e mina ion c i e ion o RVEA was 1,000 gene a ions, which was su icien o all he app oaches es ed o con e ge. To gene a e a uni o m Pa e o on , he e e ence ec o s a e ea anged o adap ed a e a ce ain numbe o gene a ions in RVEA. The e e ence ec o adap a ion a e was se o once e e y 100 gene a ions. 4.1.6 Pe o mance Indica o s The quali y o he solu ions ob ained by he MOEA was measu ed in e ms o hei hype olume (HV) a e e alua ing hem wi h he unde lying objec i e unc ions o he es p oblems. Fo compu ing he HV indica o , he e e ence poin o y∗=(y∗ 1,y∗ 2, ...,y∗ K) in he objec i e space was chosen, such ha i is domina ed by all he solu ions. In his pape , he e e ence poin used was y∗=(2√K,2√K,...,2√K) because his poin is always domina ed in DBMOPP p oblems. The mul i a ia e RMSE o he ob- jec i e alues o he solu ions ob ained by he MOEA wi h hei espec i e unde lying objec i e alues was used o measu e he accu acy. The mul i a ia e RMSE is he Eu- clidean dis ance be ween he app oxima ed and e alua ed unde lying objec i e unc- ion alues o he solu ions and is gi en by 1 ss i=1K j=1(ˆ j,i − j,i )2, whe e sis he numbe o solu ions, ˆ j,i and j,i a e he app oxima ed and he unde lying objec i e alue, espec i ely, o he i h solu ion and j h objec i e. Finally, he compu a ional cos o he a ious me hods was measu ed as he ime aken in seconds o build he su oga es. In his pape , mul i a ia e RMSE is e e ed o as RMSE o simplici y. The hype - olume and RMSE indica o s a e used he e o benchma k he pe o mance o TGPR- MO su oga es o sol ing an o line da a-d i en MOP. E alua ing he solu ions wi h he unde lying objec i e unc ions in an o line da a-d i en MOP may no be possible in eal li e. 4.2 Resul s and Discussions While unning he expe imen s, i was obse ed ha building ull GPR su oga es wi h 10,000 and 50,000 sample sizes consis en ly ga e ou -o -memo y e o s. The s o age 16 E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs complexi y o ull GPRs is O(KN2), and each elemen o he a ay is a 64-bi loa . Hence, he memo y equi emen o h ee objec i e unc ions and sample sizes o 10,000 and 50,000 is 2.3 GiB and 56 GiB, espec i ely. Thus, building GPR su oga es using all he p o ided o line da a will become almos impossible wi h eadily a ailable compu ing esou ces when he sample size is la ge. Hence, he esul s o ull GPR su oga es a e no included o sample sizes o 10,000 and 50,000. A pai wise Wilcoxon wo- ailed signi icance es was conduc ed o compa e he pe o mance o he di e en app oaches. The calcula ed p- alues we e Bon e onni co - ec ed, and α=0.05 was conside ed o ejec ing he null hypo hesis (an app oach is no signi ican ly be e o wo se han ano he app oach). The median alues we e com- pa ed o de e mine whe he an app oach is signi ican ly be e o wo se han ano he one, p o ided ha he p- alue is less han α. A pai wise compa ison o he app oaches wi h a sco ing sys em was used o anking he app oaches. An app oach is gi en a sco e o +1 i i is signi ican ly be e han he o he app oach. A sco e o −1isgi en o he app oach i i is signi ican ly wo se han he o he app oach. I he app oach is no signi ican ly be e o wo se han he o he app oach, a sco e o ze o is gi en o bo h app oaches. The sum o he sco es is used o anking all he app oaches (a highe sco e gi es a be e ank) o he indica o being compa ed. A ank o “1” indica es ha an app oach has pe o med signi ican ly be e han all o he app oaches. Equal anks indica e ha hose app oaches a e no signi ican ly di e en in hei pe o mance. The pe o mance o he app oaches is summa ized in Table 2. The anks o he ap- p oaches o h ee di e en indica o s a e colo -coded in g een, yellow, and ed in he o de o bes o wo s . I should be no ed ha hese ankings a e ca ego ized wi h e- spec o he sample size and sampling s a egies. The o al numbe o ins ances is 96 (48 o LHS and MVNS each) o each sample size. Each ins ance consis s o he com- bina ion o he a ious p oblem se ings, ha is, he numbe o samples (N), sampling s a egy, numbe o objec i e unc ions (K), numbe o decision a iables (n), and p ob- lem con igu a ion. The numbe o ins ances in which ull GPRs o spa se GPRs pe o m be e , wo se, o no signi ican ly di e en compa ed o he p oposed TGPR-MO su o- ga es is deno ed by “+,” “-,” and “≈,” espec i ely. Fo TGPR-MO su oga es, only he numbe o ins ances whe e i pe o med signi ican ly be e han bo h ull GPRs and spa se GPRs (o i anked he bes ) is shown. The ows measu ing “Time” in Table 2 show ha he p oposed TGPR-MO su o- ga es we e compu a ionally he cheapes compa ed o he ull GPR and spa se GPR o di e en sample sizes and sampling s a egies. The numbe o ins ances TGPR-MO pe - o med signi ican ly be e han he o he wo su oga es in each subca ego y is close o 48 ( ha was he o al numbe o ins ances in each subca ego y). The TGPR-MO su o- ga es pe o med be e han spa se GPR su oga es in hype olume o all sample sizes and sampling s a egies. Howe e , ull GPRs pe o med he bes in hype olume o a sample size o 2,000. Fo 2,000 samples, spa se GPRs ou pe o med TGPR-MO su o- ga es o bo h sampling s a egies in RMSE, and ull GPR pe o med he bes . The RMSE o he solu ions o TGPR-MO su oga es was be e han spa se GPRs o sample sizes o 10,000 and 50,000 o LHS sampling only, whe eas o MVNS sampling, he spa se GPR sligh ly ou pe o med TGPR-MO su oga es o 10,000 and 50,000 samples. Fo a smalle sample size, one may choose ull GPRs because hey gi e he bes pe - o mance in hype olume and RMSE. Howe e , ull GPRs become almos impossible o build due o hei high compu a ional cos when he sample size is la ge. One can use TGPR-MO su oga es o la ge sample sizes because hey pe o m be e in hy- pe olume and RMSE and excellen ly in compu a ion ime compa ed o spa se GPRs. E olu iona y Compu a ion Volume xx, Numbe x 17 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. Table 2: Summa y o pai wise compa ison o he hype olume (HV), RMSE, and ime (seconds) o he di e en app oaches. The numbe o ins ances in which ull GP and spa se GP su oga es pe o m be e , wo se, and no signi ican ly di e en compa ed o he TGP-MO su oga es is indica ed by “+,” “-,” and “≈,” espec i ely. The ull GP uns ailed due o memo y o e low in he ins ances ma ked “—.” The app oaches’ pe o mance is anked by he colo code g een, yellow, and ed in he o de o bes o wo s , espec i ely. Su oga e ype Sample Sampling Full GP Spa se GP TGP-MO size s a egy Indica o +/-/≈+/-/≈+ 2,000 LHS HV 9/9/30 4/38/6 9 RMSE 43/2/3 19/21/8 1 Time (s) 0/48/0 1/46/1 46 MVNS HV 17 / 14 / 17 14 / 28 / 6 12 RMSE 34/7/7 20 / 16 / 12 2 Time (s) 0/48/0 1/47/0 47 10,000 LHS HV — 4/33/11 33 RMSE — 19 / 23 / 6 23 Time (s) — 0/48/0 48 MVNS HV — 11 / 29 / 8 29 RMSE — 21 / 16 / 11 16 Time (s) — 0/48/0 48 50,000 LHS HV — 3/35/10 35 RMSE — 15 / 24 / 9 24 Time (s) — 0/48/0 48 MVNS HV — 10 / 29 / 9 29 RMSE — 23 / 19 / 6 19 Time (s) — 0/48/0 48 Howe e , o non-uni o m sampling s a egies, o example, MVNS, TGPR-MO su o- ga es su e sligh ly in RMSE compa ed o spa se GPR su oga es. Spa se GPR su o- ga es ha e be e RMSE han TGPR-MO su oga es because he a ia ional pa ame e s a e selec ed by minimizing he KL di e gence. Thus, he inducing inpu s selec ed a e no skewed e en i he p o ided da ase is, o example, in MVNS sampling. The pe o mances o a ew selec ed ins ances a e shown in Table 3. The able shows he median and he s anda d de ia ion o hype olume, RMSE, and ime aken o build he su oga es o he h ee di e en app oaches o 31 uns. The ins ances shown in he able we e chosen based on he maximum di e ence be ween he median hype olume o he bes and he second bes pe o ming su oga es. One ins ance was selec ed om each sample size, sampling s a egy, and numbe o objec i e unc ions. The igu es in bold ep esen he bes pe o ming app oaches. I can be obse ed ha he p oposed TGPR-MO su oga es pe o med he bes in building ime in he ins ances shown. I can also be obse ed ha TGPR-MO su oga es p oduce solu ions wi h an imp o ed hype olume and RMSE o sample sizes o 10,000 and 50,000 compa ed o spa se GPR su oga es. I can be obse ed ha he building imes o TGPR-MO su oga es a e less 18 E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs Table 3: Compa ison o selec ed es ins ances showing he median hype olume, RMSE, and ime and hei s anda d de ia ion (in i alics) o he 31 es uns. The bes pe o ming app oaches a e shown in bold. Hype olume RMSE Time (s) Sample Sampling Full Spa se Full Spa se Full Spa se size s a egy P oblem K n GPR GPR TGPR-MO GPR GPR TGPR-MO GPR GPR TGPR-MO 2,000 LHS P3 3 5 7.54E+01 5.49E+01 7.48E+01 4.18E-02 1.04E+00 2.67E-01 1.36E+02 8.64E+01 1.40E+01 4.02E+00 9.35E+00 2.73E+00 2.74E-01 5.72E-01 1.38E-01 1.85E+01 2.16E+01 1.20E+01 P4 5 10 1.46E+04 1.27E+04 1.64E+04 1.75E+00 1.79E+00 9.33E-01 6.16E+02 5.51E+02 2.62E+01 6.73E+02 1.23E+03 5.59E+02 3.69E-01 4.38E-01 1.66E-01 1.40E+02 1.43E+02 1.12E+01 P3 7 5 9.21E+06 6.55E+06 9.23E+06 5.11E-02 2.16E+00 1.66E-01 3.20E+02 2.10E+02 2.83E+01 1.00E+05 1.24E+06 9.32E+04 8.26E-02 9.70E-01 1.11E-01 5.38E+01 3.46E+01 1.53E+01 MVNS P3 3 10 5.71E+01 5.65E+01 6.27E+01 1.08E+00 1.52E+00 8.55E-01 3.25E+02 3.56E+02 1.31E+01 4.14E+00 6.35E+00 5.10E+00 4.65E-01 6.09E-01 1.99E-01 6.76E+01 1.19E+02 2.45E+00 P2 5 10 1.26E+04 1.06E+04 1.30E+04 1.86E+00 2.76E+00 2.04E+00 5.34E+02 5.51E+02 1.39E+01 2.08E+03 2.10E+03 1.20E+03 1.95E-01 1.59E-01 2.78E-01 1.01E+02 7.64E+01 8.00E+00 P2 7 7 7.81E+06 6.42E+06 8.82E+06 7.80E-01 3.76E+00 9.80E-01 3.57E+02 4.36E+02 1.27E+01 5.36E+05 2.62E+05 5.30E+05 2.07E-01 8.55E-01 2.38E-01 4.40E+01 6.32E+01 1.69E+01 10,000 LHS P3 3 5 — 6.26E+01 7.58E+01 — 8.95E-01 1.50E-01 — 1.45E+03 2.49E+01 9.78E+00 5.93E-01 4.85E-01 5.05E-02 2.22E+02 8.80E+00 P4 5 10 — 1.27E+04 1.65E+04 — 2.10E+00 7.09E-01 — 6.52E+03 5.93E+01 6.09E+02 8.26E+02 2.22E-01 2.28E-01 1.24E+03 2.11E+01 P1 7 10 — 6.71E+06 8.59E+06 — 1.56E+00 9.73E-01 — 9.37E+03 3.78E+01 1.90E+05 2.61E+05 1.84E-01 8.82E-02 1.38E+03 1.32E+01 MVNS P3 3 10 — 4.97E+01 6.27E+01 — 1.68E+00 9.09E-01 — 4.12E+03 1.42E+01 5.40E+00 3.31E+00 3.40E-01 1.52E-01 6.69E+02 1.52E+00 P3 5 5 — 1.23E+04 1.75E+04 — 2.32E+00 1.83E-01 — 2.38E+03 4.73E+01 2.50E+03 3.15E+02 1.10E+00 1.22E-01 4.65E+02 1.19E+01 P3 7 5 — 6.55E+06 9.24E+06 — 2.91E+00 1.22E-01 — 3.51E+03 8.94E+01 1.22E+06 1.01E+05 1.34E+00 1.42E-01 6.54E+02 3.02E+01 E olu iona y Compu a ion Volume xx, Numbe x 19 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. Table 3: Con inued. Hype olume RMSE Time (s) Sample Sampling Full Spa se Full Spa se Full Spa se size s a egy P oblem K n GPR GPR TGPR-MO GPR GPR TGPR-MO GPR GPR TGPR-MO 50,000 LHS P3 3 10 — 6.04E+01 7.30E+01 — 8.29E-01 3.91E-01 — 2.16E+04 3.16E+01 7.28E+00 2.62E+00 6.08E-01 8.63E-02 1.77E+03 9.25E+00 P3 5 10 — 1.45E+04 1.65E+04 — 8.64E-01 6.18E-01 — 3.59E+04 3.90E+01 4.10E+02 4.04E+02 2.10E-01 1.20E-01 6.29E+03 1.10E+01 P1 7 10 — 7.49E+06 8.65E+06 — 9.20E-01 7.44E-01 — 5.05E+04 8.77E+01 1.12E+05 1.65E+05 9.23E-02 8.59E-02 9.41E+03 3.44E+01 MVNS P3 3 5 — 5.27E+01 7.59E+01 — 1.77E+00 9.73E-02 — 8.19E+03 5.29E+01 1.14E+01 4.41E-01 9.11E-01 5.04E-02 6.35E+02 2.07E+01 P3 5 5 — 1.25E+04 1.76E+04 — 2.73E+00 1.43E-01 — 1.34E+04 8.68E+01 2.04E+03 3.03E+02 1.13E+00 1.49E-01 7.96E+02 4.14E+01 P3 7 5 — 6.55E+06 9.26E+06 — 3.75E+00 9.81E-02 — 1.87E+04 1.91E+02 1.05E+06 1.18E+05 1.45E+00 1.36E-01 2.24E+03 7.59E+01 20 E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs han hose o spa se GPR by an o de o abou 102and 103 o samples size o 10,000 and 50,000 espec i ely. The building ime also inc eases wi h he numbe o objec i e unc ions o spa se GPR su oga es. The o al numbe o samples u ilized o build TGPR-MO su oga es wi h he num- be o i e a ions o six di e en p oblem ins ances wi h sample sizes o 2,000, 10,000, and 50,000 is shown in Figu e 6. The solid line shows he mean numbe o samples used, and he shaded egion deno es he 95% con idence in e al o he uns o each eed GPR su oga e ( o each objec i e). The plo s ha e been ex ended in he i e a ion axis o he maximum allowed i e a ions, Imax =N 10n. The i e a ion axis is b oken when Imax is la ge o educe he wid h o he plo s. I can be obse ed ha he numbe o samples u ilized con e ges be o e he maximum i e a ions and a ies wi h he objec i e. Ce ain p oblem ins ances equi ed mo e samples du ing he building p ocess han o he s, es- pecially due o he numbe o decision a iables and he cha ac e is ics o he p oblem. I he ade-o egion is loca ed in a smalle egion o he decision space, he building p ocess con e ges quickly and he e o e consumes ewe samples. The quan i y o da a a ailable a he ade-o egion a ec s he accu acy and hy- pe olume o he solu ions ob ained ha is a gene al challenge while sol ing o line da a-d i en MOPs. The p oposed TGPR-MO su oga es spli he decision space in o sub egions and app oxima e he unde lying objec i e unc ions a he ade-o egion. I is bes sui able when he Pa e o se is loca ed in a smalle egion o he decision space. When he Pa e o se is in a la ge egion, he p edic ion o TGPR-MO su oga es is om mul iple lea node GPRs and has discon inui ies nea he spli s o he eg ession ees. The e o e he app oxima ed Pa e o on has some discon inui ies compa ed o spa se GPRs o ull GPRs. Fu he es s wi h he DTLZ (Deb e al., 2005) benchma k p oblems a e gi en in he supplemen a y ma e ial. 5 Conclusions This pape p oposed ailo ed su oga e models ha can be buil wi h a lowe compu a- ional cos compa ed o spa se GPRs and ull GPRs o sol ing o line da a-d i en MOPs when dealing wi h la ge da ase s. The p oposed TGPR-MO su oga es pe o med sig- ni ican ly be e han spa se GPR su oga es in hype olume, RMSE, and compu a ion ime o mos o he ins ances o he DBMOPP p oblems. The ull GPR su oga es ailed o comple e he uns o la ge sample sizes due o memo y es ic ions. Thus, i can be concluded ha he p oposed TGPR-MO su oga es a e bes sui ed o sol ing o line da a-d i en MOPs wi h a la ge size da ase . A GPR a a lea node o a ee app oxima es ce ain egions o he decision space. This ea u e can be exploi ed while sol ing o line da a-d i en MOPs wi h p e e ences om he decision make in an a p io i o in e ac i e ashion. The p o ided p e e ences o objec i e unc ions can be u ilized o build GPRs a he lea nodes, and only he solu ions ha ollow he p e e ences can be app oxima ed. De eloping an in e ac i e amewo k u ilizing he TGPR-MO su oga es will be one o he u u e wo ks. U ilizing he co ela ion be ween he objec i e unc ions using mul i- a ge eg es- sion ees (Osojnik e al., 2018) and mul i-ou pu GPRs (Bo chani e al., 2015) is an in- e es ing u u e esea ch di ec ion. Tes s and compa isons o he unce ain y p edic ion p o ided by he p oposed TGPR-MO su oga es will be conduc ed o sol ing o line da a-d i en MOPs. Selec ing a sui able ke nel o a gi en da ase is an ac i e esea ch opic. In eg a ing au oma ic ke nel selec ion in o he TGPR-MO su oga es will be qui e bene icial. One o he majo d awbacks o he TGPR-MO su oga es is he discon inui y be ween he lea node GPRs. Tackling discon inui y in he p edic ion ( an S ein e al., E olu iona y Compu a ion Volume xx, Numbe x 21 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. Figu e 6: The median and 95% con idence in e al o he o al numbe o samples u i- lized (o he sum o he samples a ailable a he lea nodes ha a e conside ed o build- ing GPRs) wi h i e a ions o h ee di e en DBMOPP p oblem ins ances as displayed acco ding o he ollowing o ma (N, sampling s a egy, p oblem, K,n). The i e a ion axis is ex ended o he maximum possible i e a ion N 10n. The i e a ion axes o plo s 6c–6 a e b oken o accommoda e he plo s. 22 E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 T eed GPR o O line Da a-D i en MOPs 2015; Wang e al., 2017) a he pa i ion o he decision space (as p o ided by he ee) will be a u u e ask. Fu he imp o emen s can be made in he way he ees pa i ion he decision space. Ins ead o using a adi ional loss unc ion, a nonlinea ade-o c i e ion can be o mula ed o spli he nodes. Such spli ing c i e ia will enable he TGPR-MO su oga es o app oxima e he ade-o egion wi h ewe samples. Tes ing he TGPR-MO su oga es o sol ing eal-li e o line da a-d i en MOPs will be a u u e ask. Acknowledgmen s This esea ch was pa ly suppo ed by he Academy o Finland (g an numbe 311877 and 322221) and is ela ed o he hema ic esea ch a ea DEMO (Decision Analy ics u i- lizing Causal Models and Mul iobjec i e Op imiza ion, h p://www.jyu. i/demo) o he Uni e si y o Jy äskylä. The nume ical expe imen s we e pe o med on Mah i su- pe compu e p o ided by CSC (h ps://www.csc. i). Re e ences Assael, J.-A. M., Wang, Z., Shah ia i, B., and de F ei as, N. (2014). He e oscedas ic eed Bayesian op imisa ion. CoRR. Re ie ed om a Xi :1410.7172. Bo chani, H., Va ando, G., Bielza, C., and La añaga, P. (2015). A su ey on mul i-ou pu eg es- sion. WIREs Da a Mining and Knowledge Disco e y, 5(5):216–233. 10.1002/widm.1157 Chapman, W., Welch, W., Bowman, K., Sacks, J., and Walsh, J. (1994). A c ic sea ice a iabili y: Model sensi i i ies and a mul idecadal simula ion. Jou nal o Geophysical Resea ch: Oceans, 99(C1):919–935. 10.1029/93JC02564 Cheng, R., Jin, Y., Olho e , M., and Sendho , B. (2016). A e e ence ec o guided e olu iona y algo i hm o many-objec i e op imiza ion. IEEE T ansac ions on E olu iona y Compu a ion, 20(5):773–791. 10.1109/TEVC.2016.2519378 Chipman, H. A., Geo ge, E. I., and McCulloch, R. E. (1998). Bayesian CART model sea ch. Jou nal o he Ame ican S a is ical Associa ion, 93(443):935–948. 10.1080/01621459.1998.10473750 Chugh, T., Sindhya, K., Hakanen, J., and Mie inen, K. (2019). A su ey on handling compu a- ionally expensi e mul iobjec i e op imiza ion p oblems wi h e olu iona y algo i hms. So Compu ing, 23:3137–3166. 10.1007/s00500-017-2965-0 Das, K., and S i as a a, A. N. (2010). Block-GP: Scalable Gaussian p ocess eg ession o mul i- modal da a. In 2010 IEEE In e na ional Con e ence on Da a Mining, pp. 791–796. Deb, K., and Jain, H. (2014). An e olu iona y many-objec i e op imiza ion algo i hm using e e ence-poin -based nondomina ed so ing app oach, Pa I: Sol ing p oblems wi h box cons ain s. IEEE T ansac ions on E olu iona y Compu a ion, 18(4):577–601. 10.1109/TEVC .2013.2281535 Deb, K., Thiele, L., Laumanns, M., and Zi zle , E. (2005). Scalable es p oblems o e olu iona y mul iobjec i e op imiza ion. In A. Ab aham, L. Jain, and R. Goldbe g (Eds.), E olu iona y mul iobjec i e op imiza ion: Theo e ical ad ances and applica ions, pp. 105–145. Sp inge . Emme ich, M., Giannakoglou, K., and Naujoks, B. (2006). Single- and mul iobjec i e e olu iona y op imiza ion assis ed by Gaussian andom ield me amodels. IEEE T ansac ions on E olu ion- a y Compu a ion, 10(4):421–439. 10.1109/TEVC.2005.859463 Fieldsend, J. E., Chugh, T., Allmendinge , R., and Mie inen, K. (2019). A ea u e ich dis ance- based many-objec i e isualisable es p oblem gene a o . In P oceedings o he Gene ic and E olu iona y Compu a ion Con e ence, pp. 541–549. E olu iona y Compu a ion Volume xx, Numbe x 23 Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023 329 A. Mazumda e al. Fo es e , A., Sobes e , A., and Keane, A. (2008). Enginee ing design ia su oga e modelling. John Wiley & Sons. GPy (2012). GPy: A Gaussian p ocess amewo k in py hon. h p://gi hub.com/She ieldML/ GPy. G amacy, R. B., and Lee, H.K.H. (2008). Bayesian eed Gaussian p ocess models wi h an applica- ion o compu e modeling. Jou nal o he Ame ican S a is ical Associa ion, 103(483):1119–1130. 10.1198/016214508000000689 Hughes, E. J. (2001). E olu iona y mul i-objec i e anking wi h unce ain y and noise. In P oceed- ings o E olu iona y Mul i-C i e ion Op imiza ion, pp. 329–343. Jin, Y., Wang, H., Chugh, T., Guo, D., and Mie inen, K. (2019). Da a-d i en e olu iona y op imiza- ion: An o e iew and case s udies. IEEE T ansac ions on E olu iona y Compu a ion, 23(3):442– 458. 10.1109/TEVC.2018.2869001 Jones, D. R., Schonlau, M., and Welch, W. J. (1998). E icien global op imiza ion o expensi e black-box unc ions. Jou nal o Global Op imiza ion, 13:455–492. 10.1023/A:1008306431147 Kim, H.-M., Mallick, B. K., and Holmes, C. C. (2005). Analyzing nons a iona y spa ial da a using piecewise Gaussian p ocesses. Jou nal o he Ame ican S a is ical Associa ion, 100(470):653–668. 10.1198/016214504000002014 Loh, W.-Y. (2011). Classi ica ion and eg ession ees. WIREs Da a Mining and Knowledge Disco e y, 1(1):14–23. 10.1002/widm.8 Mazumda , A., Chugh, T., Hakanen, J., and Mie inen, K. (2020). An in e ac i e amewo k o o - line da a-d i en mul iobjec i e op imiza ion. In P oceedings o Bioinspi ed Op imiza ion Me h- ods and Thei Applica ions, pp. 97–109. Mazumda , A., Chugh, T., Hakanen, J., and Mie inen, K. (2022). P obabilis ic selec ion ap- p oaches in decomposi ion-based e olu iona y algo i hms o o line da a-d i en mul- iobjec i e op imiza ion. IEEE T ansac ions on E olu iona y Compu a ion, 26(5):1182–1191. 10.1109/TEVC.2022.3154231 Mazumda , A., Chugh, T., Mie inen, K., and López-Ibáñez, M. (2019). On dealing wi h unce ain- ies om K iging models in o line da a-d i en e olu iona y mul iobjec i e op imiza ion. In P oceedings o E olu iona y Mul i-C i e ion Op imiza ion, pp. 463–474. Misi ano, G., Saini, B. S., A sa , B., Sha azipu , B., and Mie inen, K. (2021). DESDEO: The mod- ula and open sou ce amewo k o in e ac i e mul iobjec i e op imiza ion. IEEE Access, 9:148277–148295. 10.1109/ACCESS.2021.3123825 Osojnik, A., Pano , P., and Dže oski, S. (2018). T ee-based me hods o online mul i- a ge eg ession. Jou nal o In elligen In o ma ion Sys ems, 50:315–339. 10.1007/s10844-017 -0462-7 Ped egosa, F., Va oquaux, G., G am o , A., Michel, V., Thi ion, B., G isel, O., Blondel, M., e al. (2011). Sciki -lea n: Machine lea ning in Py hon. Jou nal o Machine Lea ning Resea ch, 12:2825–2830. Raha , A.A.M., Wang, C., E e son, R. M., and Fieldsend, J. E. (2018). Da a-d i en mul i-objec i e op imisa ion o coal- i ed boile combus ion sys ems. Applied Ene gy, 229:446–458. 10.1016/ j.apene gy.2018.07.101 Rasmussen, C. E., and Williams, C.K.I. (2006). Gaussian p ocesses o machine lea ning. MIT P ess. Shah ia i, B., Swe sky, K., Wang, Z., Adams, R. P., and de F ei as, N. (2016). Taking he human ou o he loop: A e iew o Bayesian op imiza ion. P oceedings o he IEEE, 104(1):148–175. 10.1109/JPROC.2015.2494218 24 E olu iona y Compu a ion Volume xx, Numbe x Downloaded om h p://di ec .mi .edu/e co/a icle-pd /doi/10.1162/e co_a_00329/2155340/e co_a_00329.pd by JYVASKYLAN YLIOPISTO use on 10 Oc obe 2023