!
!
!
!
CIplus
Band 5/2016
S acked Gene aliza ion o Su oga e Models -
A P ac ical App oach
Thomas Ba z-Beiels ein
!
!
!
!
!
!
!
! !
S acked Gene aliza ion o Su oga e Models
A P ac ical App oach
Thomas Ba z-Beiels ein
TH Köln
Compu e Science and Enginee ing Science
[email p o ec ed]
h p://www.spo se en.de
May 29, 2016
Abs ac
This epo p esen s a p ac ical app oach o s acked gene aliza ion in su oga e model
based op imiza ion. I exempli ies he in eg a ion o s acking me hods in o he su oga e model
building p ocess. Fi s , a b ie o e iew o he cu en s a e in su oga e model based op i-
miza ion is p esen ed. S acked gene aliza ion is in oduced as a p omising ensemble su oga e
modeling app oach. Then wo examples ( he i s is based on a eal wo ld applica ion and
he second on a se o a i icial es unc ions) a e p esen ed. These examples clea ly illus a e
wo p ope ies o s acked gene aliza ion: (i) combining in o ma ion om wo poo pe o ming
models can esul in a good pe o ming model and (ii) e en i he ensemble con ains a good
pe o ming model, combining i s in o ma ion wi h in o ma ion om poo pe o ming models
esul s in a ela i ely small pe o mance dec ease only.
1 In oduc ion
The selec ion o an adequa e me a model is c ucial in model based op imiza ion (MBO). Model
based op imiza ion plays a p ominen ole in odays modeling, simula ion, and op imiza ion
p ocesses. I is one o he mos e icien echnique o expensi e and ime demanding eal-
wo ld op imiza ion p oblems. Especially in he enginee ing domain, MBO is an impo an
echnique. This popula i y is caused by ecen ad ances in compu e science, s a is ics, and
enginee ing, in combina ion wi h p og ess in high-pe o mance compu ing. A combina ion
o heses ad anced ools enable he ea men o p oblems conside ed unsol able only a ew
decades ago. We will conside MBO in he con ex o global op imiza ion.
Global op imiza ion (GO) can be ca ego ized on di e en c i e ia, e.g., he p ope ies o
he p oblems (con inuous e sus combina o ial, linea e sus non-linea , con ex e sus non-
con ex, e c.). In many eal wo ld si ua ions, GO p oblems a e di icul , because nea ly no
s uc u al in o ma ion (e.g., numbe o local ex ema) is a ailable. These kind o GO p oblems
belong o he class o black-box unc ions, i.e., he analy ic o m is unknown. No e, he class
o black-box unc ion con ains also unc ions ha a e easy o sol e, e.g., con ex unc ions. The
op imiza ion p oblem is gi en by
minimize: (~x)subjec o ~xl≤~x ≤~xu,
whe e :Rn→Ris e e ed o as he objec i e unc ion and ~xland ~xudeno e he lowe
and uppe bounds o he sea ch space ( egion o in e es ), espec i ely. This se ing a ises in
1
many eal-wo ld sys ems, i.e., when he explici o m o he objec i e unc ion is no eadily
a ailable o e.g., use has no access o he sou ce code o a simula o .
The e m GO will be used o algo i hms ha a e ying o ind and explo e global op imal
solu ions wi h complex, mul imodal objec i e unc ions [36]. We will use an algo i hmic iew,
i.e., we will conside he p ope ies o algo i hms.
Fi s , we will desc ibe s ochas ic ( andom) sea ch algo i hms and show how su oga e
model based op imiza ion can be classi ied in he con ex o GO. The e o e we in oduce he
ollowing axonomy.
1. De e minis ic
2. Random Sea ch
(a) Ins ance based.
(b) Model based op imiza ion (MBO).
i. Dis ibu ion based.
ii. Su oga e Model Based Op imiza ion (SBO).
A. Single su oga e based.
B. Mul i- ideli y based.
C. E olu iona y su oga e based.
D. Ensemble su oga e based.
Then, we will y o answe he ques ion o selec ing an adequa e su oga e model in he con-
ex o SBO.
Tis epo is s uc u ed as ollows: Fi s , SBO is p esen ed in he con ex o s ochas ic sea ch
algo i hms (Sec ion 2). Sec ion 3p esen s some gene al conside a ions abou using mul iple
su oga e models. The sequen ial pa ame e op imiza ion (SPO), which uses SBO, is in oduced in
Sec ion 4. S acked gene aliza ion is one impo an modeling echnique in SPO. An indus ial
applica ion is used in Sec ion 5 o in oducing he key ea u es o he s acked gene aliza ion
app oach. To gain u he insigh , a second s udy, which uses a i icial es unc ions, is p e-
sen ed in Sec ion 6. This epo concludes wi h a sho summa y and an ou look in Sec ion 7.
2 S ochas ic Sea ch Algo i hms
To mo i a e he impo ance o model selec ion in MBO, we will desc ibe he ela ed GO algo-
i hms i s . S ochas ic sea ch algo i hms pe o m an i e a i e sea ch. They use a s ochas ic
p ocedu e o gene a e he nex i e a e. The nex i e a e can be a candida e solu ion o he GO o
a p obabilis ic model, whe e solu ions can be d awn om. S ochas ic sea ch algo i hms do no
depend on any s uc u al in o ma ion o he objec i e unc ion such as g adien in o ma ion
o con exi y. Hence, hey a e obus and easy o implemen . S ochas ic sea ch algo i hms can
u he be ca ego ized as ins ance-based o model-based algo i hms [51]
Ins ance-based algo i hms use a single solu ion, ~x, o popula ion,P( ), o candida e solu-
ions. The cons uc ion o new candida es depends explici ly on p e iously gene a ed solu-
ions. P ominen examples a e simula ed annealing o e olu iona y algo i hms.
Model-based op imiza ion algo i hms gene a e a popula ion o new candida e solu ions
P0( )by sampling om a model. In s a is ics, he e ms model and dis ibu ion a e used synony-
mously. The e o e, we will use he e m su oga e model, when we a e e e ing o an explici
model. The model (o dis ibu ion) e lec s s uc u al p ope ies o he unde lying ue unc-
ion, say . Adap ing he model (o he dis ibu ion), he sea ch is di ec ed in o egions wi h
imp o ed solu ions. One o he key ideas in MBO can be o mula ed as ollows: eplace ex-
pensi e, high ideli y, ine g ained unc ion e alua ions, (~x), wi h e alua ions, ˆ
(~x), o an
adequa e su oga e model, say M. Su oga e models also known as he cheap model, he e-
sponse su ace, he me a model, he app oxima ion, o he coa se g ained model.
2
2.1 Dis ibu ion-based App oaches
Dis ibu ion-based op imiza ion algo i hms main ain a dis ibu ion as a me amodel. A se-
quence o i e a es, i.e., p obabili y dis ibu ions, {p( )}is gene a ed. The i e a es should ha e
he p ope y, ha
p( )→p∗as → ∞,
whe e p∗is he limi ing dis ibu ion, which assigns mos o i s p obabili y mass o he se o
op imal solu ions. No e, dis ibu ion-based op imiza ion algo i hms p opaga e a p obabili y
dis ibu ion om one i e a ion o he nex , whe eas ins ance-based algo i hms p opaga e can-
dida e solu ions ~x.
Es ima ion o dis ibu ion algo i hms (EDA) a e e y popula in he ield o e olu iona y algo-
i hms (EA). Va ia ion ope a o s such as mu a ion and ecombina ion a e eplaced by a dis i-
bu ion based p ocedu e. A p obabili y dis ibu ion, which is es ima ed om p omising can-
dida e solu ions om he cu en popula ion, is used o gene a e new popula ion. [27] e iew
di e en ways o using p obabilis ic models. [18] discuss ad an ages and ou line many o he
di e en ypes o EDAs. [19] p esen ecen app oaches and a uni ied iew.
Al hough dis ibu ion-based app oaches play an impo an ole in GO, hey will no be
discussed u he in his epo . We will concen a e on su oga e model based app oaches,
which ha e hei o igin in s a is ical design and analysis o expe imen s, especially in esponse
su ace me hodology [12] [31].
2.2 Su oga e Model-based App oaches
In gene al, he e m su oga e is used, when he ou come o a p ocess canno be di ec ly mea-
su ed. A su oga e ies o imi a e he beha io o he eal model as closely as possible while
being compu a ionally cheape o e alua e. Simple su oga e models can be cons uc ed us-
ing a da a-d i en app oach. They can be e ined by in eg a ing addi ional poin s o domain
knowledge, e.g., cons ain s.
A wide ange o su oga es we e de eloped in he las decades. This esul s in complex
design decisions. Following he discussion in [47], su oga e design decisions a e necessa y o
he selec ion o (i) me amodels, (ii) designs, and (iii) model i ing me hods.
Me amodels Typical me amodels a e (a) classical eg ession models such as polynomial e-
g ession o esponse su ace me hodology [12] [31] (b) suppo ec o machines} (SVM)
[46] (c) neu al ne wo ks [52] (d) adial basis unc ions [35], o (e) Gaussian p ocess (GP)
models, also known as design and analysis o compu e expe imen s o K iging [42] [7],
[1], [26], [41]. [10] p esen s a comp ehensi e in oduc ion o SBO.
Designs A b oad a ie y o designs a e a ailable, e.g., classical expe imen al designs such as
ac o ial, ac ional ac o ial, cen al composi e, o A-, D-op imal (alphabe ically) designs.
Al e na i ely, space illing designs, such as simple g ids, La in hype cube designs, o -
hogonal o uni o m designs can be used. In addi ion, hyb id designs, andom o human
selec ion, and sequen ial design me hods a e a ailable.
Model Fi Model i ing can be based on se e al c i e ia, e.g., weigh ed leas squa es eg es-
sion o maximum likelihood es ima ion. Special i ing echniques exis o speci ic mod-
eling app oacches, e.g., backp opaga ion o neu al ne wo ks.
2.3 SBO Applica ions
Simula ion-based design o complex enginee ing p oblems, e.g., compu a ional luid dynamics
(CFD) and ini e elemen modeling (FEM) me hods a e he mos popula applica ion a eas o
SBO. To gene a e exac solu ions, he co esponding sol e s equi e a la ge numbe o expen-
si e compu e simula ions. Gene ally, he e a e wo SBO a ian s: (i) me amodel based me h-
ods, which use one o se e al di e en me amodels and (ii) mul i- ideli y app oxima ion me h-
ods, which uses se e al ins ances wi h di e en pa ame e iza ions o he same me amodel.
3
Helicop e o o design op imiza ion [4], ae odynamic shape design [13], mul i-objec i e
op imal design o a liquid ocke injec o [38], and ae ospace design [11] a e only a ew exam-
ples o SBO in indus y.
Mul i- ideli y app oxima ion is used in [20], who use se e al simula ion models wi h di -
e en g id sizes in FEM and in [44], who op imize a shee me al o ming p ocess.
2.4 Su oga e-assis ed E olu iona y Algo i hms
Su oga e-assis ed EA use a cheap su oga e model o eplace e alua ions o expensi e ob-
jec i e unc ion. Se e al a ian s o su oga e-assis ed EAs we e de eloped in he las yea s,
e.g., a combina ion o a gene ic algo i hm and neu al ne wo ks o ae odynamic design op i-
miza ion [16], an app oxima e model o he i ness landscape using K iging in e pola ion o
accele a e he con e gence o EAs [39], an E olu ion s a egy (ES) wi h neu al ne wo k based
i ness e alua ions [24], o a su oga e-assis ed EA amewo k wi h online lea ning [50]. A
su ey o su oga e-assis ed EA app oaches is p esen ed in [23]. SBO app oaches o e olu ion
s a egies a e desc ibed in [9].
3 Mul iple Models and Model Selec ion
Ins ead o using one su oga e model only, se e al models Mi,i= 1,2, . . . , p, gene a ed and
e alua ed in pa allel can be used. Each model Mi:X→yuses he same candida e solu ions,
X, om he popula ion Pand he same esul s, y, om expensi e unc ion e alua ions.
Mul iple models can also be used o pa i ion he sea ch space. He e we can men ion ee-
based Gaussian p ocesses, which use eg ession ees o pa i ion he sea ch space and i local
GP su oga es in each egion [15]. A ee-based pa i ioning o an ae odynamic design space,
which uses independen K iging su aces in each pa i ion, is desc ibed in [33]. The combi-
na ion o an e olu iona y model selec ion algo i hm wi h expec ed imp o emen (EI) c i e ion,
which selec s he bes pe o ming su oga e model ype a each i e a ion o he EI algo i hm
was p oposed by [8].
Ensembles o su oga e models gained popula i y. An adap i e weigh ed a e age model
o he indi idual su oga es was p esen ed in [49]. An app oach which uses he bes su o-
ga e model o a weigh ed a e age su oga e model ins ead was in oduced in [14]. In hese
app oaches, he models o he ensemble a e chosen based on hei pe o mance. Usually, he
weigh s a e adap i e and in e sely p opo ional o he local modeling e o s.
The simples model selec ion p ocess is a e inemen me hod: he same (ini ial) model will
be e ined du ing he op imiza ion. This me hod equi es a selec ion c i e ia o sampling
new poin s, so-called in ill poin s. The balance be ween explo a ion, i.e., imp o ing he model
quali y ( ela ed o he model, global), and exploi a ion, i.e., imp o ing he op imiza ion and
de e mining he op imum ( ela ed o he objec i e unc ion, local) plays a cen al ole in his
s a egy. Expec ed imp o emen (EI) is a popula adap i e sampling me hod [30] [25].
The EI app oach handles he ini ializa ion and e inemen o a su oga e model, bu no
he selec ion o he model i sel . Fo example, he popula e icien global op imiza ion (EGO)
algo i hm uses a K iging model, because K iging inhe en ly de e mines he p edic ion a iance
(necessa y o he EI c i e ion). Bu he e is no p oo ha K iging is he bes choice. Al e na i e
su oga e models, e.g., neu al ne wo ks, eg ession ees, suppo ec o machines, o lasso
and idge eg ession may be be e sui ed. Howe e : An a p io y selec ion o he bes sui ed
su oga e model is concep ually impossible in he amewo k ea ed in his epo , because o
he black-box se ing.
Rega ding he model choice, he use can decide whe he o use (i) one single model, i.e.,
one unique global model o (ii) mul iple models, i.e., an ensemble o di e en , possibly local,
models.
Now, we do no conside he selec ion o a new sample poin (as done in EI). Ins ead, we
4
conside c i e ia o he selec ion o one (o se e al) su oga e models. Usually, su oga e mod-
els chosen acco ding o hei es ima ed ue e o [22], [43].
Commonly used pe o mance me ics a e he mean absolu e e o (MAE) o he oo mean
squa e e o (RMSE). Gene ally, a aining a su oga e model ha has minimal e o is he de-
si ed ea u e. Me hods om s a is ics, s a is ical lea ning [17] and machine lea ning [32] a e
popula , e.g., simple holdou me hods, c oss- alida ion, o he boo s ap.
An al e na e app oach is p esen ed in [28]. He e, he model e o is no he only c i e ion
o selec ing su oga e models. The au ho s p esen an e ol abili y lea ning o su oga es ap-
p oach, which uses i ness imp o emen o de e mining he quali y o su oga e models.
4 Sequen ial Pa ame e Op imiza ion
The sequen ial pa ame e op imiza ion (SPO) uses a cen alized, global in o ma ion based ap-
p oach o handling su oga e model in o ma ion. I implemen s a s acked gene aliza ion ap-
p oach de eloped by [48]. Ea ly e sions o he SPO [3], [2] combined me hods om design
o expe imen s (DOE) [37], esponse su ace me hodology (RSM) [5] [31], design and analysis o com-
pu e expe imen s (DACE) [29] [41], and eg ession ees [6]. The s a is ical analysis and an
unde s anding o op imiza ion algo i hms was he main goal o he SPO. In addi ion, i was
ecognized ha SPO can be used as an op imize .
The SPO p o ides a sequen ial, model based app oach o op imiza ion and is nowadays an
es ablished pa ame e une and an op imiza ion algo i hm. I was ex ended in se e al ways,
e.g., [21] benchma k an SPO de i a i e, he so-called sequen ial model-based algo i hm con igu a-
ion (SMAC) p ocedu e, on he BBOB se o blackbox unc ions. Gi en a small budge o 10 ×d
e alua ions o d-dimensional unc ions, SMAC in mos cases ou pe o ms he s a e-o - he-a
blackbox op imize CMA-ES.
The mos ecen e sion, SPO2, is cu en ly unde de elopmen . I in eg a es s a e-o - he-
a ensemble lea ne s. The SPO2 ensemble engine, which is desc ibed in his epo , uses a
po olio o su oga e models, such as eg ession ees and andom o es , leas angle eg es-
sion (la s), and K iging as le el-0 models. I uses c oss alida ion o gene a e a weigh ed com-
bina ion o se e al su oga e models o build a gene alized le el-1 model.
S acked gene aliza ion is implemen ed o combine se e al le el-0 models o di e en ypes
wi h one le el-1 model in o an ensemble [48]. The le el-1 aining algo i hm is a simple linea
model.
The SPO2 ensemble engine can lead o signi ican pe o mance imp o emen s. [40] p esen
a compa ison o di e en da a d i en modeling me hods, e.g., a Bayesian model, se e al linea
eg ession models, a K iging model, and gene ic p og amming. These me hods we e used
o model he beha io o a obus gas senso . The unde lying da a has a limi ed amoun o
samples and a high a iance.
5 S acked Gene aliza ion in P ac ice. Pa I: Indus ial Appli-
ca ion
This sec ion illus a es in de ail how he s acked gene aliza ion wo ks. I uses he p og am-
ming language Py hon, see h ps://www.py hon.o g. Sec ion 5.1 desc ibes he echnical
equi emen s, e.g., he Py hon lib a ies, which a e needed o pe o m he expe imen s. Sec-
ion 5.2 desc ibes he da a. The k- old c oss alida ion is p epa ed in Sec ion 5.3. How models
a e added o he SPOT2 ensemble engine is explained in Sec ion 5.4. C oss- alida ion o he
s acking p ocedu e is desc ibed in Sec ion 5.5. The le el-1 model cons uc ion and how i can
be used o p edic ions is shown in Sec ion 5.6. A schema ic illus a ion o he s acked gene al-
iza ion is shown in Fig. 1.
5
Da a
T aining:
{(Xi,y
i)}i=1,...n
{(Xi,y
i)}i=1,...n
{y i}{y i}
A1
A1
A2
A2
Val_T aining:
{(Xi,y
i)}i=1,...q
{(Xi,y
i)}i=1,...q
Val_Tes :
{(Xi,y
i)}i=q+1,...n
{(Xi,y
i)}i=q+1,...n
AV
1
AV
1
AV
2
AV
2
Val_Tes :
{(Xi)}i=q+1,...n
{(Xi)}i=q+1,...n
Val_Tes :
{(yi)}i=q+1,...n
{(yi)}i=q+1,...n
ˆyAV
1
ˆyAV
1
ˆyAV
2
ˆyAV
2
A3
A3
A(ˆy,y)
3
A(ˆy,y)
3
AT
2
AT
2
AT
1
AT
1
ˆyAT
1
ˆyAT
1
ˆyAT
2
ˆyAT
2
ˆyˆy
{X i}{X i}
Tes :
{(X i,y
i)}i=1,...,m
{(X i,y
i)}i=1,...,m
Figu e 1: Illus a ion o he da a low in s acked gene aliza ion. The da a is spli in o a es and
aining se . The aining da a is used o CV, i.e., in each old, a new aining and alida ion da a
se is gene a ed. He e, we conside wo algo i hms, say A1and A2, which a e used o gene a e
wo le el-0 models, AV
1and AV
2. These models a e used o p edic ions on he alida ion da a se ,
which esul in p edic ions ˆyAV
1and ˆyAV
2. A le el-1 algo i hm, A3is i ed o he da a se s ˆyAV
1,
ˆyAV
2, and he yi’s om he alida ion da a se . The esul ing model, A(ˆy,y)
3, is used o he inal
p edic ions.
6
5.1 Technical Requi emen s
This sec ion illus a es and demons a es he key ing edien s o he SPO2 ensemble engine. I
implemen s ideas om [48] and is based on Py hon code om [34]. Fi s , we ha e o
1. impo lib a ies and
2. se he SPO2 pa ame e s, i.e., he numbe o olds o he c oss- alida ions.
impo sys
impo ma plo lib
impo ma plo lib.pyplo as pl
impo numpy as np
impo pandas as pd
impo s a smodels. o mula.api as sm
impo ma h
om IPy hon.display impo se _ma plo lib_ o ma s
om __ u u e__ impo di ision
om sklea n.model_selec ion impo KFold
om sklea n.ensemble impo RandomFo es Reg esso
om sklea n.linea _model impo Linea Reg ession
om sklea n.me ics impo mean_squa ed_e o
om sklea n.me ics impo 2_sco e
om pandas impo ead_cs
np. andom.seed(0)# seed o shu le he ain se
n_ olds = 10
5.2 The Da a
The comple e da a se is desc ibed in [40]. I consis s o wo sepa a e da a se s o wo di e en
gas senso s: one aining da a se and one es da a se . He e, we conside da a om he
second senso . The e a e se en inpu alues and one ou pu alue (y). The goal o his s udy is
o p edic he ou come yusing he se en inpu measu emen s.
In [10]: d T ain = ead_cs (' aining.cs ')
d Tes = ead_cs (' es ing.cs ')
XT ain =d T ain.ix[:,0:7]
yT ain =d T ain.ix[:,7:9]
yT ain1 =yT ain.ix[:,1]
X=XT ain.as_ma ix()
y=yT ain1.as_ma ix()
XTes 1 =d Tes .ix[:,0:7]
yTes 1 =d Tes .ix[:,7:9]
yTes =yTes 1.ix[:,1]
XTes =XTes 1.as_ma ix()
5.3 CV Spli s
The aining da a a e spli in o olds. KFold() di ides all he samples in k=n olds g oups o
samples (called olds) o equal sizes (i possible). The p edic ion unc ion is lea ned using k−1
olds, and he old le ou is used o es .
In [11]: sk =KFold(n_ olds);
sk .ge _n_spli s(X, y);
7
5.4 Le el-0 Models in he Ensemble
A linea eg ession model and a andom o es eg ession model a e included in his s udy.
Addi ional models such as Lasso o Gaussian p ocess models can be included e y easily.
In [12]: models =[Linea Reg ession()
, RandomFo es Reg esso ()
]
5.5 C oss-Valida ion
Le ndeno e he size o he aining se (numbe o samples in he aining se ) and p he
numbe o models. Summa izing, we will conside he ollowing dimensions:
•n: size o he aining se (samples)
•k: numbe o olds o CV
•p: numbe o models
•m: size o he es da a (samples)
We will use wo ma ices o s o e he CV esul s:
1. YCV is a (n×p)-ma ix. I s o es he esul s om he c oss alida ion o each model. The
aining se is pa i ioned in o k olds (n_ olds=k).
2. YBT is a (m×p)-ma ix. I s o es he agg ega ed esul s om he c oss alida ion models
on he es da a. Fo each old, psepa a e models a e build, which a e used o p edic ion
on he es da a. The p edic ed alues om he k olds a e a e aged o each model, which
esul s in (m×p)di e en alues.
In [13]: YCV =np.ze os((X.shape[0], len(models)))
YBT =np.ze os((XTes .shape[0], len(models)))
o j, AV in enume a e(models):
YBT_j =np.ze os((XTes .shape[0], sk .n_ olds))
o i, ( ain, al) in enume a e(lis (sk .spli (X,y))):
XValT aining =X[ ain,]
yValT aining =y[ ain]
XValTes =X[ al]
AV. i (XValT aining, yValT aining)
YCV[ al, j] =AV.p edic (XValTes )
YBT_j[:, i] =AV.p edic (XTes )
YBT[:,j] =YBT_j.mean(1)
5.6 The Le el-1 Model
5.6.1 Model Building
The le el-1 model is a unc ion o he CV- alues o each model o he known, aining y- alues.
I p o ides an es ima e o he in luence o he single models. Fo example, i a linea le el-1
model is used, he coe icien βi ep esen s he e ec o he i- h model.
5.6.2 Model P edic ion
The le el-1 model is used o p edic ions on he YBT da a, i.e., on he a e aged p edic ions o
he CV-models. I is cons uc ed using he e ec s o he p edic ed alues o he single models
(de e mined by linea eg ession) on he ue alues o he aining da a. I a model p edic s a
simila alue as he ue alue du ing he CV, hen i has a s ong e ec .
8
0123
0: in e cep , 1-3: be a
0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
5: coe icien s
0123
0: SPO2, 1: L, 2: R, 3: K
0.76
0.78
0.80
0.82
0.84
0.86
0.88
0.90
5: R squa ed
Figu e 8: Resul s om he linea unc ion, 5.Le : Boxplo s showing he βcoe icien s o he
le el-1 model. Righ : Boxplo s showing he R2 alues.
0123
0: in e cep , 1-3: be a
12
10
8
6
4
2
0
2
6: coe icien s
0123
0: SPO2, 1: L, 2: R, 3: K
0.3
0.2
0.1
0.0
0.1
0.2
0.3
0.4
0.5
6: R squa ed
Figu e 9: Resul s om he noise unc ion, 6.Le : Boxplo s showing he βcoe icien s o he
le el-1 model. Righ : Boxplo s showing he R2 alues.
15
Re e ences
[1] Alessand o Baldi An ognini and Ma oussa Zago aiou. Exac op imal designs o com-
pu e expe imen s ia K iging me amodelling. Jou nal o S a is ical Planning and In e ence,
140(9):2607–2617, Sep embe 2010.
[2] Thomas Ba z-Beiels ein, Ch is ian Lasa czyk, and Mike P euß. Sequen ial Pa ame e Op-
imiza ion. In B McKay e al., edi o s, P oceedings 2005 Cong ess on E olu iona y Compu a-
ion (CEC’05), Edinbu gh, Sco land, pages 773–780, Pisca away NJ, 2005. IEEE P ess.
[3] Thomas Ba z-Beiels ein, Kons an inos E Pa sopoulos, and Michael N V aha is. Design
and analysis o op imiza ion algo i hms using compu a ional s a is ics. Applied Nume ical
Analysis and Compu a ional Ma hema ics (ANACM), 1(2):413–433, 2004.
[4] And ew J Booke , J E Dennis J , Paul D F ank, Da id B Se a ini, and Vi ginia To czon.
Op imiza ion Using Su oga e Objec i es on a Helicop e Tes Example. In Compu a ional
Me hods o Op imal Design and Con ol, pages 49–58. Bi khäuse Bos on, Bos on, MA, 1998.
[5] G E P Box and N R D ape . Empi ical Model Building and Response Su aces. Wiley, New
Yo k NY, 1987.
[6] L B eiman, J H F iedman, R A Olshen, and C J S one. Classi ica ion and Reg ession T ees.
Wadswo h, Mon e ey CA, 1984.
[7] D Büche, N N Sch audolph, and P Koumou sakos. Accele a ing E olu iona y Algo i hms
Wi h Gaussian P ocess Fi ness Func ion Models. IEEE T ansac ions on Sys ems, Man and
Cybe ne ics, Pa C (Applica ions and Re iews), 35(2):183–194, May 2005.
[8] I o Couckuy , Filip De Tu ck, Tom Dhaene, and Di k Go issen. Au oma ic su oga e
model ype selec ion du ing he op imiza ion o expensi e black-box p oblems. In 2011
Win e Simula ion Con e ence - (WSC 2011), pages 4269–4279. IEEE, 2011.
[9] Michael Emme ich, Alexios Gio is, Mu lu özdemi , Thomas Bäck, and Ky iakos Gian-
nakoglou. Me amodel-assis ed e olu ion s a egies. In J J Me elo Gue ós, P Adamidis,
H G Beye , J L Fe nández-Villacañas, and H P Schwe el, edi o s, Pa allel P oblem Sol ing
om Na u e—PPSN VII, P oceedings~Se en h In e na ional Con e ence, G anada, pages 361–
370, Be lin, Heidelbe g, New Yo k, 2002. Sp inge .
[10] Alexande Fo es e , And ás Sóbes e , and Andy Keane. Enginee ing Design ia Su oga e
Modelling. Wiley, 2008.
[11] Alexande I J Fo es e and Andy J Keane. Recen ad ances in su oga e-based op imiza-
ion. P og ess in Ae ospace Sciences, 45(1-3):50–79, Janua y 2009.
[12] K B Wilson G E P Box. On he Expe imen al A ainmen o Op imum Condi ions. Jou nal
o he Royal S a is ical Socie y. Se ies B (Me hodological), 13(1):1–45, 1951.
[13] K C Giannakoglou. Design o op imal ae odynamic shapes using s ochas ic op imiza-
ion me hods and compu a ional in elligence. P og ess in Ae ospace Sciences, 38(1):43–76,
Janua y 2002.
[14] Tusha Goel, Raphael T Ha ka, Wei Shyy, and Nes o V Queipo. Ensemble o su oga es.
S uc . Mul idisc. Op im., 33(3):199–216, Sep embe 2006.
[15] Robe B G amacy. gp: An R Package o Bayesian Nons a iona y, Semipa ame ic Non-
linea Reg ession and Design by T eed Gaussian P ocess Models. Jou nal o S a is ical
So wa e, 19(9):1–46, June 2007.
[16] P Hajela and E Lee. Topological op imiza ion o o o c a sub loo s uc u es o c ash-
wo hiness conside a ions. Compu e s & S uc u es, 64(1-4):65–76, July 1997.
[17] T e o Has ie. The elemen s o s a is ical lea ning : da a mining, in e ence, and p edic ion.
Sp inge , New Yo k, 2nd ed. edi ion, 2009.
[18] Ma k Hauschild and Ma in Pelikan. An in oduc ion and su ey o es ima ion o dis i-
bu ion algo i hms. Swa m and E olu iona y Compu a ion, 1(3):111–128, Sep embe 2011.
16
[19] Jiaqiao Hu, Yongqiang Wang, Enlu Zhou, Michael C Fu, and S e en I Ma cus. A Su ey o
Some Model-Based Me hods o Global Op imiza ion. In Daniel He nández-He nández
and J Adol o Minjá ez-Sosa, edi o s, Op imiza ion, Con ol, and Applica ions o S ochas ic
Sys ems, pages 157–179. Bi khäuse Bos on, Bos on, 2012.
[20] Edwa d Huang, Jie Xu, Si Zhang, and Chun Hung Chen. Mul i- ideli y Model In eg a ion
o Enginee ing Design. P ocedia Compu e Science, 44:336–344, 2015.
[21] F ank Hu e , Holge Hoos, and Ke in Ley on-B own. An E alua ion o Sequen ial
Model-based Op imiza ion o Expensi e Blackbox Func ions. In P oceedings o he 15 h
Annual Con e ence Companion on Gene ic and E olu iona y Compu a ion, pages 1209–1216,
New Yo k, NY, USA, 2013. ACM.
[22] R Jin, W Chen, and T W Simpson. Compa a i e s udies o me amodelling echniques
unde mul iple modelling c i e ia. S uc . Mul idisc. Op im., 23(1):1–13, Decembe 2001.
[23] Y Jin. A comp ehensi e su ey o i ness app oxima ion in e olu iona y compu a ion.
So Compu ing, 9(1):3–12, Oc obe 2003.
[24] Y Jin, M Olho e , and B Sendho . On E olu iona y Op imiza ion wi h App oxima e Fi -
ness Func ions. GECCO, 2000.
[25] D R Jones, M Schonlau, and W J Welch. E icien Global Op imiza ion o Expensi e Black-
Box Func ions. Jou nal o Global Op imiza ion, 13:455–492, 1998.
[26] Jack P C Kleijnen. K iging me amodeling in simula ion: A e iew. Eu opean Jou nal o
Ope a ional Resea ch, 192(3):707–716, Feb ua y 2009.
[27] P La aaga and J A Lozano. Es ima ion o Dis ibu ion Algo i hms. A New Tool o E olu ion-
a y Compu a ion. Kluwe , Bos on MA, 2002.
[28] Minh Nghia Le, M N Le, Yew Soon Ong, Y S Ong, S Menzel, S e an Menzel, Yaochu Jin,
Y Jin, B Sendho , and Be nha d Sendho . E olu ion by adap ing su oga es. E olu iona y
Compu a ion, 21(2):313–340, 2013.
[29] S N Lopha en, H B Nielsen, and J Sønde gaa d. DACE—A Ma lab K iging Toolbox.
Technical epo , 2002.
[30] J Mockus, V Tiesis, and A Zilinskas. Bayesian Me hods o Seeking he Ex emum. In
L C W Dixon and G P Szegö, edi o s, Towa ds Global Op imiza ion, pages 117–129. Ams e -
dam, 1978.
[31] D C Mon gome y. Design and Analysis o Expe imen s. Wiley, New Yo k NY, 5 h edi ion,
2001.
[32] K P Mu phy. Machine lea ning: a p obabilis ic pe spec i e, 2012.
[33] And ea Nelson, Juan Alonso, and Thomas Pulliam. Mul i-Fideli y Ae odynamic Op i-
miza ion Using T eed Me a-Models. In Fluid Dynamics and Co-loca ed Con e ences. Ame i-
can Ins i u e o Ae onau ics and As onau ics, Res on, Vi igina, June 2007.
[34] Emanuele Oli e i. blend.py, 2012. Code a ailable on h ps://gi hub.com/
emanuele/kaggle_pb /blob/mas e /blend.py. Published unde a BSD 3 license.
[35] MJD Powell. Radial Basis Func ions. Algo i hms o App oxima ion, 1987.
[36] Mike P euss. Mul imodal Op imiza ion by Means o E olu iona y Algo i hms. Na u al Com-
pu ing Se ies. Sp inge In e na ional Publishing, Cham, 2015.
[37] F Pukelsheim. Op imal Design o Expe imen s. Wiley, New Yo k NY, 1993.
[38] Nes o V Queipo, Raphael T Ha ka, Wei Shyy, Tusha Goel, Rajkuma Vaidyana han, and
P Ke in Tucke . Su oga e-based analysis and op imiza ion. P og ess in Ae ospace Sciences,
41(1):1–28, Janua y 2005.
[39] Alain Ra le. Pa allel P oblem Sol ing om Na u e — PPSN V: 5 h In e na ional Con-
e ence Ams e dam, The Ne he lands Sep embe 27–30, 1998 P oceedings. pages 87–96.
Sp inge Be lin Heidelbe g, Be lin, Heidelbe g, 1998.
17
[40] Ma ga i a Alejand a Rebolledo Coy, Sebas ian K ey, Thomas Ba z-Beiels ein, Oli e
Flasch, And eas Fischbach, and Jö g S o k. Modeling and Op imiza ion o a Robus Gas
Senso . Technical Repo 03/2016, Cologne Open Science, Cologne, 2016.
[41] T J San ne , B J Williams, and W I No z. The Design and Analysis o Compu e Expe imen s.
Sp inge , Be lin, Heidelbe g, New Yo k, 2003.
[42] M Schonlau. Compu e Expe imen s and Global Op imiza ion. PhD hesis, Uni e si y o
Wa e loo, On a io, Canada, 1997.
[43] L Shi and K Rasheed. A Su ey o Fi ness App oxima ion Me hods Applied in E olu-
iona y Algo i hms. In Compu a ional In elligence in Expensi e Op imiza ion P oblems, pages
3–28. Sp inge Be lin Heidelbe g, Be lin, Heidelbe g, 2010.
[44] G Sun, G Li, S Zhou, W Xu, X Yang, and Q Li. Mul i- ideli y op imiza ion o shee me al
o ming p ocess. S uc u al and Mul idisciplina y ... , 2011.
[45] M J an de Laan and E C Polley. Supe Lea ne in P edic ion. UC Be keley Di ision o
Bios a is ics Wo king Pape . .., 2010.
[46] V N Vapnik. S a is ical lea ning heo y. Wiley, 1998.
[47] G Ga y Wang and S Shan. Re iew o Me amodeling Techniques in Suppo o Enginee ing
Design Op imiza ion. Jou nal o Mechanical ... , 129(4):370–380, 2007.
[48] Da id H Wolpe . S acked gene aliza ion. Neu al Ne wo ks, 5(2):241–259, Janua y 1992.
[49] Luis E Ze pa, Nes o V Queipo, Sal ado Pin os, and Jean-Louis Salage . An op imiza ion
me hodology o alkaline–su ac an –polyme looding p ocesses using ield scale nume -
ical simula ion and mul iple su oga es. Jou nal o Pe oleum Science .. ., 47(3-4):197–208,
June 2005.
[50] Z Zhou, Y S Ong, P B Nai , A J Keane, and K Y Lum. Combining Global and Local
Su oga e Models o Accele a e E olu iona y Op imiza ion. IEEE T ansac ions on Sys ems,
Man and Cybe ne ics, Pa C (Applica ions and Re iews), 37(1):66–76, 2007.
[51] Ma k Zlochin, Mau o Bi a a i, Nicolas Meuleau, and Ma co Do igo. Model-Based Sea ch
o Combina o ial Op imiza ion: A C i ical Su ey. Annals o Ope a ions Resea ch, 131(1-
4):373–395, 2004.
[52] J M Zu ada. Analog implemen a ion o neu al ne wo ks. IEEE Ci cui s and De ices Maga-
zine, 8(5):36–41, 1992.
18
!
!
CIplus
Band 5/2016
S acked Gene aliza ion o Su oga e Models -
A P ac ical App oach
Technische Be ich /
Technical Repo
P o . D . Thomas Ba z-Beiels ein
Ins i u ü In o ma ik
Fakul ä ü In o ma ik und Ingenieu wissenscha en
Technische Hochschule Köln
Mai 2016
Die Ve an wo ung ü den Inhal diese
Ve ö en lichung lieg beim Au o .
