In a Nutshell: Sequential Parameter Optimization

CIplus
Band 7/2017
In a Nu shell: Sequen ial Pa ame e
Op imiza ion
Thomas Ba z-Beiels ein, Lo enzo Gen ile, Ma in Zae e e
In a Nu shell: Sequen ial Pa ame e
Op imiza ion
Thomas Ba z-Beiels ein
homas.ba z-beiels ein@ h-koeln.de
Lo enzo Gen ile
lo enzo.gen[email p o ec ed]
Ma in Zae e e
ma in.zae e e @ h-koeln.de
SPOTSe en Lab
TH Koeln
S einm¨ulle allee 6, 51643 Gumme sbach
www.spo se en.de
In oduc ion 1
Abs ac
The pe o mance o op imiza ion algo i hms elies c ucially on hei pa ame e iza ions. Finding
good pa ame e se ings is called algo i hm uning. Using a simple simula ed annealing algo i hm,
we will demons a e how op imiza ion algo i hms can be uned using he Sequen ial Pa ame e
Op imiza ion Toolbox (SPOT). SPOT p o ides se e al ools o au oma ed and in e ac i e uning.
The unde ling concep s o he SPOT app oach a e explained. This includes key echniques such as
explo a o y i ness landscape analysis and esponse su ace me hodology. Many examples illus a e
how SPOT can be used o unde s anding he pe o mance o algo i hms and gaining insigh in o
algo i hm’s beha io . Fu he mo e, we demons a e how SPOT can be used as op imize and how a
sophis ica ed ensemble app oach is able o combine se e al me a models ia s acking.
1 In oduc ion
The pe o mance o mode n sea ch heu is ics such as e olu ion s a egies (ES), di e en ial e olu ion
(DE), o simula ed annealing (SANN) elies c ucially on hei pa ame e iza ions—o , s a is ically speak-
ing, on hei ac o se ings. Finding good pa ame e se ings o an op imiza ion algo i hm will be
e e ed o as uning. We will illus a e how an exis ing sea ch heu is ic can be uned using he Sequen-
ial Pa ame e Op imiza ion Toolbox (SPOT), which is one possible implemen a ion o he sequen ial
pa ame e op imiza ion (SPO) amewo k in oduced in Ba z-Beiels ein (2006). The e sion o SPOT
p esen ed in his a icle is implemen ed in R.Ris “a eely a ailable language and en i onmen o s a-
is ical compu ing and g aphics which p o ides a wide a ie y o s a is ical and g aphical echniques:
linea and nonlinea modelling, s a is ical es s, ime se ies analysis, classi ica ion, clus e ing, e c.” (R
Co e Team, 2017). Rcan be downloaded om h ps://c an. -p ojec .o g. The SPOT package can
be ins alled om wi hin Rusing he ins all.packages() command.
ins all.packages("SPOT")
No e, ha he package only has o be ins alled once (unless an upda e o a mo e ecen e sion is
needed). Unlike ins alla ion, he package has o be loaded o he Rwo k space e e y ime a new R
session is s a ed. SPOT can be loaded o he wo k space wi h R’s lib a y() command.
lib a y("SPOT")
In o de o keep he se up as simple as possible, we will use simula ed annealing o illus a ing
he uning p ocedu e (Ki kpa ick e al., 1983). Simula ed annealing is eely a ailable in R. This
implemen a ion o he simula ed annealing heu is ic will be e e ed o as SANN in he ollowing.
Response su ace me hodology (RSM) will be used in his a icle (Box & D ape , 1987). I can be
seen as a se o s a is ical me hods o empi ical model building. Using design o expe imen s, a esponse
(dependen a iable, ou pu a iable, o i ness alue, y) ha depends on one o se e al inpu a iables
(independen a iables o solu ions, ~x) is op imized. The unde lying model can be o mula ed as
y= (~x) + ,
whe e  ep esen s some noise (unce ain y, e o obse ed) in he esponse y. The e m esponse su ace
e e s o he su ace ep esen ed by (~x). In o de o es ima e he quali y o a solu ion, he e m i ness
is used in e olu iona y op imiza ion. In physics, he concep o a po en ial o ene gy unc ion is used.
Since we a e dealing wi h minimiza ion, a low i ness alue (~x) implies ha ~x is a good solu ion.
This epo is s uc u ed as ollows. The algo i hm- uning amewo k consis s o h ee le els, which
a e use ul o dis inguishing se e al p oblem domains. These le els a e in oduced in Sec. 2. The i s
le el is de ailed in Sec. 3. Sec ion 4 desc ibes he second le el. He e, he well known simula ed annealing
algo i hm is used o exempli y he second le el (op imiza ion algo i hm). Sec ion 5 desc ibes he hi d
le el ( uning). This uning example can be used as a s a ing poin o beginne s. De ails o he SPOT
con igu a ion a e desc ibed in Sec. 6. Visual inspec ion o he esul s plays an impo an ole in he
SPOT app oach. Sec ion 7.1 desc ibes he plo unc ions ha a e p o ided in he SPOT oolbox and
ha enable an explo a o y i ness landscape analysis. The esponse su ace me hodology is in oduced
in Sec. 8. Tools o an s a is ical analysis a e desc ibed in Sec. 9. Applying SPOT o de e minis ic
p oblems is b ie ly explained in Sec. 10. Finally, ensemble models ia s acking is desc ibed in Sec. 11.
This epo concludes wi h a summa y in Sec. 12.
spo se en.de
Le els Du ing Tuning and Op imiza ion 2
L3: Tuning Algo i hm
L2: Op imiza ion
Algo i hm
L1: Objec i e Func ion
SPOT
SANN
Sphe e
L2: Op imiza ion
Algo i hm
L1: Objec i e Func ion
SPOT
Sphe e
Fig. 1: Le : The h ee le els ha occu when uning an op imiza ion algo i hm wi h SPOT. This
se ing will be e e ed o as algo i hm uning.Righ : SPOT can be applied as an op imize . In his
case, SPOT ies o ind a gumen s o he objec i e unc ion ha esul in an op imal unc ion alue.
Following he axonomy in oduced in (Ba z-Beiels ein & Zae e e , 2017), his se ing will be e e ed
o as su oga e model based op imiza ion.
2 Le els Du ing Tuning and Op imiza ion
We will conside algo i hm uning and su oga e model based op imiza ion in his a icle. Algo i hm
uning is also e e ed o as o -line pa ame e uning in con as o on-line pa ame e uning (Eiben &
Smi h, 2003). Figu e 1 shows he di e en le els ha a e used in hese scena ios.
2.1 Algo i hm Tuning
Tuning an op imiza ion algo i hm in ol es he ollowing h ee le els can be used o desc ibe he expe i-
men al se up.
(L1) The eal-wo ld sys em. This sys em allows he speci ica ion o an objec i e unc ion, say . As an
example, we will use he sphe e unc ion in he ollowing.
(L2) The op imiza ion algo i hm, he e SANN. I equi es he speci ica ion o algo i hm pa ame e s.
(L3) The uning algo i hm, he e SPOT.
An op imiza ion algo i hm (L2) equi es pa ame e s, e.g., he ini ial empe a u e o SANN o he
mu a ion a e o e olu ion s a egies. These pa ame e s de e mine he pe o mance o he op imiza ion
algo i hms. The e o e, hey should be uned o ge be e pe o mance o one algo i hm. The algo i hm
is in u n used o de e mine op imal alues o he objec i e unc ion om le el (L1).
The e m algo i hm design summa izes ac o s ha in luence he beha io (pe o mance) o an algo-
i hm, whe eas p oblem design e e s o ac o s om he op imiza ion (simula ion) p oblem. The ini ial
empe a u e in SANN is one ypical ac o which belongs o he algo i hm design, he sea ch space
dimension belongs o he p oblem design.
2.2 Su oga e Model Based Op imiza ion
Ins ead o uning an op imiza ion algo i hm, SPOT i sel can be used as a su oga e model based
op imiza ion algo i hm. Then, SPOT has he same ole as SANN in he algo i hm uning scena io. In
his se ing, only wo le el exis : he objec i e unc ion (L1) and he op imiza ion algo i hm (L2). This
si ua ion is seen on he igh hand side o Figu e 1.
2.3 The SPOT Algo i hm
SPOT inds imp o ed solu ions in he ollowing way: ini ially, a popula ion o ( andom) solu ions is
c ea ed. The ini ializa ion s ep is shown in line 1 in Algo i hm 1. Then, he solu ions a e e alua ed
on he objec i e unc ion (le el L1). This is line 3 in Algo i hm 1. Nex , su oga e models a e buil
(line 5). A global sea ch is pe o med o gene a e new candida e solu ions (line 6). The new solu ions
spo se en.de

Le el 1: Objec i e Func ion 3
a e e alua ed on he objec i e unc ion (le el L1) (line 7). These s eps a e epea ed, un il a sa is ying
solu ion has been ound.
Algo i hm 1 Sequen ial pa ame e op imiza ion
1: = 0. P( ) = Se Ini ialPopula ion().
2: Selec one o se e al su oga e models M.
3: E alua e(P( )) on .
4: while no Te mina ionC i e ion() do
5: Use P( ) o build a model M( ) using M.
6: P0( + 1) = GlobalSea ch(M( )).
7: E alua e(P0( + 1)) on .
8: P( + 1) = P( )∪P0( + 1).
9: = + 1.
10: end while
3 Le el 1: Objec i e Func ion
Be o e SANN can be s a ed, he use has o speci y an objec i e unc ion . To keep hings as simple
as possible, he sphe e unc ion
(x) =
n
X
i=1
x2
i(1)
will be used:
sphe e <- unc ion(x){
sum(xˆ2)
}
sphe e(c(1,2))
## [1] 5
The sphe e() unc ion uses ec o inpu s. A ma ix-based implemen a ion is de ined as unSphe e()
in he SPOT package.
unSphe e
## unc ion (x)
## ma ix(apply(x, 1, unc ion(x) sum(xˆ2)), , 1)
## <en i onmen : namespace:SPOT>
A su ace plo o his unc ion in he in e al [0; 1] ×[0; 1] is shown in Fig. 2.
spo se en.de
Le el 2: Simula ed Annealing SANN 4
0.0
0.5
1.0
1.5
2.0
y
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
x1
x2
Fig. 2: Su ace plo o he sphe e unc ion.
4 Le el 2: Simula ed Annealing SANN
Simula ed annealing is a gene ic p obabilis ic heu is ic o global op imiza ion (Ki kpa ick e al., 1983).
The name comes om annealing in me allu gy. Con olled hea ing and cooling o a ma e ial educes
de ec s. Hea ing enables a oms o lea e hei ini ial posi ions (which a e local minima o hei in e nal
ene gy), and con olled cooling imp o es he p obabili y o ind posi ions wi h lowe s a es o in e nal
ene gy han he ini ial posi ions. The SANN algo i hm eplaces he cu en solu ion wi h a andomly
gene a ed new solu ion. Be e solu ions a e accep ed de e minis ically, whe e wo se solu ions a e ac-
cep ed wi h a p obabili y ha depends on he di e ence be ween he co esponding unc ion alues and
on a global pa ame e , which is commonly e e ed o as he empe a u e. The algo i hm pa ame e emp
speci ies he ini ial empe a u e o he SANN algo i hm. The empe a u e is g adually dec eased du ing
he op imiza ion. A second pa ame e , max, is used o model his cooling scheme.
We conside he Rimplemen a ion o SANN, which is a ailable ia he gene al-pu pose op imiza ion
unc ion op im() om he Rpackage s a s, which is pa o e e y Rins alla ion. The unc ion
op im() is pa ame ized as ollows
op im(pa , n, g =NULL, ..., me hod =c("Nelde -Mead","BFGS","CG","L-BFGS-B",
"SANN","B en "), lowe =-In ,uppe =In ,con ol =lis (), hessian =FALSE)
He e, pa deno es ini ial alues o he pa ame e s o be op imized o e . No e, he p oblem dimen-
sion i speci ied by he leng h o his ec o , so pa =c(1,1,1,1) deno es a ou -dimensional op imiza ion
p oblem. n is a unc ion o be minimized (o maximized), wi h i s a gumen he ec o o pa ame e s
o e which minimiza ion is o ake place. g de ines a unc ion o e u n he g adien o he“BFGS”,
“CG” and “L-BFGS-B” me hods. I i is NULL, a ini e-di e ence app oxima ion will be used. Fo he
SANN me hod i speci ies a unc ion o gene a e a new candida e poin . I i is NULL, a de aul Gaussian
Ma ko ke nel is used. The symbol . . . ep esen s u he a gumen s (op ional) ha can be be passed o
n and g . The pa ame e me hod deno es he op imiza ion me hod o be used. He e, we will use he
pa ame e alue SANN.
The pa ame e s lowe , uppe speci y bounds on he a iables o he ”L-BFGS-B” me hod, o
bounds in which o sea ch o me hod “B en ”. So, we will no use hese a iables in ou examples.
The a gumen con ol de ines a ela i ely long lis o con ol pa ame e s. We will use he ollowing
pa ame e s om his lis :
1. maxi , i.e., he maximum numbe o i e a ions, which is o SANN he maximum numbe o
spo se en.de
Le el 2: Simula ed Annealing SANN 5
unc ion e alua ions. This is he s opping c i e ion.
2. emp con ols he SANN algo i hm. I is he s a ing empe a u e o he cooling schedule wi h a
de aul alue o 10.
3. Finally, we will use max, which is he numbe o unc ion e alua ions a each empe a u e o he
SANN me hod. I s de aul alue is also 10.
To ob ain ep oducible esul s, we will se he andom numbe gene a o (RNG) seed. Using a wo-
dimensional objec i e unc ion (sphe e) and he s a ing poin (ini ial alues o he pa ame e s o be
op imized o e ) (10,10), we can execu e he op imiza ion uns as ollows:
se .seed(123)
esSANN <- op im(c(10,10), sphe e, me hod ="SANN",con ol =lis (maxi =100,
emp =10, max =10))
esSANN
## $pa
## [1] 4.835178 4.664964
##
## $ alue
## [1] 45.14084
##
## $coun s
## unc ion g adien
## 100 NA
##
## $con e gence
## [1] 0
##
## $message
## NULL
The bes , i.e., smalles , unc ion alue, which was ound by SANN, eads 45.14084. The co esponding
poin in he sea ch space is app oxima ely (4.835178, 4.664964). No g adien in o ma ion was used and
one hund ed unc ion e alua ions we e pe o med. The a iable con e gence is an in ege code, and
i s alue 0 indica es success ul comple ion o he SANN un. No addi ional message is e u ned.
Now ha we ha e pe o med a i s un o he SANN algo i hm on ou simple es unc ion, we a e
in e es ed in imp o ing SANN’s pe o mance. The SANN heu is ic equi es some pa ame e se ings,
namely emp and max. I hese alues a e omi ed, a de aul alue o en is used. The ques ions is: A e
he de aul algo i hm pa ame e se ings, namely emp =10 and max =10, adequa e o SANN o can
hese alues be imp o ed? Tha is, we a e ying o une he SANN op imiza ion algo i hm.
A ypical beginne in algo i hm uning would y o imp o e he algo i hm’s pe o mance by manually
inc easing o dec easing he algo i hm pa ame e alues, e.g., choosing emp = 20 and max = 5.
se .seed(123)
esSANN <- op im(pa =c(10,10), n = sphe e, me hod ="SANN",con ol =lis (maxi =100,
emp =20, max =5))
esSANN
## $pa
## [1] 6.163905 6.657100
##
## $ alue
## [1] 82.3107
##
## $coun s
## unc ion g adien
## 100 NA
##
spo se en.de
Le el 3: SPOT 6
## $con e gence
## [1] 0
##
## $message
## NULL
Ob iously, he manual “ uning” s ep wo sened he esul . And, his p ocedu e is e y ime consuming
and does no allow e icien s a is ical conclusions. The e o e, we will p esen a di e en app oach, which
uses he SPOT. Al hough he se up o he uning p ocedu e wi h SPOT is e y simila o he se up
discussed in his sec ion, i enables deepe insigh s in o he algo i hm’s pe o mance.
5 Le el 3: SPOT
This sec ion p esen s an example, which demons a es, how SPOT can be used o une he SANN
algo i hm de ined a le el 2. The goal o his uning p ocedu e is o de e mine imp o ed pa ame e
se ings o he SANN algo i hm. As a le el 1 unc ion, which will be op imized by SANN, he sphe e
unc ion was chosen.
Example 1 (Using Random Fo es ). Fi s , he p oblem se up o Le el 1 has o be speci ied. To keep
he si ua ion as simple as possible, we will use he sphe e es unc ion, which is de ined in Equa ion (1).
The p oblem design equi es he speci ica ion o he s a ing poin x0 o he sea ch.
x0 =c(-1,1,-1)
Since x0 has h ee elemen s, we a e acing a h ee dimensional op imiza ion p oblem. SANN will be
used o de e mine i s minimum unc ion alue.
Secondly, he p oblem se up o le el 2 has o be de ined. Again, se e al se ings ha e o be speci ied
o he SANN algo i hm o be uned. The budge , i.e., he maximum numbe o unc ion e alua ions
ha can be used by SANN is speci ied ia
maxi =100
As abo e, he Rimplemen a ion o SANN will be used ia he op im() unc ion. We will conside
wo pa ame e s:
1. he ini ial empe a u e ( emp) and
2. he numbe o unc ion e alua ions a each empe a u e ( max).
Bo h a e in ege alues. All pa ame e s and se ings o SANN which we e used o his simple example
a e summa ized in Table 1.
Thi dly, he uning p ocedu e a le el L3 has o be speci ied. To in e ace SANN wi h SPOT, a
w appe unc ion sann2spo () is used. No e, SPOT uses ma ices as he basic da a s uc u e. The
ma ix o ma was chosen as a comp omise be ween speed and lexibili y. The ma ix command can be
used as ollows:
ma ix(da a =NA,n ow =1,ncol =1,by ow =FALSE,dimnames =NULL)
The in e ace unc ion ecei es a ma ix whe e each ow is p oposed pa ame e se ing ( emp, max),
and each column speci ies he pa ame e s. I gene a es a (n, 1)-ma ix as ou pu , whe e nis he numbe
o ( emp, max) pa ame e se ings.
sann2spo <- unc ion(algpa ){
pe o mance <- NULL
o (i in 1:n ow(algpa )) {
esul Lis <- op im(pa =c(10,10), n = sphe e, me hod ="SANN",con ol =lis (maxi =100,
emp = algpa [i, 1], max = algpa [i, 2]))
pe o mance <- c(pe o mance, esul Lis $ alue)
}
e u n(ma ix(pe o mance, , 1))
}
spo se en.de
De ails 13
−2 −1 0 1 2
−2 −1 0 1 2
x2[,1]
x2[,2]
Fig. 3: Combina ion o wo designs based on La in Hype cube Sampling.
se .seed(123)
x1 <- designLHD(, c(-1,-1), c(1,1), con ol =lis (size =50, e ies =100))
x2 <- designLHD(x1, c(-2,-2), c(2,2), con ol =lis (size =50, e ies =100))
plo (x2, pch =1)
poin s(x1, pch =4)
Designs based on uni o m andom sampling can be gene a ed wi h he unc ion designUni o mRandom()
as ollows:
designUni o mRandom(, c(-1,0), c(1,10), con ol =lis (size =5))
## [,1] [,2]
## [1,] -0.59701202 3.6807802
## [2,] 0.95638068 0.4767697
## [3,] -0.17457507 4.5883452
## [4,] -0.04937504 1.3611036
## [5,] -0.96457920 4.8153870
6.4 De ails 4: Models
In SPOT, a me a model o su oga e model is used o de e mine p omising algo i hm design poin s. To
ha end, i aims o lea n he ela ion be ween algo i hm pa ame e s and he co esponding algo i hm
pe o mance. The di e en models and hei op ions a e desc ibed nex .
6.4.1 K iging
As de aul , he buildK iging unc ion is used o modeling. This unc ion builds a K iging model (also
known as Gaussian p ocess eg ession) loosely based on code by Fo es e e al. (2008). K iging models
a e based on measu es o simila i y, o ke nels. He e, a Gaussian ke nel is used:
k(x, x0) = exp −
n
X
i=1
θi|xi−x0
i|pi!
spo se en.de

De ails 14
By de aul exponen s a e ixed a a alue o wo, pi= 2∀i, and he nugge e ec (o egula iza ion
cons an ) is used. To co ec he unce ain y es ima es in case o using he nugge e ec , e-in e pola ion
is also by de aul u ned on (see Fo es e e al. (2008) o mo e de ails on hese ea u es o he model).
The ollowing code exempli ies how a K iging model is buil wi h an a i icial da a se . No e, his
example exempli ies how SPOT can be used as a su oga e model based op imiza ion algo i hm, i.e., no
algo i hm is uned.
## Tes - unc ion:
b aninFunc ion <- unc ion(x){
(x[2]-5.1/(4*piˆ2)*(x[1]ˆ2)+5/pi *x[1]-6)ˆ2+10 *(1-1/(8*
pi)) *cos(x[1]) +10
}
## C ea e design poin s
se .seed(1)
x<- cbind( uni (20)*15 -5, uni (20)*15)
## Compu e obse a ions a design poin s ( o B anin unc ion)
y<- as.ma ix(apply(x, 1, b aninFunc ion))
## C ea e model wi h de aul se ings
i <- buildK iging(x, y, con ol =lis (algThe a = op imLHD))
## P in model pa ame e s
p in ( i )
## ------------------------
## Fo es e K iging model.
## ------------------------
## Es ima ed ac i i y pa ame e s ( he a) so ed
## om mos o leas impo an a iable
## x1 x2
## 6.90722 0.4533334
##
## exponen (s) p:
## 2
##
## Es ima ed egula iza ion cons an (o nugge ) lambda:
## 1.098503e-05
##
## Numbe o Likelihood e alua ions du ing MLE:
## 600
## ------------------------
## De ine a new loca ion
newloc <- ma ix(c(1,2), n ow =1)
## P edic a new loca ion
p edic ( i , newloc)
## $y
## [1] 22.29809
## T ue alue a loca ion
b aninFunc ion(newloc)
## [1] 21.62764
##
6.4.2 Handling Fac o Va iables in he K iging Model
Some imes, pa ame e s op imized o modeled by SPOT unc ions will no be nume ical, bu a he
ca ego ical. This may, e.g., occu i an e olu iona y algo i hm is uned: while some pa ame e s like mu-
a ion a es may be eal alued, he selec ion be ween di e en mu a ion ope a o s may be a ca ego ical
pa ame e . Hence, i dimension xio a pa ame e con igu a ion xis a ac o a iable (see pa ame e
ypes), Hamming dis ance, ha de e mines he numbe o posi ions a which he co esponding alues
spo se en.de
De ails 15
a e di e en , will be used ins ead o |xi−zi|. To illus a e how ac o a iables can be handled, we c ea e
a es unc ion ha uses a ac o a iable. He e, he hi d dimension x3is ca ego ical.
b aninFunc ionFac o <- unc ion(x){
y<- (x[2]-5.1/(4*piˆ2)*(x[1]ˆ2)+5/pi *x[1]-6)ˆ2+10 *(1-
1/(8*pi)) *cos(x[1]) +10
i (x[3]== 1)
y<- y+1else i (x[3]== 2)
y<- y-1
y
}
To es how his a ec s ou model, we i s gene a e some aining da a and i he model wi h de aul
se ings, which igno es ac o in o ma ion and uses he s anda d ke nel.
se .seed(1)
## Replace x wi h new da a
x<- cbind( uni (50)*15 -5, uni (50)*15,sample(1:3,50, eplace =TRUE))
##
y<- as.ma ix(apply(x, 1, b aninFunc ionFac o ))
i De aul <- buildK iging(x, y, con ol =lis (algThe a = op imLBFGSB))
A e wa ds we i he model, which includes in o ma ion abou he ac o a iable:
i Fac o <- buildK iging(x, y, con ol =lis (algThe a = op imLBFGSB, ypes =c("nume ic",
"nume ic"," ac o ")))
We gene a e some new, unseen da a o es ing and pe o m some p edic ions wi h bo h models.
## Replace x es wi h new da a
x es <- cbind( uni (200)*15 -5, uni (200)*15,sample(1:3,200, eplace =TRUE))
##
y es <- as.ma ix(apply(x es , 1, b aninFunc ionFac o ))
## P edic es da a wi h bo h models, and compu e e o
yp edDe <- p edic ( i De aul , x es )$y
yp edFac <- p edic ( i Fac o , x es )$y
mean((yp edDe -y es )ˆ2)
## [1] 4.530052
mean((yp edFac -y es )ˆ2)
## [1] 1.514662
The e o o he ac o -awa e model is lowe . This demons a es ha use s should make su e o
decla e he na u e o he modeled a iables co ec ly, ia he ypes a iable.
6.5 De ails 5: Op imiza ion on he Me a Model
Minimiza ion by La in hype cube sampling (LHS) is he de aul op imize used o inding he nex algo-
i hm design pa ame e s on he me a model. The LHS p ocedu e gene a es a se o design poin s which
is called a La in hype cube design (LHD). The unc ion op imLHD() uses LHS o op imize a speci ied
a ge unc ion as ollows: A La in hype cube Design (LHD) is c ea ed wi h designLHD() , hen e al-
ua ed by he objec i e unc ion. All esul s a e epo ed, including he bes (minimal) objec i e alue,
and co esponding design poin . A s andalone op imiza ion un using op imLHD can be implemen ed as
ollows. I uses 100 design poin s as a de aul alue.
esOp imumLHD <- op imLHD(, un = unSphe e, lowe =c(-10,-20), uppe =c(20,
8))
s ( esOp imumLHD)
## Lis o 6
## $ x : num [1:100, 1:2] 2.55 9.62 10.73 10.49 1.43 ...
spo se en.de
De ails 16
## $ y : num [1:100, 1] 8.13 92.91 193.1 112.67 234.13 ...
## $ xbes : num [1, 1:2] 1.145 0.698
## $ ybes : num [1, 1] 1.8
## $ coun : num 100
## $ msg : ch "success"
esOp imumLHD$ybes
## [,1]
## [1,] 1.799124
Using mo e sophis ica ed algo i hms, as he a iable me ic algo i hm (L-BFGS-B), migh lead o
be e esul s. Howe e , hey a e no as obus as he simple op imLHD sea ch. Also, L-BFGS-B is a
pu e local sea ch, which may no be ideal o sol e po en ially mul i-modal uning p oblems.
esOp imBFGS <- op imLBFGSB(, un = unSphe e, lowe =c(-10,-20), uppe =c(20,
8))
esOp imBFGS$ybes
## [1] 1.98805e-30
Hence, SPOT also includes in e aces o mo e sophis ica ed algo i hms, such as di e en ial e olu ion
om DEop im package o a ious me hods included in he nlop package. Fo u he de ails on hese,
see e.g., he help o he co esponding unc ions
`?`(op imDE)
`?`(op imNLOPTR)
6.6 De ails 6: SPOT Loop: Con inued E alua ion
Some imes, use s may desi e o con inue a p e iously inished SPOT un. We will demons a e, how
SPOT can be es a ed, eusing he collec ed da a.
Example 3 (SPOT wi h con inued e alua ion). The su oga e model based op imiza ion se ing
will be used o exempli y he con inued e alua ion. The wo dimensional sphe e unc ion is used as
an objec i e unc ion (le el L1) and SPOT will be used a le el L2. SPOT uses 5 unc ion e alua ions.
con ol01 <- lis (designCon ol =lis (size =5, eplica es =1), unE als =5)
es1 <- spo (, unSphe e, lowe =c(-2,-3), uppe =c(1,2), con ol01)
cbind( es1$x, es1$y)
## [,1] [,2] [,3]
## [1,] 0.6316685 -0.3315333 0.5089193
## [2,] -1.9919658 1.7942399 7.1872244
## [3,] -1.1705672 -1.8920564 4.9501050
## [4,] 0.3218145 -2.2762891 5.2850564
## [5,] -0.5957906 0.4112744 0.5241131
Now, we con inue wi h a la ge budge . I we would like o add 3 unc ion e alua ions, he o al
numbe o unc ion e alua ions is 5 + 3 = 8. To con inue a SPOT un, he command spo Loop() can
be used as ollows: spo Loop(x, y, un, lowe , uppe , con ol, ...). The a gumen s a e
•x: he known candida e solu ions ha he SPOT loop is s a ed wi h, speci ied as a ma ix. One
ow o each poin , and one column o each op imized pa ame e .
•y: he co esponding obse a ions o each solu ion in x, speci ied as a ma ix. One ow o each
poin .
• un: is he objec i e unc ion. I should ecei e a ma ix xand e u n a ma ix y.
spo se en.de
Explo a o y Fi ness Landscape Analysis 17
•lowe : a ec o ha de ines he lowe bounda y o sea ch space. This de e mines also he dimen-
sionali y o he p oblem.
•uppe : a ec o ha de ines he uppe bounda y o sea ch space.
•con ol: a lis wi h con ol se ings o spo .
•...: addi ional pa ame e s passed o un.
con ol01$ unE als <- 8
es2 <- spo Loop( es1$x, es1$y, unSphe e, lowe =c(-2,-3), uppe =c(1,
2), con ol01)
cbind( es2$x, es2$y)
## [,1] [,2] [,3]
## [1,] 0.6316685 -0.331533262 0.50891934
## [2,] -1.9919658 1.794239861 7.18722443
## [3,] -1.1705672 -1.892056374 4.95010495
## [4,] 0.3218145 -2.276289054 5.28505643
## [5,] -0.5957906 0.411274430 0.52411310
## [6,] -1.8502780 0.025438423 3.42417578
## [7,] 0.3335189 0.311271847 0.20812503
## [8,] 0.1119788 0.009135746 0.01262271

7 Explo a o y Fi ness Landscape Analysis
7.1 In oduc ion o SPOT’s Plo Func ions
The SPOT package o e s h ee plo unc ions ha can be used o isualize da a o e alua e a model’s
pe o mance c ea ing 2D and 3D su ace plo s.
•plo Func ion() plo s unc ion objec s
•plo Da a() plo s da a
•plo Model() plo s model objec s, c ea ed by build* unc ions om he SPOT package.
7.1.1 plo Func ion
The unc ion plo Func ion() isualizes he i ness landscape. I gene a es a ( illed) con ou plo o
pe spec i e / su ace plo o a unc ion. I has se e al op ions o changing plo ing s yles, colo s e c.,
bu de aul alues should be ine in many cases. The basic a gumen s a e:
plo Func ion( , lowe , uppe , ype)
Example 4 (Plo o he sphe e unc ion). The ollowing code gene a es a 2D illed con ou plo o
he sphe e unc ion (see Fig. 4). No e ha plo Func ion() equi es a unc ion ha handles ma ix
objec s, so unSphe e() is used.
plo Func ion( unSphe e, ep(-1,2), ep(1,2))

Example 5 (Plo ing a use de ined unc ion). Figu e 5 shows he plo o he use de ined unc ion
(~x) =
n
X
i=1
(x3
i−1).(2)
I also illus a es how he colo scheme can be modi ied.
spo se en.de
Explo a o y Fi ness Landscape Analysis 18
0.0
0.5
1.0
1.5
2.0
y
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
x1
x2
Fig. 4: Example 4: 2D plo o he sphe e unc ion gene a ed wi h plo Func ion().
myFunc ion <- unc ion (x){
ma ix(apply(x, # ma ix
1,# ma gin (apply o e ows)
unc ion(x)sum(xˆ3-1)# objec i e unc ion
),
,1)# numbe o columns
}
plo Func ion(myFunc ion,
ep(-1,2),
ep(1,2),
colo .pale e = ainbow)
We can also gene a e a pe spec i e plo o he use de ined unc ion (see Fig. 6):
plo Func ion(myFunc ion, ep(-1,2), ep(1,2), ype ="pe sp", he a =10,
phi =25,bo de =NA)

7.1.2 plo Model
Fu he mo e, plo Model() o e s he possibili y o isualize models ha ha e al eady been ained, e.g.,
du ing a SPOT un. Some simple examples a e gi en below.
Example 6 (Plo ing a ained model). Fi s , we gene a e some aining da a. To demons a e how
plo s om da a wi h mo e han wo inpu dimensions can be gene a ed, we gene a e a h ee-dimensional
inpu design, i.e., we conside a unc ional ela ionship o he ype
:R3→R, y = (x1, x2, x3) (3)
The ou pu is one dimensional.
se .seed(123)
k<- 30
x. es <- designLHD(, ep(-1,3), ep(1,3), con ol =lis (size = k))
y. es <- unSphe e(x. es )
head(cbind(x. es , y. es ))
spo se en.de

Explo a o y Fi ness Landscape Analysis 19
−4
−3
−2
−1
0
y
−1.0 −0.5 0.0 0.5 1.0
−1.0
−0.5
0.0
0.5
1.0
x1
x2
Fig. 5: Example 5. Plo ing he use de ined unc ion om Equa ion 2 wi h plo Func ion() using he
colo pale e ainbow
x1
x2
y
Fig. 6: Example 5: Pe spec i e plo o he use de ined unc ion om Equa ion 2 using plo Func ion()
spo se en.de
Explo a o y Fi ness Landscape Analysis 20
0.0
0.5
1.0
1.5
2.0
y
−0.5 0.0 0.5
−0.5
0.0
0.5
x1
x2
Fig. 7: Example 6: 2D ep esen a ion o a model using plo Model()
## [,1] [,2] [,3] [,4]
## [1,] 0.698802840 0.1247362 0.57448235 0.8339145
## [2,] -0.383135671 0.2670867 -0.98091431 1.1803211
## [3,] 0.281083653 -0.7285295 0.44918649 0.8115318
## [4,] -0.488374907 0.1442807 0.32227029 0.3631851
## [5,] 0.003177575 -0.4153111 0.08761805 0.1801703
## [6,] -0.153276461 0.2490123 -0.16716223 0.1134440
Then, we ain a s anda d esponse su ace using SPOT’s buildRSM() unc ion. We gene a e he
de aul con ou plo using plo Model().
i . es <- buildRSM(x. es , y. es )
plo Model( i . es )
Passing he a gumen ype="con ou " o he plo Model() unc ion, a 2D con ou plo can be gen-
e a ed as shown in Fig. 8. By de aul , he dependen a iable yis plo ed agains he i s wo xi a iables
om Equa ion 3. No e, ha he a gumen which speci ies he independen a iables xi ha a e plo ed.
To plo yagains x1and x3, he a gumen which=c(1,3) can be used.
plo Model( i . es , which =c(1,3), ype ="con ou ",pch1 =24,col1 ="blue")
Pe spec i e plo s o he same model can be gene a ed as ollows. The a gumen s he a and phi can
be used o modi y he iew poin . The esul is shown in Fig. 9.
pa (m ow =c(1,2))
plo Model( i . es , ype ="pe sp",bo de ="NA", he a =255,phi =20)
plo Model( i . es , which =1:2, ype ="pe sp",bo de ="NA")
pa (m ow =c(1,1))

7.1.3 plo Da a
Finally, using plo Da a(), di e en models buil on p o ided da a can be compa ed. The plo Da a()
unc ion gene a es a ( illed) con ou o pe spec i e plo o a da a se wi h wo independen and one
dependen a iable. The plo is gene a ed by some in e pola ion o eg ession model. By de aul , he
loess unc ion is used. Some simple examples a e gi en below.
spo se en.de
Explo a o y Fi ness Landscape Analysis 21
y
x1
x3
0.2
0.4
0.6
0.8
1
1
1.2
1.2
1.2
1.2
1.4
1.4
1.4
1.4
1.6
1.6
1.6
1.6
1.8
1.8
1.8
−1.0 −0.5 0.0 0.5 1.0
−1.0 −0.5 0.0 0.5 1.0
Fig. 8: Example 6: 2D ep esen a ion o a model using plo Model() wi h he a gumen con ou
x1
x2
y
x1
x2
y
Fig. 9: Example 6: 3D ep esen a ions o a model using plo Model() wi h di e en iew poin s.
spo se en.de
Explo a o y Fi ness Landscape Analysis 22
Example 7 (Plo ing da a using loess and andom o es ). Figu e 10 shows a compa ison o wo
di e en models i ing he same da a.
plo Da a(x. es , y. es )
plo Da a(x. es , y. es , ype =" illed.con ou ",cex1 =1,col1 =" ed",pch1 =21,
model = buildRandomFo es )

Example 8 (Pe spec i e plo ). Figu e 11) shows a pe spec i e plo , which is based on he same da a
ha we e used in Example 7.
## Plo 3D igu e
plo Da a(x. es , y. es , ype ="pe sp",bo de =NA,model = buildLOESS)

7.2 Explo a o y Fi ness Landscape Analysis Using SPOT
This sec ion demons a es how unc ions om he SPOT package, which we e in oduced in Sec. 7.1,
can be used o pe o m a isual inspec ion o he i ness landscape du ing an in e ac i e SPOT un.
These esul s can also be used o illus a e he algo i hm’s pe o mance. In his sec ion, e e ence will
be made o he applica ion shown in sec ion 5, in which SPOT is used o he uning p ocedu e o wo
SANN design pa ame e s: s a ing empe a u e emp and he numbe o unc ion e alua ions a each
empe a u e max.
The i ness landscape can be isualized in wo di e en ways:
1. Because SPOT builds a su oga e model du ing he sequen ial op imiza ion, his model can be
used o isualize he i ness landscape. In his case, he plo Model() unc ion will be used.
2. using s anda d in e pola ion o local eg ession unc ions. In his case, he plo Da a() unc ion
will be used. No e, ha he plo Da a() unc ion allows he speci ica ion o se e al in e pola ion
unc ions (loess is de aul ).
Using plo Func ion() is usually no applicable, because he unde lying ( ue) analy ical unc ion is no
known. We conside he SPOT model based app oach i s .
7.2.1 Plo ing he Final Model wi h plo Model
Plo ing he inal model om he SPOT un migh be he mos gene ic way o isualizing he esul s,
because du ing he op imiza ion, he op imize us ed his model. So, why should i be conside ed
un eliable a e he op imiza ion is inished? Based on he uning esul s om Example 1, we will
demons a e how he inal model, which was buil du ing he SPOT un, can be plo ed. Since he
model is s o ed in he esul lis om he SPOT un, i.e., in esR , he pa ame e esR $modelFi can
be passed as an a gumen o he plo Model() unc ion. The esul is shown in Fig. 12.
plo Model( esR $modelFi )
7.2.2 Plo ing he Da a wi h plo Da a
Resul s om Example 1 we e ob ained wi h he andom o es model. Bu , he da a can be i ed o a
di e en model, e.g., a locally (weigh ed) sca e plo smoo hing (LOESS) o K iging model as ollows.
Example 9 (Plo ing da a using loess). Figu e 13 illus a es he LOESS model i , which uses he
da a gene a ed wi h a andom o es model.
plo Da a( esR $x, esR $y, model = buildLOESS)

Example 10 (Plo ing da a using K iging). Plo ing he same da a as in Fig. 13 wi h a K iging
model can easily be done. The esul is shown in Fig. 14.
spo se en.de
Response Su ace Me hodology (RSM) 29
8 Response Su ace Me hodology (RSM)
Using he sm package, which is main ained by Len h (2012), he buildRSM() unc ion builds a linea
esponse su ace model. The a gumen s o buildRSM(x, y, con ol = lis ()) a e as ollows:
•x: design ma ix (sample loca ions), ows o each sample, columns o each a iable.
•y: ec o o obse a ions a x
•con ol: (lis ), wi h he op ions o he model building p ocedu e:
–mainE ec sOnly: Logical, de aul s o FALSE. Se o TRUE i a model wi h main e ec s only
is desi ed (no in e ac ions, second o de e ec s).
–canonical Logical, de aul s o FALSE. I his is TRUE, use he canonical pa h o descen om
saddle poin s. Else, simply use s eepes descen
Example 14 (Pa h o he s eepes descen ). Fi s , we c ea e some design poin s and compu e ob-
se a ions a design poin s. Then, using buildRSM(), he esponse su ace model is build. The unc ion
descen Spo RSM() e u ns he pa h o he s eepes descen .
x<- designUni o mRandom(lowe = ep(-5,2), uppe = ep(15,2), con ol =lis (size =20))
y<- unSphe e(x)
## C ea e model wi h de aul se ings
i <- buildRSM(x, y)
## P edic new poin
p edic ( i , cbind(1,2))
## $y
## [,1]
## [1,] 5
## T ue alue a loca ion
sphe e(c(1,2))
## [1] 5
descen Spo RSM( i )
## Pa h o s eepes descen om idge analysis:
## $x
## V1 V2
## 1 4.594420 5.21888
## 2 3.955628 4.60130
## 3 3.335624 3.96680
## 4 2.725014 3.32384
## 5 2.133192 2.66396
## 6 1.560158 1.99562
## 7 0.996518 1.31036
## 8 0.461060 0.61664
## 9 -0.065004 -0.09400
## 10 -0.572280 -0.80464
##
## $y
## [,1]
## [1,] 48.34540359
## [2,] 36.81895456
## [3,] 26.86188971
## [4,] 18.47361365
## [5,] 11.64719099
## [6,] 6.41659217
## [7,] 2.71009145
## [8,] 0.59282121
## [9,] 0.01306152
## [10,] 0.97494993
spo se en.de

Response Su ace Me hodology (RSM) 30
0 5 10 15
0 5 10
V1
V2
50
100
150
200
250
300
350
Fig. 17: Example 14. Illus a ing he esponse su ace, which i s he sphe e unc ion.
This si ua ion is illus a ed in Fig. 17.
plo ( i )

8.1 RSM and SPOT
Example 15 (RSM wi h SPOT). We can use RSM o an in e ac i e uning app oach. The s a ing
poin in his example a e he 100 design poin s ha we e gene a ed du ing he K iging-based SPOT un
in Example 13. These 100 da a poin s a e used o build a esponse su ace wi h buildRSM().
sm100K <- buildRSM(x= es100K$x, y= es100K$y)
summa y( sm100K$ sm i )
##
## Call:
## sm( o mula = y ˜ FO(x1, x2) + TWI(x1, x2) + PQ(x1, x2), da a = codedDa a)
##
## Es ima e S d. E o alue P (>| |)
## (In e cep ) 50.5759 5.4677 9.2499 7.234e-15 ***
## x1 37.4445 3.9033 9.5931 1.347e-15 ***
## x2 -36.7993 4.0066 -9.1847 9.957e-15 ***
## x1:x2 -18.5259 4.1972 -4.4138 2.708e-05 ***
## x1ˆ2 -11.9005 5.6435 -2.1087 0.0376291 *
## x2ˆ2 -25.4375 6.5577 -3.8790 0.0001944 ***
## ---
## Signi . codes: 0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1
##
## Mul iple R-squa ed: 0.7836,Adjus ed R-squa ed: 0.7721
## F-s a is ic: 68.08 on 5 and 94 DF, p- alue: < 2.2e-16
##
## Analysis o Va iance Table
##
## Response: y
spo se en.de
Response Su ace Me hodology (RSM) 31
## D Sum Sq Mean Sq F alue P (>F)
## FO(x1, x2) 2 42038 21019.0 147.7727 < 2.2e-16
## TWI(x1, x2) 1 4110 4109.9 28.8940 5.546e-07
## PQ(x1, x2) 2 2269 1134.7 7.9771 0.000631
## Residuals 94 13370 142.2
## Lack o i 39 4944 126.8 0.8275 0.730891
## Pu e e o 55 8426 153.2
##
## S a iona y poin o esponse su ace:
## x1 x2
## 2.981237 -1.808933
##
## S a iona y poin in o iginal uni s:
## V1 V2
## 198.07121 -37.01983
##
## Eigenanalysis:
## eigen() decomposi ion
## $ alues
## [1] -7.196669 -30.141326
##
## $ ec o s
## [,1] [,2]
## x1 -0.8916227 0.4527792
## x2 0.4527792 0.8916227
Following he pa h o he s eepes descen on he RSM me a model, we ob ain a new design poin ,
which can be e alua ed.
(xS eep <- descen Spo RSM( sm100K))
## Pa h o s eepes descen om idge analysis:
## $x
## V1 V2
## 1 47.0845 51.384
## 2 43.8670 54.956
## 3 40.9465 58.810
## 4 38.3725 62.852
## 5 36.2935 67.223
## 6 34.7095 71.876
## 7 33.7195 76.764
## 8 33.2740 81.840
## 9 33.3235 87.057
## 10 33.8680 92.274
##
## $y
## [,1]
## [1,] 45.246195
## [2,] 39.708586
## [3,] 33.918724
## [4,] 27.953296
## [5,] 21.717621
## [6,] 15.163739
## [7,] 8.309532
## [8,] 1.063617
## [9,] -6.654501
## [10,] -14.722493
We ha e chosen he eigh h poin , i.e.,
spo se en.de
Response Su ace Me hodology (RSM) 32
xNew <- xS eep$x[8, ]
Then we de e mine i s unc ion alue.
(yNew <- sann2spo (xNew))
## [,1]
## [1,] 0.09871212
Nex , we can e ine he sm me a model by including his poin o he se o design poin s.
x101 <- bind( es100K$x, xNew)
y101 <- bind( es100K$y, yNew)
sm101K <- buildRSM(x= x101, y= y101)
summa y( sm101K$ sm i )
##
## Call:
## sm( o mula = y ˜ FO(x1, x2) + TWI(x1, x2) + PQ(x1, x2), da a = codedDa a)
##
## Es ima e S d. E o alue P (>| |)
## (In e cep ) 50.5892 5.4365 9.3054 5.045e-15 ***
## x1 37.4612 3.8772 9.6618 8.727e-16 ***
## x2 -36.8123 3.9823 -9.2440 6.825e-15 ***
## x1:x2 -18.5126 4.1719 -4.4374 2.451e-05 ***
## x1ˆ2 -11.8961 5.6137 -2.1191 0.0366875 *
## x2ˆ2 -25.4750 6.5066 -3.9153 0.0001699 ***
## ---
## Signi . codes: 0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1
##
## Mul iple R-squa ed: 0.7839,Adjus ed R-squa ed: 0.7725
## F-s a is ic: 68.91 on 5 and 95 DF, p- alue: < 2.2e-16
##
## Analysis o Va iance Table
##
## Response: y
## D Sum Sq Mean Sq F alue P (>F)
## FO(x1, x2) 2 42109 21054.5 149.5869 < 2.2e-16
## TWI(x1, x2) 1 4103 4102.8 29.1492 4.927e-07
## PQ(x1, x2) 2 2284 1141.8 8.1120 0.000559
## Residuals 95 13371 140.8
## Lack o i 40 4945 123.6 0.8069 0.759838
## Pu e e o 55 8426 153.2
##
## S a iona y poin o esponse su ace:
## x1 x2
## 2.978901 -1.804897
##
## S a iona y poin in o iginal uni s:
## V1 V2
## 197.95560 -36.83017
##
## Eigenanalysis:
## eigen() decomposi ion
## $ alues
## [1] -7.206176 -30.164858
##
## $ ec o s
## [,1] [,2]
spo se en.de
Response Su ace Me hodology (RSM) 33
0 20 40 60 80 100
0 20 40 60 80
V1
V2
−20
0
0
20
40
60
80
100
Fig. 18: Example 15. buildRSM() was used o build a esponse su ace. Da a om he K iging-based
SPOT un wi h 100 unc ion e alua ions and one addi ional poin , which was calcula ed using he
s eepes descen unc ion descen Spo RSM(), we e used o gene a e his model.
## x1 -0.8920338 0.4519686
## x2 0.4519686 0.8920338
plo ( sm101K)
Now, we ha e wo op ions o con inuing he op imiza ion. Ei he , we con inue ollowing he pa h o
he s eepes descen , i.e., we use SPOT’s descen Spo RSM() unc ion:
descen Spo RSM( sm101K)
## Pa h o s eepes descen om idge analysis:
## $x
## V1 V2
## 1 47.0845 51.384
## 2 43.8670 54.956
## 3 40.9465 58.810
## 4 38.3725 62.852
## 5 36.2935 67.223
## 6 34.7095 71.876
## 7 33.7195 76.764
## 8 33.2740 81.840
## 9 33.3730 87.057
## 10 33.8680 92.274
##
## $y
## [,1]
## [1,] 45.257160
## [2,] 39.716711
## [3,] 33.923393
## [4,] 27.953878
## [5,] 21.713329
## [6,] 15.153711
spo se en.de
S a is ical Analysis 34
0
20
40
60
80
100
120
y
20 40 60 80 100
20
40
60
80
x1
x2
Fig. 19: Example 15. buildRSM() was used o build a esponse su ace. Da a om he K iging-based
SPOT un wi h 100 unc ion e alua ions, one addi ional poin , which was calcula ed using he s eepes
descen unc ion descen Spo RSM(), and one addi ional SPOT un wi h nine addi ional design poin s,
we e used o gene a e his model. Al oge he , 110 design poin s we e used o gene a e his model.
## [7,] 8.292901
## [8,] 1.039489
## [9,] -6.656699
## [10,] -14.764062
O , we can con inue wi h SPOT. Following his second op ion, we ha e buil an upda ed K iging
model using nine addi ional unc ion e alua ions:
spo Con ig$model =buildK iging
spo Con ig$op imize =op imLBFGSB
spo Con ig$modelCon ol =lis (algThe a = op imLBFGSB)
spo Con ig$ unE als <- 110
es110K <- spo Loop(x= x101, y= y101, un = sann2spo , lowe = lowe , uppe = uppe ,
con ol = spo Con ig)
Finally, we can plo he upda ed model.
plo Model( es110K$modelFi )

9 S a is ical Analysis
This sec ion desc ibes basic app oaches o he analysis o he esul s om he p e ious example. We
will con inue using he da a om Example 12.
9.1 T ee-based Analysis
SPOT’s buildRandomFo es () unc ion is a w appe unc ion o he andomFo es unc ion om he
andomFo es package. Since he andomFo es package has no de aul plo unc ion, we swi ch o he
pa y package. This package p o ides he c ee() unc ion, which can be applied as ollows:
spo se en.de

S a is ical Analysis 35
max
p < 0.001
1
≤49 >49
max
p = 0.042
2
≤5>5
emp
p = 0.015
3
≤15 >15
n = 20
y = 1.362
4n = 22
y = 0.021
5
emp
p < 0.001
6
≤72 >72
n = 7
y = 13.475
7n = 43
y = 3.722
8
n = 8
y = 77.697
9
Fig. 20: T ee based analysis wi h K iging da a. max and emp
max empz.d <- da a. ame( es100K$x[, 1], es100K$x[, 2], es100K$y)
names( max empz.d ) <- c(" max"," emp","y")
max empz. ee <- c ee(y ˜., da a = max empz.d )
plo ( max empz. ee, ype ="simple")
9.2 Reg ession Analysis
9.2.1 Building he Linea Model
Da a om he SPOT un can be used o building linea models. Fi s , we ex ac he da a om he
esul ile. Then we can use he s anda d lm() unc ion o building he linea model.
xyz100K.d <- da a. ame( es100K$x[, 1], es100K$x[, 2], es100K$y)
names(xyz100K.d ) <- c("x","y","z")
lm100K <- lm(z ˜x*y, da a = xyz100K.d )
summa y(lm100K)
##
## Call:
## lm( o mula = z ˜ x * y, da a = xyz100K.d )
##
## Residuals:
## Min 1Q Median 3Q Max
## -62.017 -4.323 -2.869 2.339 58.572
##
## Coe icien s:
## Es ima e S d. E o alue P (>| |)
## (In e cep ) 4.017644 3.328637 1.207 0.2304
## x 1.100721 0.076897 14.314 < 2e-16 ***
## y -0.142629 0.079602 -1.792 0.0763 .
## x:y -0.009159 0.001824 -5.023 2.35e-06 ***
## ---
## Signi . codes: 0 ’***’ 0.001 ’**’ 0.01 ’*’ 0.05 ’.’ 0.1 ’ ’ 1
##
## Residual s anda d e o : 12.76 on 96 deg ees o eedom
spo se en.de
S a is ical Analysis 36
0 20 40 60 80 100
−60 0 60
Fi ed alues
Residuals
Residuals s Fi ed
42
285
−2 −1 0 1 2
−6 0 4
Theo e ical Quan iles
S anda dized esiduals
No mal Q−Q
42
2
85
0 20 40 60 80 100
0.0 1.5
Fi ed alues
S anda dized esiduals
Scale−Loca ion
42
285
0.00 0.05 0.10 0.15 0.20 0.25 0.30
−6 0 4
Le e age
S anda dized esiduals
Cook's dis ance
1
0.5
0.5
1
Residuals s Le e age
42
85
2
Fig. 21: Diagnos ic plo s o he linea eg ession model.
## Mul iple R-squa ed: 0.7469,Adjus ed R-squa ed: 0.739
## F-s a is ic: 94.42 on 3 and 96 DF, p- alue: < 2.2e-16
Diagnos ic plo s can be gene a ed as ollows:
pa (m ow =c(2,2))
plo (lm100K)
pa (m ow =c(1,1))
9.2.2 Es ima ing Va iable E ec s
R’s e mplo () unc ion can be used o plo eg ession e ms agains hei p edic o s, op ionally wi h
s anda d e o s and pa ial esiduals added.
pa (m ow =c(1,2))
e mplo (lm100K, pa ial =TRUE,smoo h = panel.smoo h, ask =FALSE)
pa (m om =c(1,1))
The ca package p o ides he unc ion a Plo s(), which can be used o isualiza ion as ollows.
pa (m ow =c(1,3))
a Plo s(lm100K, ask = F)
pa (m ow =c(1,1))
spo se en.de
S a is ical Analysis 37
0 20 40 60 80 100
−50 0 50 100
x
Pa ial o x
0 20 40 60 80
−50 0 50 100
y
Pa ial o y
Fig. 22: Example 12: Te mplo s.
−20 0 20 40 60 80
0 50 100
x | o he s
z | o he s
−60 −40 −20 0 20
−60 0 40
y | o he s
z | o he s
−2000 −1000 0 1000 2000 3000
−40 0 40
x:y | o he s
z | o he s
Added−Va iable Plo s
Fig. 23: Example 12: Added a iable plo s.
spo se en.de
Model Ensembles: S acking 38
10 De e minis ic P oblems
P e ious sec ions discussed he uning o non-de e minis ic, i.e., noisy, algo i hms, which is he usual
case when uning e olu iona y algo i hms. This sec ion p esen s an applica ion o SPOT in a simple
se ing: We will desc ibe how SPOT can be used o uning de e minis ic p oblems di ec ly. To p esen
a e y simple example, SPOT will be used o minimizing he sphe e unc ion. Le el (L2) om Fig. 1
is omi ed, and he uning algo i hm o le el (L3) is applied di ec ly o he eal-wo ld sys em on le el
(L1). So ins ead o uning SANN, which in u n op imizes he sphe e unc ion, SPOT ies o ind he
minimum o he sphe e unc ion di ec ly. This example illus a es necessa y modi ica ions o he SPOT
con igu a ion in de e minis ic se ings. Since no andomness occu s, epea s o o he mechanism o cope
wi h noise a e no necessa y anymo e.
The de e minis ic sphe e unc ion (1) is used as an example. The in e al [−5; 5]×[−5; 5] was chosen
as he egion o in e es , i.e., we a e conside ing a wo-dimensional op imiza ion p oblem.
es <- spo (, unSphe e, c(-5,-5), c(5,5), con ol =lis (op imize = op imLBFGSB))
We can ex ac he bes solu ion wi h he ollowing command.
es$xbes
## [,1] [,2]
## [1,] 0.002330654 0.0006385206
es$ybes
## [,1]
## [1,] 5.839654e-06
11 Model Ensembles: S acking
SPOT p o ides se e al models ha can be used as su oga es. Some imes i is no ob ious, which
su oga e should be chosen. Ensemble-based models p o ide a well-es ablished solu ion o his model
selec ion p oblem (Ba z-Beiels ein & Zae e e , 2017). The e o e, SPOT p o ides a s acking app oach,
ha combines se e al models in a sophis ica ed manne . The s acking p ocedu e is desc ibed in de ail
in Ba z-Beiels ein (2016).
Example 16 (S acking). We will use he da a om Example 6 o illus a e he s acking app oach.
i .s ack <- buildEnsembleS ack(x. es , y. es )
plo Model( i .s ack)
We can compa e p edic ed and ue alues as ollows.
xNew <- cbind(1,1,1)
p edic ( i .s ack, xNew)
## $y
## 1
## 2.984905
unSphe e(xNew)
## [,1]
## [1,] 3

spo se en.de