Au oma ed Gene a ion o Compu a ionally Ha d Fea u e Models using
E olu iona y Algo i hms
Se gio Segu aa,∗, Jos´
e A. Pa ejoa,∗∗, Robe M. Hie onsb, Da id Bena idesa, An onio Ruiz-Co ´
esa
aDepa men o Compu e Languages and Sys ems, Uni e si y o Se ille
A Reina Me cedes S/N, 41012 Se ille, Spain
bSchool o In o ma ion Sys ems, Compu ing and Ma hema ics, B unel Uni e si y
Uxb idge, Middlesex, UB7 7NU Uni ed Kingdom
Abs ac
A ea u e model is a compac ep esen a ion o he p oduc s o a so wa e p oduc line. The au oma ed ex ac ion
o in o ma ion om ea u e models is a h i ing opic in ol ing nume ous analysis ope a ions, echniques and ools.
Pe o mance e alua ions in his domain mainly ely on he use o andom ea u e models. Howe e , hese only p o ide
a ough idea o he beha iou o he ools wi h a e age p oblems and a e no su icien o e eal hei eal s eng hs and
weaknesses. In his a icle, we p opose o model he p oblem o inding compu a ionally ha d ea u e models as an
op imiza ion p oblem and we sol e i using a no el e olu iona y algo i hm o op imized ea u e models (ETHOM).
Gi en a ool and an analysis ope a ion, ETHOM gene a es inpu models o a p ede ined size maximizing aspec s such
as he execu ion ime o he memo y consump ion o he ool when pe o ming he ope a ion o e he model. This
allows use s and de elope s o know he pe o mance o ools in pessimis ic cases p o iding a be e idea o hei
eal powe and e ealing pe o mance bugs. Expe imen s using ETHOM on a numbe o analyses and ools ha e
success ully iden i ied models p oducing much longe execu ions imes and highe memo y consump ion han hose
ob ained wi h andom models o iden ical o e en la ge size.
Keywo ds: Sea ch-based es ing, so wa e p oduc lines, e olu iona y algo i hms, ea u e models, pe o mance
es ing, au oma ed analysis.
1. In oduc ion1
So wa e P oduc Line (SPL) enginee ing is a sys-2
ema ic euse s a egy o de eloping amilies o e-3
la ed so wa e sys ems [16]. The emphasis is on de-4
i ing p oduc s om a common se o eusable asse s5
and, in doing so, educing p oduc ion cos s and ime–6
o–ma ke . The p oduc s o an SPL a e de ined in e ms7
o ea u es whe e a ea u e is any inc emen in p od-8
uc unc ionali y [6]. An SPL cap u es he commonal-9
i ies (i.e. common ea u es) and a iabili ies (i.e. a i-10
an ea u es) o he sys ems ha belong o he p oduc 11
line. This is commonly done by using a so-called ea-12
u e model. A ea u e model [32] ep esen s he p od-13
uc s o an SPL in e ms o ea u es and ela ionships14
amongs hem (see he example in Fig. 1).15
∗P incipal co esponding au ho
∗∗Co esponding au ho
Email add esses: [email p o ec ed] (Se gio Segu a),
[email p o ec ed] (Jos´
e A. Pa ejo)
The au oma ed ex ac ion o in o ma ion om ea u e16
models (a.k.a au oma ed analysis o ea u e models) is17
a h i ing opic ha has ecei ed much a en ion in he18
las wo decades [10]. Typical analysis ope a ions allow19
us o know whe he a ea u e model is consis en (i.e.20
i ep esen s a leas one p oduc ), he numbe o p od-21
uc s ep esen ed by a ea u e model, o whe he a model22
con ains any e o s. Ca alogues wi h up o 30 anal-23
ysis ope a ions on ea u e models ha e been epo ed24
[10]. Techniques ha pe o m hese ope a ions a e yp-25
ically based on p oposi ional logic [6, 45], cons ain 26
p og amming [9, 76], o desc ip ion logic [70]. Also,27
hese analysis capabili ies can be ound in se e al com-28
me cial and open sou ce ools including AHEAD Tool29
Sui e [3], Big Le e So wa e Gea s [15], FaMa F ame-30
wo k [19], Fea u e Model Plug-in [20], pu e:: a ian s31
[53] and SPLOT [43].32
The de elopmen o ools and benchma ks o e al-33
ua e he pe o mance and scalabili y o ea u e model34
analysis ools has been ecognised as a challenge [7,35
P ep in submi ed o Else ie Ma ch 25, 2015
10, 51, 62]. Also, ecen publica ions e lec an in-36
c easing in e es in e alua ing and compa ing he pe o -37
mance o echniques and ools o he analysis o ea u e38
models [4, 25, 26, 31, 45, 39, 50, 51, 52, 55, 64, 71].39
One o he main challenges when pe o ming expe i-40
men s is inding ough p oblems ha show he s eng hs41
and weaknesses o he ools unde e alua ion in ex-42
eme si ua ions, e.g. hose p oducing longes execu-43
ion imes. Fea u e models om eal domains a e by a 44
he mos appealing inpu p oblems. Un o una ely, al-45
hough he e a e e e ences o eal ea u e models wi h46
hund eds o e en housands o ea u es [7, 37, 66], only47
po ions o hem a e usually a ailable. This lack o 48
ha d ealis ic ea u e models has led au ho s o e al-49
ua e hei ools wi h la ge andomly gene a ed ea u e50
models o 5,000 [46, 76], 10,000 [23, 45, 67, 74] and51
up o 20,000 [47] ea u es. In ac , he size o he ea-52
u e models used in expe imen s has been inc easing,53
sugges ing ha au ho s a e looking o complex p ob-54
lems on which o e alua e hei ools [10]. Mo e e-55
cen ly, some au ho s ha e sugges ed looking o ha d56
and ealis ic ea u e models in he open sou ce commu-57
ni y [13, 21, 49, 61, 62]. Fo ins ance, She e al. [62]58
ex ac ed a ea u e model con aining mo e han 5,00059
ea u es om he Linux ke nel.60
The p oblem o gene a ing es da a o e alua e he61
pe o mance o so wa e sys ems has been la gely s ud-62
ied in he ield o so wa e es ing. In his con ex ,63
esea che s ealised long ago ha andom alues a e64
no e ec i e in e ealing he ulne abili ies o a sys-65
em unde es . As poin ed ou by McMinn [42]: “ an-66
dom me hods a e un eliable and unlikely o exe cise67
‘deepe ’ ea u es o so wa e ha a e no exe cised by68
me e chance”. In his con ex , me aheu is ic sea ch69
echniques ha e p o ed o be a p omising solu ion o 70
he au oma ed gene a ion o es da a o bo h unc ional71
[42] and non– unc ional p ope ies [2]. Me aheu is ic72
sea ch echniques a e amewo ks which use heu is ics73
o ind solu ions o ha d p oblems a an a o dable com-74
pu a ional cos . Examples o me aheu is ic echniques75
include e olu iona y algo i hms, hill climbing, and sim-76
ula ed annealing [69]. Fo he gene a ion o es da a,77
hese s a egies ansla e he es c i e ion in o an ob-78
jec i e unc ion (also called a i ness unc ion) ha is79
used o e alua e and compa e he candida e solu ions80
wi h espec o he o e all sea ch goal. Using his in-81
o ma ion, he sea ch is guided owa d p omising a -82
eas o he sea ch space. Wegene e al. [72, 73] we e83
one o he i s o p opose he use o e olu iona y al-84
go i hms o e i y he ime cons ain s o so wa e back85
in 1996. In hei wo k, he au ho s used gene ic algo-86
i hms o ind inpu combina ions ha iola e he ime87
cons ain s o eal– ime sys ems, ha is, hose inpu s88
p oducing an ou pu oo ea ly o oo la e. Thei expe -89
imen al esul s showed ha e olu iona y algo i hms a e90
much mo e e ec i e han andom sea ch in inding in-91
pu combina ions maximising o minimising execu ion92
imes. Since hen, a numbe o au ho s ha e ollowed93
hei s eps using me aheu is ics and especially e olu-94
iona y algo i hms o es ing non– unc ional p ope ies95
such as execu ion ime, quali y o se ice, secu i y, us-96
abili y o sa e y [2, 42].97
P oblem desc ip ion. Cu en pe o mance e alu-98
a ions on he analysis o ea u e models a e mainly99
ca ied ou using andomly gene a ed ea u e models.100
Howe e , hese only p o ide a ough idea o he a e -101
age pe o mance o ools and do no e eal hei speci ic102
weak poin s. Thus, he SPL communi y lacks mech-103
anisms ha ake analysis ools o hei limi s and e-104
eal hei eal po en ial in e ms o pe o mance. This105
p oblem has nega i e implica ions o bo h ool use s106
and de elope s. On he one hand, ool de elope s ha e107
no means o pe o ming exhaus i e e alua ions o he108
s eng hs and weaknesses o hei ools making i ha d109
o ind aul s a ec ing hei pe o mance. On he o he 110
hand, use s a e no p o ided wi h ull in o ma ion abou 111
he pe o mance o ools in pessimis ic cases and his112
makes i di icul o hem o choose he ool ha bes 113
mee s hei needs. Hence, o ins ance, a use could114
choose a ool based on i s a e age pe o mance and la e 115
ealise ha i pe o ms e y badly in pa icula cases ha 116
appea equen ly in hei applica ion domain.117
In his a icle, we add ess he p oblem o gene a ing118
compu a ionally ha d ea u e models as a means o e-119
eal he pe o mance s eng hs and weaknesses o ea-120
u e model analysis ools. The p oblem o gene a ing121
ha d ea u e models has adi ionally been add essed122
by he SPL communi y by simply andomly gene a ing123
huge ea u e models wi h housands o ea u es and con-124
s ain s. Tha is, i is gene ally obse ed and assumed125
ha he la ge he model he ha de i s analysis. How-126
e e , we ema k ha hese models a e s ill andomly127
gene a ed and he e o e, as wa ned by so wa e es ing128
expe s, hey a e no su icien o exe cise he speci ic129
ea u es o a ool unde e alua ion. Ano he nega i e130
consequence o using huge ea u e models o e alua e131
he pe o mance o ools is ha hey equen ly all ou 132
o he scope o hei use s. Hence, bo h de elope s and133
use s would p obably be mo e in e es ed in knowing134
whe he a ool may c ash wi h a ha d model o small135
o medium size.136
Finally, we may men ion ha using ealis ic o s an-137
da d collec ions o p oblems (i.e. benchma ks) is138
equally insu icien o an exhaus i e pe o mance e al-139
2
ua ion since hey do no conside he speci ic aspec s140
o a ool o echnique unde es . Thus, ea u e mod-141
els ha one ool inds ha d o analyse could be i ially142
p ocessed by ano he and ice e sa.143
Solu iono e iewandcon ibu ions. In his a icle,144
we p opose o model he p oblem o inding compu a-145
ionally ha d ea u e models as an op imisa ion p ob-146
lem and we sol e i using a no el E olu iona y algo-147
iTHm o Op imised ea u e Models (ETHOM). Gi en148
a ool and an analysis ope a ion, ETHOM gene a es in-149
pu models o a p ede ined size maximising aspec s such150
as he execu ion ime o he memo y consumed by he151
ool when pe o ming he ope a ion o e he model. Fo 152
he e alua ion o ou app oach, we pe o med se e al153
expe imen s using di e en analysis ope a ions, ools154
and op imisa ion c i e ia. In pa icula , we used FaMa155
and SPLOT, wo ools o he au oma ed analysis o ea-156
u e models de eloped and main ained by independen 157
labo a o ies. In o al, we pe o med o e 50 million158
execu ions o analysis ope a ions o he con igu a ion159
and e alua ion o ou algo i hm, du ing mo e han six160
mon hs o wo k. The esul s showed how ETHOM suc-161
cess ully iden i ied inpu models causing much longe 162
execu ions imes and highe memo y consump ion han163
andomly gene a ed models o iden ical o e en la ge 164
size. As an example, we compa ed he e ec i eness165
o andom and e olu iona y sea ch in gene a ing ea-166
u e models wi h up o 1,000 ea u es maximising he167
ime equi ed by a cons ain p og amming sol e (a.k.a.168
CSP sol e ) o check hei consis ency. The esul s e-169
ealed ha he ha des andomly gene a ed model ound170
equi ed 0.2 seconds o analyse while ETHOM was able171
o ind se e al models aking be ween 1 and 27.5 min-172
u es o p ocess. Besides his, we ound ha he ha d-173
es ea u e models gene a ed by ETHOM in he ange174
500-1,000 ea u es we e ema kably ha de o p ocess175
han andomly gene a ed models wi h 10,000 ea u es.176
Mo e impo an ly, we ound ha he ha d ea u e mod-177
els gene a ed by ETHOM had simila p ope ies o e-178
alis ic models ound in he li e a u e. This sugges s ha 179
he long execu ion imes and high memo y consump ion180
de ec ed by ETHOM migh be ep oduced when using181
eal models wi h he consequen nega i e e ec on he182
use .183
Ou wo k enhances and complemen s he cu en 184
s a e o he a on pe o mance e alua ion o ea u e185
model analysis ools as ollows:186
•To he bes o ou knowledge, his is he i s ap-187
p oach ha uses a sea ch–based s a egy o exploi 188
he in e nal weaknesses o he analysis ools and189
echniques unde e alua ion a he han ying o190
de ec hem by chance using andomly gene a ed191
models.192
•Ou wo k allows de elope s o ocus on he sea ch193
o compu a ionally ha d models o ealis ic size194
ha could e eal pe o mance p oblems in hei 195
ools a he han using huge ea u e models ou o 196
hei scope. I a ool pe o ms poo ly wi h he gen-197
e a ed models, de elope s could use he in o ma-198
ion as inpu o in es iga e possible imp o emen s.199
•Ou app oach p o ides use s wi h help ul in o -200
ma ion abou he beha iou o ools in pessimis ic201
cases helping hem o choose he ool ha bes 202
mee s hei needs.203
•Ou algo i hm is highly gene ic and can be applied204
o any au oma ed ope a ion on ea u e models in205
which he quali y (i.e. i ness) o models wi h e-206
spec o an op imisa ion c i e ion can be quan i ied.207
•Ou expe imen al esul s show ha he ha dness o 208
ea u e models depends on di e en ac o s in con-209
as o ela ed wo k in which he complexi y o he210
models is mainly associa ed wi h hei size.211
•Ou algo i hm is eady- o-use and publicly a ail-212
able as a pa o he open-sou ce BeTTy F ame-213
wo k [14, 58].214
Scope o he con ibu ion. The a ge audience o 215
his a icle is p ac i ione s and esea che s wan ing o216
e alua e and es he pe o mance o hei ools ha 217
analyse ea u e models. Se e al aspec s ega ding he218
scope o ou con ibu ion may be cla i ied, namely:219
•Ou wo k ollows a black-box app oach. Tha 220
is, ou algo i hm does no make any assump ions221
abou an analysis ool and ope a ion unde es .222
ETHOM can he e o e be applied o any ool o 223
analysis ope a ion ega dless o how i is imple-224
men ed.225
•Ou app oach ocuses on es ing, no debugging.226
Tha is, ou wo k con ibu es o he de ec ion o 227
pe o mance ailu es (unexpec ed beha iou in he228
so wa e) bu no aul s (causes o he unexpec ed229
beha iou ). Once a ailu e is de ec ed using he230
es da a gene a ed by ETHOM, a ool’s de elop-231
e s and designe s should use debugging o iden i y232
he aul causing i , e.g. bad a iable o de ing, bad233
p oblem encoding, pa sing p oblems, e c.234
•I is no ewo hy ha many di e en ac o s could235
con ibu e o a echnique inding i ha d o analyse236
3
a gi en ea u e model, some o hem no di ec ly237
ela ed o he analysis algo i hm used. Examples238
including: bad a iable o de ing, bad p oblem en-239
coding, pa sing p oblems, bad heu is ic selec ion,240
e c. Howe e , as p e iously men ioned, he p ob-241
lem o iden i ying he ac o s ha make a ea u e242
model ha d o analyse when using a speci ic ool is243
ou o he scope o his a icle.244
The es o he a icle is s uc u ed as ollows. Sec-245
ion 2 in oduces ea u e models and e olu iona y algo-246
i hms. In Sec ion 3, we p esen ETHOM, an e olu-247
iona y algo i hm o he gene a ion o op imised ea-248
u e models. Then, in Sec ion 4, we p opose a speci ic249
con igu a ion o ETHOM o au oma e he gene a ion250
o compu a ionally ha d ea u e models. The empi i-251
cal e alua ion o ou app oach is p esen ed in Sec ion252
5. Sec ion 6 p esen s he h ea s o alidi y o ou wo k.253
Rela ed wo k is desc ibed in Sec ion 7. Finally, we sum-254
ma ise ou conclusions and desc ibe ou u u e wo k in255
Sec ion 8.256
2. P elimina ies257
2.1. Fea u e models and hei analyses258
Fea u e models de ine he alid combina ions o ea-259
u es in a domain and a e commonly used as a compac 260
ep esen a ions o all he p oduc s o an SPL. A ea u e261
model is isually ep esen ed as a ee-like s uc u e in262
which nodes ep esen ea u es and connec ions illus-263
a e he ela ionships be ween hem. These ela ion-264
ships cons ain he way in which ea u es can be com-265
bined. Fig. 1 depic s a simpli ied sample ea u e model.266
The model illus a es how ea u es a e used o speci y267
and build so wa e o Global Posi ion Sys em (GPS)268
de ices. The so wa e loaded in he GPS is de e mined269
by he ea u es ha i suppo s. The oo ea u e (i.e.270
‘GPS’) iden i ies he SPL.271
Fea u e models we e i s in oduced in 1990 as a272
pa o he FODA (Fea u e–O ien ed Domain Analysis)273
me hod [32]. Since hen, ea u e modelling has been274
widely adop ed by he so wa e p oduc line communi y275
and a numbe o ex ensions ha e been p oposed in a -276
emp s o imp o e p ope ies such as succinc ness and277
na u alness [56]. Ne e heless, he e seems o be a con-278
sensus ha a a minimum ea u e models should be able279
o ep esen he ollowing ela ionships among ea u es:280
•Manda o y. I a child ea u e is manda o y, i is281
included in all p oduc s in which i s pa en ea u e282
appea s. In Fig. 1, all GPS de ices mus p o ide283
suppo o Rou ing.284
•Op ional. I a child ea u e is de ined as op ional,285
i can be op ionally included in p oduc s in which286
i s pa en ea u e appea s. Fo ins ance, he sample287
model de ines Mul imedia o be an op ional ea-288
u e.289
•Al e na i e. Child ea u es a e de ined as al e -290
na i e i only one ea u e can be selec ed when291
he pa en ea u e is pa o he p oduc . In ou 292
SPL, so wa e o GPS de ices mus p o ide sup-293
po o ei he an LCD o Touch sc een bu only one294
o hem.295
•O -Rela ion. Child ea u es a e said o ha e an296
o - ela ion wi h hei pa en when one o mo e o 297
hem can be included in he p oduc s in which he298
pa en ea u e appea s. In ou example, GPS de-299
ices can p o ide suppo o an MP3 playe , a300
Pho o iewe o bo h o hem.301
No ice ha a child ea u e can only appea in a p od-302
uc i i s pa en ea u e does. The oo ea u e is a pa 303
o all he p oduc s wi hin he SPL. In addi ion o he304
pa en al ela ionships be ween ea u es, a ea u e model305
can also con ain c oss- ee cons ain s be ween ea u es.306
These a e ypically o he o m:307
•Requi es. I a ea u e A equi es a ea u e B, he308
inclusion o A in a p oduc implies he inclusion o 309
B in he p oduc . GPS de ices wi h T a ic a oid-310
ing equi e Au o- e ou ing.311
•Excludes. I a ea u e A excludes a ea u e B, bo h312
ea u es canno be pa o he same p oduc . In ou 313
sample SPL, a GPS wi h Touch sc een canno in-314
clude a Keyboa d and ice- e sa.315
The au oma ed analysis o ea u e models deals wi h316
he compu e -aided ex ac ion o in o ma ion om ea-317
u e models. I has been no ed ha in he o de o 30 di -318
e en analysis ope a ions on ea u e models ha e been319
epo ed du ing he las wo decades [10]. The analy-320
sis o ea u e models is usually pe o med in wo s eps.321
Fi s , he analysis p oblem is ansla ed in o an in e me-322
dia e p oblem such as a boolean sa is iabili y p oblem323
(SAT) o a Cons ain Sa is ac ion P oblem (CSP). SAT324
p oblems a e o en modelled using Bina y Decision Di-325
ag ams (BDD). Then, an o - he-shel sol e is used o326
analyse he p oblem. Mos analysis p oblems ela ed o327
ea u e models a e NP-ha d [7, 51]. Howe e , sol e s328
p o ide heu is ics ha wo k well in p ac ice. Expe i-329
men s ha e shown ha each echnique has i s s eng hs330
and weaknesses. Fo ins ance, SAT sol e s a e e icien 331
when checking he consis ency o a ea u e model bu 332
4
GPS
Rou ing In e ace
MP3 playe
3D map iew
Mul imedia
Sc een
LCDTouch
Manda o y
Op ional
Al e na i e
O
Requi es
Excludes
Pho o iewe
T a ic a oiding
Rada de ec o
Au o- e ou ing P edic i e en y Keyboa d
Figu e 1: A sample ea u e model
incapable o calcula ing he numbe o p oduc s in a333
easonable amoun o ime [11, 45, 51]. BDD sol e s334
a e he mos e icien solu ion known o calcula ing he335
numbe o p oduc s bu a he p ice o high memo y con-336
sump ion [11, 46, 51]. Finally, CSP sol e s a e espe-337
cially sui able o dealing wi h nume ic cons ain s as-338
socia ed wi h ea u e models wi h a ibu es (so-called339
ex ended ea u e models) [9].340
2.2. E olu iona y algo i hms341
The p inciples o biological e olu ion ha e inspi ed342
he de elopmen o a whole b anch o op imisa ion ech-343
niques called E olu iona y Algo i hms (EAs). These al-344
go i hms manage a se o candida e solu ions o an op i-345
misa ion p oblem ha a e combined and modi ied i e a-346
i ely o ob ain be e solu ions. Each candida e solu ion347
is e e ed o as an indi idual o ch omosome in analogy348
o he e olu ion o species in biological gene ics whe e349
he DNA o indi iduals is combined and modi ied along350
gene a ions enhancing he species h ough na u al se-351
lec ion. Two o he main p ope ies o EAs a e ha hey352
a e heu is ic and s ochas ic. The o me means ha an353
EA is no gua an eed o ob ain he global op imum o 354
he op imisa ion p oblem. The la e means ha di e -355
en execu ions o he algo i hm wi h he same inpu pa-356
ame e s can p oduce di e en ou pu , i.e. hey a e no 357
de e minis ic. Despi e his, EAs a e among he mos 358
widely used op imisa ion echniques and ha e been ap-359
plied success ully in nea ly all scien i ic and enginee -360
ing a eas by housands o p ac i ione s. This success is361
due o he abili y o EAs o ob ain nea op imal solu-362
ions o ex emely ha d op imisa ion p oblems wi h a -363
o dable ime and esou ces.364
As an example, le us conside he design o a ca as365
an op imisa ion p oblem. A simila example was used366
o illus a e he wo king o EAs in [73]. Le us suppose367
ha ou goal is o ind a ca design ha maximises368
Ini ializa ion
S op c i e ia me ?
Selec ion
Mu a ion
C osso e
E alua ion
[NOT]
[YES]
E alua ion
Encoding
Decoding
Su i al
Figu e 2: Gene al wo king scheme o e olu iona y algo i hms
speed. This p oblem is ha d since a ca is a highly369
complex sys em in which speed depends on a numbe 370
o pa ame e s such as engine ype and he shape o he371
ca . Mo eo e , he e a e likely o be ex a cons ain s372
like keeping he cos o he ca unde a ce ain alue,373
making some designs in easible. All EA a ian s a e374
based on a common wo king scheme shown in Fig. 2.375
Nex , we desc ibe i s main s eps and ela e hem o ou 376
example.377
378
Ini ialisa ion. The ini ial popula ion (i.e. se o 379
candida e solu ions o he p oblem) is usually gene a ed380
andomly. In ou example, his could be done by381
andomly choosing a se o alues o he design382
pa ame e s o he ca . O cou se, i is unlikely ha 383
his ini ial popula ion wi h con ain an op imal o 384
5

nea op imal ca design. Howe e , p omising al-385
ues ound a his s ep will be used o p oduce a ian s386
along he op imisa ion p ocess leading o be e designs.387
388
E alua ion. Nex , indi iduals a e e alua ed using a389
i ness unc ion. A i ness unc ion is a unc ion ha 390
ecei es an indi idual as inpu and e u ns a nume ical391
alue indica ing he quali y o he indi idual. This392
enables he objec i e compa ison o candida e solu ions393
wi h espec o an op imisa ion p oblem. The i ness394
unc ion should be de e minis ic o a oid in e e ences395
in he algo i hm, i.e. di e en calls o he unc ion wi h396
he same se o pa ame e s should p oduce he same397
ou pu . In ou ca example, a simula o could be used398
o p o ide he maximum speed p edic ion as i ness.399
400
S opping c i e ion. I e a ions o he emaining s eps401
o he algo i hm a e pe o med un il a e mina ion c i-402
e ion is me . Typical s opping c i e ia a e: eaching a403
maximum o a e age i ness alue, maximum execu ion404
imes o he i ness unc ion, numbe o i e a ions o 405
he loop (so-called gene a ions) o numbe o i e a ions406
wi hou imp o emen s on he bes indi idual ound.407
408
Encoding. In o de o c ea e o sp ing, an indi idual409
needs o be encoded ( ep esen ed) in a o m ha acili-410
a es i s manipula ion du ing he es o he algo i hm.411
In biological gene ics, DNA encodes an indi idual’s412
cha ac e is ics on ch omosomes ha a e used in e-413
p oduc ion and whose modi ica ions p oduce mu an s.414
Classical encoding mechanisms o EAs include he415
use o bina y ec o s ha encode nume ical alues in416
gene ic algo i hms (so-called bina y encoding) and ee417
s uc u es ha encode he abs ac syn ax o p og ams418
in gene ic p og amming (so-called ee encoding)419
[1, 54]. In ou ca example, his s ep would equi e420
design pa e ns o ca s o be exp essed using a da a421
s uc u e, e.g. bina y ec o s o each design pa ame e .422
423
Selec ion. In he main loop o he algo i hm (see Fig.424
2), indi iduals a e selec ed om he cu en popula ion425
in o de o c ea e new o sp ing. In his p ocess, be e 426
indi iduals usually ha e a g ea e p obabili y o being427
selec ed, wi h his esembling na u al e olu ion whe e428
s onge indi iduals a e mo e likely o ep oduce. Fo 429
ins ance, wo classic selec ion mechanisms a e oule e430
wheel and ou namen selec ion [1]. When using he431
o me , he p obabili y o choosing an indi idual is432
p opo ional o i s i ness and his can be seen as de e -433
mining he wid h o he slice o a hypo he ical spinning434
oule e wheel. This mechanism is o en modi ied435
by assigning p obabili ies based on he posi ion o 436
Figu e 3: Sample c osso e and mu a ion in he sea ch o an op imal
ca design.
he indi iduals in a i ness–o de ed anking (so-called437
ank-based oule e wheel). When using ou namen 438
selec ion, a g oup o nindi iduals is andomly chosen439
om he popula ion and a winning indi idual is selec ed440
acco ding o i s i ness.441
442
C osso e . These a e he echniques used o combine443
indi iduals and p oduce new indi iduals in an analo-444
gous way o biological ep oduc ion. The c osso e 445
mechanism used depends on he encoding scheme bu 446
he e a e a numbe o widely-used mechanisms [1].447
Fo ins ance, wo classical c osso e mechanisms o 448
bina y encoding a e one-poin c osso e and uni o m449
c osso e . When using he o me , a loca ion in he450
ec o is andomly chosen as he b eak poin and451
po ions o ec o s a e he b eak poin a e exchanged452
o p oduce o sp ing (see Fig. 5 o a g aphical example453
o his c osso e mechanism). When using uni o m454
c osso e , he alue o each ec o elemen is aken455
om one pa en o o he wi h a ce ain p obabili y,456
usually 50%. Fig. 3(a) shows an illus a i e applica ion457
o c osso e in ou example o ca design. An F1458
ca and a small amily ca a e combined by c osso e 459
p oducing a spo s ca . The new ehicle has some460
design pa ame e s inhe i ed di ec ly om each pa en 461
such as numbe o sea s o engine ype and o he s462
mixed such as shape and in e media e size.463
464
Mu a ion. A his s ep, andom changes a e applied o465
he indi iduals. Changes a e pe o med wi h a ce ain466
p obabili y whe e small modi ica ions a e mo e likely467
han la ge ones. Mu a ion plays he impo an ole468
o p e en ing he algo i hm om ge ing s uck p ema-469
u ely a a locally op imal solu ion. An example o 470
mu a ion in ou ca op imisa ion p oblem is p esen ed471
in Fig. 3(b). The shape o a amily ca is changed472
by adding a back spoile while he es o i s design473
pa ame e s emain in ac .474
475
6
Decoding. In o de o e alua e he i ness o new476
and modi ied indi iduals decoding is pe o med.477
Fo ins ance, in ou ca design example, da a s o ed478
on da a s uc u es is ans o med in o a sui able ca 479
design ha ou i ness unc ion can e alua e. I o en480
happens ha he changes pe o med in he c osso e 481
and mu a ion s eps c ea e indi iduals ha a e no alid482
designs o b eak a cons ain , his is usually e e ed483
o as an in easible indi idual, e.g. a ca wi h h ee484
wheels. Once an in easible indi idual is de ec ed, his485
can be ei he eplaced by an ex a co ec one o i 486
can be epai ed, i.e. sligh ly changed o make i easible.487
488
Su i al. Finally, indi iduals a e e alua ed and he nex 489
popula ion is o med in which indi iduals wi h be e 490
i ness alues a e mo e likely o emain in he popula-491
ion. This p ocess simula es he na u al selec ion o he492
be e adap ed indi iduals ha su i e and gene a e o -493
sp ing, hus imp o ing a species.494
3. ETHOM: an E olu iona y algo iTHm o Op i-495
mized ea u e Models496
In his sec ion, we p esen ETHOM, a no el e o-497
lu iona y algo i hm o he gene a ion o op imised498
ea u e models. The algo i hm akes se e al cons ain s499
and a i ness unc ion as inpu and e u ns a ea u e500
model o he gi en size maximising he op imisa ion501
c i e ion de ined by he unc ion. A key bene i o ou 502
algo i hm is ha i is e y gene ic and so is applicable503
o any au oma ed ope a ion on ea u e models in which504
he quali y (i.e. i ness) o he models can be measu ed505
quan i a i ely. In he ollowing, we desc ibe he basic506
s eps o ETHOM as shown in Fig. 2.507
508
Ini ial popula ion. The ini ial popula ion is gene a ed509
andomly acco ding o he size cons ain s ecei ed510
as inpu . The cu en e sion o ETHOM allows he511
use o speci y he numbe o ea u es, pe cen age o 512
c oss- ee cons ain s and maximum b anching ac o o 513
he ea u e model o be gene a ed. Se e al algo i hms514
o he andom gene a ion o ea u e models ha e been515
p oposed in he li e a u e [57, 67, 78]. The e a e also516
ools such as BeTTy [14, 58] and SPLOT [43, 65] ha 517
suppo he andom gene a ion o ea u e models.518
519
E alua ion. Fea u e models a e e alua ed acco ding520
o he i ness unc ion ecei ed as inpu ob aining a521
nume ic alue ha ep esen s he quali y o a candida e522
solu ion, i.e. i s i ness.523
524
0
2
13
45
6
Op,2 O ,1 M,0 O ,0 Al ,0 Al ,1 M,0
E,3,6
7
Op,0
R,6,7
TREE
CTC
Indi idual
0 1 2 3 4 5 6 7
Figu e 4: Encoding o a ea u e model in ETHOM
Encoding. Fo he ep esen a ion o ea u e models as525
indi iduals (a.k.a. ch omosomes) we p opose using a526
cus om encoding. Gene ic encodings o e olu iona y527
algo i hms we e uled ou since hese ei he we e no 528
sui able o ee s uc u es (i.e. bina y encoding) o 529
we e no able o p oduce solu ions o a ixed size (e.g.530
ee encoding), a key equi emen in ou app oach. Fig.531
4 depic s an example o ou encoding. As illus a ed,532
each model is ep esen ed by means o wo a ays,533
one s o ing in o ma ion abou he ee and ano he one534
con aining in o ma ion abou C oss-T ee Cons ain s535
(CTC). The o de o each ea u e in he a ay co e-536
sponds o he Dep h–Fi s T a e sal (DFT) o de o 537
he ee. Hence, a ea u e labelled wi h ‘0’ in he ee538
is s o ed in he i s posi ion o he a ay, he ea u e539
labelled wi h ‘1’ is s o ed he second posi ion and so540
on. Each ea u e in he ee a ay is de ined by a pai 541
<PR,C>whe e PR is he ype o ela ionship wi h542
i s pa en ea u e (M: Manda o y, Op: Op ional, O :543
O - ela ionship, Al : Al e na i e) and Cis he numbe 544
o child en o he gi en ea u e. As an example, he545
i s posi ion in he ee a ay, <Op,2>, indica es ha 546
he ea u e labelled wi h ‘0’ in he ee has an op ional547
ela ionship wi h i s pa en ea u e and has wo child548
ea u es ( hose labelled wi h ‘1’ and ‘3’). Analogously,549
each posi ion in he CTC a ay s o es in o ma ion abou 550
one cons ain in he o m <TC,O,D>whe e TC is551
he ype o cons ain (R: Requi es, E: Excludes) and552
Oand Da e he indexes o he o igin and des ina ion553
ea u es in he ee a ay espec i ely.554
555
Selec ion. Selec ion s a egies a e gene ic and can556
be applied ega dless o how he indi iduals a e557
ep esen ed. In ou algo i hm, we implemen ed bo h558
ank-based oule e-wheel and bina y ou namen 559
selec ion s a egies. The selec ion o one o he o he 560
7
E,3,6
Op,2 O ,1 M,0 O ,0 Al ,0
E,3,6 R,6,7
M,0 O ,0 Al ,0 Al ,1 M,0 Op,0Op,2 O ,1
0
2
13
45
6
7
TREE
CTC
0
2
136 7
5
Op,2 O ,1 Op,0 O ,0 Op,3 Al ,0 Al ,0
R,3,5
4
Al ,0
R,2,6
0
2
1 3
5 6
4
Al ,0 Al ,0 Al ,0
R,2,6
7
Pa en A Pa en B O sp ing
C osso e poin
0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7
Figu e 5: Example o one-poin c osso e in ETHOM
mainly depends on he applica ion domain.561
562
C osso e . We p o ided ou algo i hm wi h wo563
di e en c osso e echniques, one-poin and uni o m564
c osso e . Fig. 5 depic s an example o he applica ion565
o one-poin c osso e in ETHOM. The p ocess s a s566
by selec ing wo pa en ch omosomes o be combined.567
Fo each a ay in he ch omosomes, he ee and568
CTC a ays, a andom poin is chosen ( he so-called569
c osso e poin ). Finally, he o sp ing is c ea ed by570
copying he con en s o he a ays om he beginning571
o he c osso e poin om one pa en and he es om572
he o he one. No ice ha he cha ac e is ics o ou 573
encoding gua an ee a ixed size o he indi iduals in574
e ms o ea u es and CTCs.575
576
Mu a ion. Mu a ion ope a o s mus be speci ically de-577
signed o he ype o encoding used. ETHOM uses ou 578
di e en ypes o cus om mu a ion ope a o s, namely:579
•Ope a o 1. This andomly changes he ype580
o a ela ionship in he ee a ay, e.g. om581
manda o y,<M,3>, o op ional,<Op,3>.582
•Ope a o 2. This andomly changes he numbe o 583
child en o a ea u e in he ee, e.g. om <M,3>584
o <M,5>. The new numbe o child en is in he585
ange [0,BF] whe e BF is he maximum b anching586
ac o indica ed as inpu .587
•Ope a o 3. This changes he ype o a c oss- ee588
cons ain in he CTC a ay, e.g. om excludes589
<E,3,6> o equi es <R,3,6>.590
•Ope a o 4. This andomly changes (wi h equal591
p obabili y) he o igin o des ina ion ea u e o a592
cons ain in he CTC a ay, e.g. om <E,3,6>593
o <E,1,6>. The implemen a ion o his ensu es594
ha he o igin and des ina ion ea u es a e di e -595
en .596
These ope a o s a e applied andomly wi h he same597
p obabili y.598
599
Decoding. A his s age, he a ay-based ch omosomes600
a e ansla ed back in o ea u e models so ha hey601
can be e alua ed. In ETHOM, we iden i ied h ee602
ypes o pa e ns making a ch omosome in easible o 603
seman ically edundan , namely: i) hose encoding se 604
ela ionships (o - and al e na i e) wi h a single child605
ea u e (e.g. Fig. 6(a)), ii) hose con aining c oss- ee606
cons ain s be ween ea u es wi h pa en al ela ionship607
(e.g. Fig. 6(b)), and iii) hose con aining ea u es linked608
by con adic o y o edundan c oss- ee cons ain s609
(e.g. Fig. 6(c)). The speci ic app oach used o add ess610
in easible indi iduals, eplacing o epai ing (see611
Sec ion 2.2 o de ails), mainly depends on he p oblem612
and i is ul ima ely up o he use . In ou wo k, we used613
a epai ing s a egy desc ibed in he nex sec ion.614
615
B
A
B
A
B
A
B
ABA
B
A
BA
B
A
BA
(a)
(d)
(b)
(e)
(c)
( )
Inconsis encyRepai
Figu e 6: Examples o in easible indi iduals and epai s
Su i al. Finally, he nex popula ion is c ea ed by616
including all he new o sp ing plus hose indi iduals617
8
om he p e ious gene a ion ha we e selec ed o 618
c osso e bu did no gene a e descendan s.619
620
Fo a pseudo-code lis ing o he algo i hm we e e 621
he eade o [59].622
4. Au oma ed gene a ion o ha d ea u e models623
In his sec ion we p opose a me hod ha models he624
p oblem o inding compu a ionally ha d ea u e mod-625
els as an op imisa ion p oblem and explain how his is626
sol ed using ETHOM. In o de o ind a sui able con-627
igu a ion o ETHOM, we pe o med nume ous execu-628
ions o a sample op imisa ion p oblem e alua ing di -629
e en combina ion o alues o he key pa ame e s o 630
he algo i hm, p esen ed in Table 1. The op imisa ion631
p oblem was o ind a ea u e model maximising he632
execu ion ime aken by he analysis ool when check-633
ing model consis ency, i.e. whe he i ep esen s a leas 634
one p oduc . We chose his analysis ope a ion because635
i is cu en ly he mos equen ly quo ed in he li e a-636
u e [10]. In pa icula , we sea ched o ea u e models637
o di e en size maximising execu ion ime in he CSP638
sol e JaCoP [29] in eg a ed in o he amewo k o he639
analysis o ea u e models FaMa [19]. Nex , we cla i y640
he main aspec s o he con igu a ion o ETHOM:641
•Ini ial popula ion. We used a Ja a p og am im-642
plemen ing he algo i hm o he andom gene a-643
ion o ea u e models desc ibed by Th¨
um e al.644
[67]. Fo a de ailed desc ip ion o he gene a ion645
app oach, we e e he eade o [59].646
•Fi ness unc ion. Ou i s a emp was o mea-647
su e he ime (in milliseconds) aken by FaMa o648
pe o m he ope a ion. Howe e , we ound ha 649
he esul o he unc ion was signi ican ly a ec ed650
by he sys em load and was no de e minis ic. To651
sol e his p oblem, we decided o measu e he i -652
ness o a ea u e model as he numbe o back-653
acks p oduced by he analysis ool du ing i s anal-654
ysis. A back ack ep esen s a pa ial candida e so-655
lu ion o a p oblem ha is disca ded because i can-656
no be ex ended o a ull alid solu ion [68]. In con-657
as o he execu ion ime, mos CSP back acking658
heu is ics a e de e minis ic, i.e. di e en execu-659
ions o he ool wi h he same inpu p oduce he660
same numbe o back acks. Toge he wi h execu-661
ion ime, he numbe o back acks is commonly662
used o measu e he complexi y o cons ain sa is-663
ac ion p oblems [68]. Thus, we can assume ha 664
he highe he numbe o back acks he longe he665
compu a ion ime.666
•In easible indi iduals. We e alua ed he e ec-667
i eness o bo h eplacemen and epai echniques.668
Mo e speci ically, we e alua ed he ollowing e-669
pai algo i hm applied o in easible indi iduals: i)670
isola ed se ela ionships a e con e ed in o op-671
ional ela ionships (e.g. he model in Fig. 6(a) is672
changed as in Fig. 6(d)), ii) c oss- ee cons ain s673
be ween ea u es wi h pa en al ela ionships a e e-674
mo ed (e.g. he model in Fig. 6(b) is changed as in675
Fig. 6(e)), and iii) wo ea u es canno be linked by676
mo e han one c oss- ee cons ain (e.g. he model677
in Fig. 6(c) is changed as in Fig. 6( )).678
•S opping c i e ion. The e is no means o decid-679
ing when an op imum inpu has been ound and680
ETHOM should be s opped [73]. Fo he con ig-681
u a ion o ETHOM, we decided o allow he al-682
go i hm o con inue o a gi en numbe o execu-683
ions o he i ness unc ion (i.e. maximum numbe 684
o gene a ions) aking he la ges numbe o back-685
acks ob ained as he op imum, i.e. he solu ion o686
he p oblem.687
Table 1 depic s he alues e alua ed o each con ig-688
u a ion pa ame e o ETHOM. These alues we e based689
on ela ed wo k using e olu iona y algo i hms [23], he690
li e a u e on pa ame e se ing [18], and ou p e ious691
expe ience in his domain [48]. Each combina ion o 692
pa ame e s used was execu ed 10 imes o a oid he e o-693
geneous esul s and o allow us o pe o m s a is ical694
analysis on he da a. The alues unde lined a e hose695
ha p o ided be e esul s and we e he e o e selec ed696
o he inal con igu a ion o ETHOM. In o al, we pe -697
o med o e 40 million execu ions o he objec i e unc-698
ion o ind a good se up o ou algo i hm.699
Pa ame e Values e alua ed and selec ed
Selec ion s a egy Roule e-wheel, 2-Tou namen
C osso e s a egy One-poin , Uni o m
C osso e p obabili y 0.7, 0.8, 0.9
Mu a ion p obabili y 0.005, 0.0075, 0.02
Size ini ial popula ion 50, 100, 200
#Execu ions i ness unc ion 2000, 5000
In easible indi iduals Replacing, Repai ing
Table 1: ETHOM con igu a ion
5. E alua ion700
In o de o e alua e ou app oach, we de eloped a701
p o o ype implemen a ion o ETHOM. The p o o ype702
was implemen ed in Ja a o acili a e i s in eg a ion in o703
9
CSP Sol e SAT Sol e BDD Sol e
Modelling elemen Min A g Max Min A g Max Min A g Max
% ela i e o no. o ea u es
Manda o y 25.3 27.9 31.0 20.0 25.1 28.0 10.0 17.1 24.8
Op ional 27.5 34.9 45.0 30.5 36.9 44.0 18.0 35.7 46.5
Se sub ea u es 29.0 37.0 41.5 31.0 37.8 45.5 34.5 46.3 62.0
Se ela ionships 11.0 14.1 16.0 12.0 13.8 15.3 13.3 16.1 20.0
- O 5.5 7.0 9.0 5.5 7.1 8.3 6.0 8.9 12.0
- Al e na i e 5.5 7.1 8.5 4.0 6.7 8.8 3.3 7.2 10.0
% ela i e o no. o cons ain s
Requi es 31.3 47.5 56.6 41.1 51.9 68.4 31.0 48.5 64.3
Excludes 43.4 52.5 68.7 31.6 48.1 58.9 35.7 51.5 69.0
Table 5: P ope ies o he ha des ea u e models ound in ou expe imen s.
we e i ially analysed in a ew seconds. Then, we e-1145
pea ed he analysis o he ha des ea u e models ound1146
in Expe imen #1 using he o he se en heu is ics a ail-1147
able in he CSP sol e JaCoP. The esul s e ealed ha 1148
he ha des ea u e models ound in ou expe imen , us-1149
ing he heu is ic Mos Cons ainedDynamic, we e i -1150
ially sol ed by some o he o he s heu is ics. Fo exam-1151
ple, he ha des model in he ange o 800 ea u es and1152
10% CTC p oduced 5.3 million back acks when us-1153
ing he heu is ic Mos Con ainedDynamic and only 431154
back acks when using he heu is ic Smalles Min. This1155
inding clea ly shows ha ea u e models ha a e ha d1156
o analyse by one ool o echnique could be i ially1157
p ocessed by o he s and ice- e sa. Hence, we con-1158
clude ha using a s anda d se o p oblems, andomly1159
gene a ed o no , is no su icien o a ull e alua ion1160
o he pe o mance o di e en ools. Ins ead, as in1161
ou app oach, he echniques and ools unde e alua ion1162
should be exe cised o iden i y hei s eng hs and weak-1163
nesses p o iding help ul in o ma ion o bo h use s and1164
de elope s.1165
The a e age e ec i eness o ou app oach anged1166
om 85.8% o 94.4% in all he expe imen s. As ex-1167
pec ed om an e olu iona y algo i hm, we ound ha 1168
hese a ia ions in he e ec i eness we e caused by he1169
cha ac e is ics o he sea ch spaces o each p oblem.1170
In pa icula , ETHOM beha es be e when he sea ch1171
space is he e ogeneous and he e a e many di e en i -1172
ness alues, i.e. i is easy o compa e he quali y o 1173
he indi iduals. Howe e , esul s ge wo se in homo-1174
geneous sea ch spaces in which mos i ness alues a e1175
equal (e.g. Expe imen #1, ange o 10% o CTCs).1176
A common s a egy o alle ia e his p oblem is o use1177
a la ge popula ion, inc easing he chances o he al-1178
go i hm inding p omising indi iduals du ing ini ialisa-1179
ion. We also ound ha he maximum imeou o 301180
minu es was insu icien in some size anges (e.g. Ex-1181
pe imen #2, 250 ea u es and 30% CTCs), ad e sely1182
a ec ing he esul s. Inc easing his imeou would ha e1183
ce ainly inc eased he e ec i eness o ETHOM a he1184
p ice o making ou expe imen s mo e ime-consuming.1185
Finally, as a sa e y check, we es ed ETHOM wi h1186
di e en op imisa ion p oblems. In pa icula , we used1187
p oblems wi h a known global maximum whe e he e -1188
icacy o ETHOM was easie o obse e. Fo ins ance,1189
we used ETHOM o sea ch o ea u e models wi h1190
n ea u es and m% o CTCs ha ep esen as many1191
p oduc s as possible, 2nbeing he maximum. In e es -1192
ingly, he algo i hm p og essi ely emo ed he ela ion-1193
ships cons aining he se o p oduc s (i.e. manda o y1194
and al e na i e), gene a ing models wi h op ional and1195
o - ela ionships only. This demons a es he abili y o 1196
ETHOM o change he model i ha helps o make i 1197
be e o he gi en p oblem. This and o he examples1198
a e a ailable as a pa o he BeTTy es ing amewo k1199
[14].1200
5.4. S a is ical analysis1201
S a is ical analysis is usually pe o med by o mula -1202
ing wo con a y hypo heses. The i s hypo hesis is e-1203
e ed o as he null hypo hesis (Hi
0) and says ha he1204
algo i hm has no impac a all on he goodness o he e-1205
sul s ob ained, i.e. he e is no di e ence be ween he e-1206
sul s ob ained by ETHOM and andom sea ch. Opposi e1207
o he null hypo hesis, an al e na i e hypo hesis (Hi
1) is1208
o mula ed, s a ing ha ETHOM has a signi ican e -1209
ec in he quali y o he esul s ob ained. S a is ical1210
es s p o ide a p obabili y (named p- alue) anging in1211
[0,1]. A low p- alue indica es ha he null hypo hesis is1212
p obably alse and he al e na i e hypo hesis is p obably1213
ue, i.e. ETHOM wo ks. Al e na i ely, high p- alues1214
sugges ha ETHOM does no wo k. Resea che s ha e1215
es ablished by con en ion ha p- alues unde 0.05 o 1216
16

0.01 a e so-called s a is ically signi ican and a e su -1217
icien o ejec he null hypo hesis, i.e. demons a e1218
ha ETHOM p o ides be e esul s ha andom sea ch.1219
The s a is ical analysis desc ibed in his sec ion was pe -1220
o med using he SPSS 17 s a is ical package [28].1221
The echniques used o pe o m he s a is ical analy-1222
sis and ob ain he p- alues depend on whe he he da a1223
ollows a no mal equency dis ibu ion o no . A e 1224
some p elimina y es s (Kolmogo o -Smi no [35, 63]1225
and Shapi o-Wilk [60] es s) we concluded ha ou 1226
da a did no ollow a no mal dis ibu ion and hus ou 1227
es s equi ed he use o so-called non–pa ame ic ech-1228
niques. In pa icula , we applied he Mann-Wi hney U1229
non–pa ame ic es [41] o he expe imen al esul s ob-1230
ained wi h ETHOM and andom sea ch. Tables A.61231
and A.7 show he esul s o hese es s in SPSS o 1232
Expe imen s #1 and #2 espec i ely. Fo each num-1233
be o ea u es and pe cen age o c oss- ee cons ain s,1234
he alues o he es a e p o ided. As illus a ed, he1235
es s ejec ed he null hypo heses wi h ex emely low p-1236
alues (ze o in mos cases) o nea ly all expe imen al1237
con igu a ions o bo h expe imen s. This, coupled wi h1238
he esul s shown in Sec ion 5, clea ly shows he su-1239
pe io i y o ou algo i hm when compa ed o andom1240
sea ch. As expec ed, s a is ical es s accep ed some null1241
hypo heses in he ange o 10% o CTCs in Expe imen 1242
#1. As explained in Sec ion 6, his is due o he small1243
complexi y o he analysis on hose models which made1244
he i ness landscape ex emely la . Simila ly, he es s1245
accep ed some null hypo heses in he ange o 250 ea-1246
u es and 30% o CTCs in Expe imen #2. This was1247
due o he maximum imeou o 30 minu es used o ou 1248
expe imen s ha made ou algo i hm s op p ema u ely,1249
s opping i om e ol ing owa d p omising solu ions.1250
Fo a mo e de ailed explana ion o ou s a is ical anal-1251
ysis o he da a we e e he eade o [59].1252
6. Th ea s o alidi y1253
In o de o clea ly delinea e he limi a ions o he1254
expe imen al s udy, nex we discuss in e nal and1255
ex e nal alidi y h ea s.1256
1257
In e nal alidi y. This e e s o whe he he e is1258
su icien e idence o suppo he conclusions and1259
he sou ces o bias ha could comp omise hose1260
conclusions. In o de o minimise he impac o 1261
ex e nal ac o s in ou esul s, ETHOM was execu ed1262
25 imes o each p oblem o ge a e ages. Mo eo e ,1263
s a is ical es s we e pe o med o ensu e signi icance1264
o he di e ences iden i ied. Rega ding he andom1265
gene a ion o ea u e models, we a oided he isk o 1266
c ea ing syn ac ically inco ec models as ollows.1267
Fi s , we used a publicly a ailable (and p e iously1268
used) algo i hm o he andom gene a ion o ea u e1269
models. Second, we pe o med se e al checks using1270
he pa se o BeTTy, FaMa and SPLOT o make su e1271
ha he gene a ed models we e syn ac ically co ec 1272
and had he desi ed p ope ies, e.g. a maximum1273
b anching ac o . A ela ed isk is he possibili y o ou 1274
andom and e olu iona y algo i hms ha ing di e en 1275
exp essi eness, e.g. ee pa e ns ha can be gene a ed1276
wi h ETHOM bu no wi h ou andom algo i hm. To1277
minimise his isk, we imposed he same gene a ion1278
cons ain s on bo h ou andom and e olu iona y1279
gene a o s. Mo e speci ically, bo h gene a o s ecei ed1280
exac ly he same inpu cons ain s: numbe o ea u es,1281
pe cen age o CTC and maximum b anching ac o 1282
o he model o be gene a ed. Also, bo h gene a o s1283
p ohibi he gene a ion o CTCs be ween ea u es wi h1284
pa en al ela ion and ea u es linked by mo e han1285
one CTC. A ela ed limi a ion o he cu en ETHOM1286
encoding is ha i does no allow he e o be mo e han1287
one se ela ionship o he same ype (e.g. al e na i e1288
g oup) unde a pa en ea u e. Hence, o ins ance,1289
i wo al e na i e g oups a e loca ed unde he same1290
ea u e, hese a e me ged in o one du ing decoding.1291
We may ema k, howe e , ha his only a ec s he1292
exp essi eness o ETHOM pu ing i a a disad an age1293
agains andom sea ch. Also, he esul s do no e eal1294
any co ela ion be ween he numbe o se ela ionships1295
and he ha dness o he models which means ha his1296
es ic ion did no bene i ou algo i hm. Besides his,1297
he esul s show ha ETHOM is equally capable o 1298
gene a ing consis en o inconsis en models i ha 1299
make hem ha de o he a ge sol e . The e o e, i 1300
seems unlikely ha ou algo i hm has a endency o1301
gene a e only consis en o inconsis en models.1302
1303
Ex e nal alidi y. This is conce ned wi h how he ex-1304
pe imen s cap u e he objec i es o he esea ch and he1305
ex en o which he conclusions d awn can be gene -1306
alised. This can be mainly di ided in o limi a ions o 1307
he app oach and gene alizabili y o he conclusions.1308
Rega ding he limi a ions, he expe imen s showed1309
no signi ican imp o emen s when using ETHOM wi h1310
p oblems o low complexi y, i.e. ea u e models wi h1311
10% o cons ain s in Expe imen #1. As s a ed in Sec-1312
ion 5.1, his limi a ion is due o he i ness landscape1313
being ela i ely la o simple p oblems; mos i ness1314
alues a e ze o o close o ze o. Ano he limi a ion o 1315
he expe imen al app oach is ha expe imen s o ex-1316
emely ha d ea u e models become oo ime consum-1317
ing, e.g. ea u e models wi h 250 ea u es in Expe i-1318
17
men #2. This h ea is caused by he na u e o he ha d1319
ea u e models we in end o ind, wi h he analysis o 1320
p omising ea u e models becoming inc easingly ime1321
consuming and memo y in ensi e. We may ema k,1322
howe e , ha his limi a ion is in insic o he p oblem1323
o looking o ha d ea u e models and hus i equally1324
a ec s andom sea ch. Finally, we emphasise ha in1325
he wo s case ETHOM beha es andomly equalling he1326
s a egies o he gene a ion o ha d ea u e models used1327
in he cu en s a e o he a .1328
Rega ding he gene alisa ion o he conclusions, we1329
used wo di e en analysis ope a ions and he esul s1330
migh no gene alise u he . We ema k, howe e ,1331
ha hese ope a ions a e cu en ly he mos equen ly1332
quo ed in he li e a u e, ha e di e en complexi y and,1333
mo e impo an ly, a e he basis o he implemen a ion1334
o many o he analysis ope a ions on ea u e models1335
[10]. Thus, ea u e models ha a e ha d o analyse1336
o hese ope a ions would ce ainly be ha d o anal-1337
yse o hose ope a ions ha use hem as an auxilia y1338
unc ion making ou esul s ex ensible o o he analy-1339
ses. Simila ly, we only used wo analysis ools o he1340
expe imen s, FaMa and SPLOT. Howe e , hese ools1341
a e de eloped and main ained by independen labo a-1342
o ies p o iding a su icien deg ee o he e ogenei y o 1343
ou s udy. Also, he esul s e ealed ha a numbe o 1344
me ics o he gene a ed models (e.g. pe cen age o 1345
CTCs) we e in he anges obse ed in ealis ic models1346
ound in he li e a u e, which suppo s he ealism o he1347
ha d ea u e models being gene a ed. We may ema k,1348
howe e , ha hese models could s ill con ain s uc u es1349
ha a e unlikely in eal-wo ld models and he e o e his1350
issue equi es u he esea ch. Finally, ou andom and1351
e olu iona y gene a o s do no allow wo ea u es o be1352
linked by mo e han one CTC o simplici y (see Sec ion1353
4). This implici ly p ohibi s he gene a ion o cycles o 1354
equi es cons ain s, i.e. A−>Band B−>A. How-1355
e e , hese cycles exp ess equi alence ela ionships and1356
seem o appea in eal models (e.g. Linux ke nel ea-1357
u e model [49]) which could sligh ly a ec he gene -1358
alisa ion o ou esul s. These cycles will be allowed in1359
u u e e sions o ou algo i hm.1360
7. Rela ed wo k1361
In his sec ion we discuss ela ed wo k in he a eas o 1362
so wa e p oduc lines and sea ch-based es ing.1363
7.1. So wa e p oduc lines1364
A numbe o au ho s ha e used ealis ic ea u e mod-1365
els o e alua e hei ools [4, 9, 24, 26, 31, 33, 46,1366
45, 50, 51, 55, 64, 67, 70]. By ealis ic models we1367
mean hose modelling eal–wo ld domains o a sim-1368
pli ied e sion o hem. Some o he ealis ic ea u e1369
models mos quo ed in he li e a u e a e e-Shop [36]1370
wi h 287 ea u es, g aph p oduc line [38] wi h up o1371
64 ea u es and Be keleyDB [34] wi h 55 ea u es. Al-1372
hough he e a e epo s om indus y o ea u e models1373
wi h hund eds o e en housands o ea u es [7, 37, 66],1374
only a po ion o hem is ypically published. This has1375
led au ho s o gene a e ea u e models au oma ically1376
o show he scalabili y o hei app oaches wi h la ge1377
p oblems. These models a e gene a ed ei he andomly1378
[12, 11, 22, 26, 44, 47, 57, 74, 75, 76, 78, 79] o using a1379
p ocess ha ies o p oduce models wi h he p ope ies1380
o hose ound in he li e a u e [23, 45, 64, 67]. Mo e e-1381
cen ly, some au ho s ha e sugges ed looking o ough1382
and ealis ic ea u e models in he open sou ce commu-1383
ni y [13, 21, 49, 61, 62]. As an example, She e al. [62]1384
ex ac ed a ea u e model om he Linux ke nel con-1385
aining mo e han 5,000 ea u es and compa ed i wi h1386
publicly a ailable ealis ic ea u e models.1387
Rega ding he size o he models used o expe i-1388
men s, he e is a clea endency o model size o in-1389
c ease: his anges om he model wi h 15 ea u es used1390
in 2004 [8] o models wi h up o 10,000 and 20,000 ea-1391
u es used in ecen yea s [23, 45, 47, 67, 74]. These1392
indings e lec an inc easing in e es in using complex1393
ea u e models in pe o mance e alua ion. This also1394
sugges s ha he only mechanism used o inc ease he1395
complexi y o he models is by inc easing size. When1396
compa ed o p e ious wo k, ou app oach is he i s o1397
use a sea ch–based s a egy o e eal he pe o mance1398
weaknesses o he ools and echniques unde e alua-1399
ion a he han simply using la ge andomly gene a ed1400
models. This allows de elope s o ocus on he sea ch1401
o ough models o ealis ic size ha could e eal de-1402
iciencies in hei ools a he han using huge ea u e1403
models ou o hei scope. Simila ly, use s could ha e1404
mo e in o ma ion abou he expec ed beha iou o he1405
ools in pessimis ic cases helping hem o choose he1406
ool o echnique ha bes mee s hei needs.1407
The applica ion o op imisa ion algo i hms in he1408
con ex o so wa e p oduc lines has been explo ed by1409
se e al au ho s. Guo e al. [23] p oposed a gene ic al-1410
go i hm called GAFES o op imised ea u e selec ion1411
in ea u e models, e.g. selec ing he se o ea u es1412
ha minimises he o al cos o he p oduc . Sayyad1413
e al. [55] compa ed he e ec i eness o i e mul i-1414
objec i e op imiza ion algo i hms o he selec ion o 1415
op imised p oduc s. O he au ho s [25, 39, 71] ha e1416
p oposed algo i hms o he selec ion o es sui es (i.e.1417
se o p oduc s) maximising o minimising ce ain p e -1418
18
e ences, e.g. ea u e co e age. Compa ed o hei 1419
wo k, ou app oach di e s in se e al aspec s. Fi s , ou 1420
wo k add esses a di e en p oblem domain, ha d ea-1421
u e model gene a ion. Second, and mo e impo an ly,1422
ETHOM sea ches o op imum ea u e models while1423
ela ed algo i hms sea ch o op imum p oduc con ig-1424
u a ions. This means ha ETHOM and ela ed algo-1425
i hms bea no esemblance and ace comple ely di e -1426
en challenges. Fo ins ance, ela ed algo i hms use a1427
s anda d bina y encoding o ep esen p oduc con igu-1428
a ions while ETHOM uses a cus om a ay encoding o1429
ep esen ea u e models o ixed size.1430
Pohl e al. [51] p esen ed a pe o mance compa ison1431
o nine CSP, SAT and BDD sol e s on he au oma ed1432
analysis o ea u e models. As inpu p oblems, hey1433
used 90 ealis ic ea u e models wi h up o 287 ea u es1434
aken om he SPLOT eposi o y [65]. The longes 1435
execu ion ime ound in he consis ency ope a ion was1436
23.8 seconds, a om he 27.5 minu es ound in ou 1437
wo k. Memo y consump ion was no e alua ed. As pa 1438
o hei wo k, he au ho s ied o ind co ela ions be-1439
ween he p ope ies o he models and he pe o mance1440
o he sol e s. Among o he esul s, hey iden i ied an1441
exponen ial un ime inc ease wi h he numbe o ea-1442
u es in CSP and SAT sol e s. This is no suppo ed1443
by ou esul s, a leas no in gene al, since we ound1444
ea u e models p oducing much longe execu ion imes1445
han la ge andomly gene a ed models. Also, he au-1446
ho s men ioned ha SAT and CSP sol e s p o ided a1447
simila pe o mance in hei expe imen . This was no 1448
obse ed in ou wo k in which he SAT sol e was much1449
mo e e icien han he CSP sol e , i.e. andom and1450
e olu iona y sea ch we e unable o ind ha d p oblems1451
o SAT. O e all, we conside ha using ealis ic ea-1452
u e models is help ul bu no su icien o an exhaus-1453
i e e alua ion o he pe o mance o sol e s. In con-1454
as , ou wo k p o ides he communi y wi h a limi less1455
sou ce o mo i a ing p oblems o explo e he s eng hs1456
and weaknesses o analysis ools.1457
In la e wo k, Pohl e al. [52] p oposed using wid h1458
measu es om g aph heo y o cha ac e ise he s uc-1459
u al complexi y o ea u e models as a way o es ima e1460
he di icul y in analysing hem. They pe o med se e al1461
expe imen s unning he consis ency ope a ion on an-1462
domly gene a ed models o up o 1,000 ea u es in nine1463
s a e o he a CSP, SAT and BDD sol e s. As a esul ,1464
o some o he sol e s hey ound a co ela ion be ween1465
one o he me ics and he ime aken by he analysis.1466
When compa ed o hei wo k, ETHOM uses a black-1467
box s a egy and hus i may be used o ind ha d inpu 1468
ea u e models o any analysis ool o analysis ope a-1469
ion ega dless o hei implemen a ion de ails. Fu he -1470
mo e, ETHOM explo es he whole sea ch space o ea-1471
u e models, no only hose wi h di e en wid h p op-1472
e ies, in looking o inpu p oblems ha inc ease he1473
execu ion imes o analysis ools. Ha ing said his, we1474
hink ha bo h wo ks a e complemen a y since ETHOM1475
gene a es ha d ea u e models and hei app oach ies o1476
de e mine wha makes he models ha d o analyse.1477
Du ing he p epa a ion o his a icle, we p esen ed a1478
no el applica ion o ETHOM in he con ex o e e se1479
enginee ing o ea u e models [40]. Mo e speci ically,1480
we used ETHOM o sea ch o a ea u e model ha ep-1481
esen s a speci ic se o p oduc s p o ided as inpu . The1482
esul s showed ha wi hin a ew gene a ions ou algo-1483
i hm was able o ind ea u e models ha ep esen a1484
supe se o he desi ed p oduc s. This con ibu ion sup-1485
po s ou claims abou he gene alisabili y o ou algo-1486
i hm showing i s applicabili y o o he domains beyond1487
he analysis o ea u e models.1488
Finally, we would like o ema k ha ou app oach1489
does no in end o eplace he use o ealis ic o an-1490
domly gene a ed models which can be used o e alu-1491
a e he a e age pe o mance o analysis echniques. In-1492
s ead, ou wo k complemen s p e ious app oaches en-1493
abling a mo e exhaus i e e alua ion o he pe o mance1494
o analysis ools using ha d p oblems.1495
7.2. Sea ch-based es ing1496
Rega ding ela ed wo k in sea ch-based es ing, We-1497
gene e al. [72] we e he i s o use gene ic algo i hms1498
o sea ch o inpu alues ha p oduce e y long o e y1499
sho execu ion imes in he con ex o eal ime sys ems.1500
In hei expe imen s, hey used C p og ams ecei ing1501
hund eds o e en housands o in ege pa ame e s. Thei 1502
esul s showed ha gene ic algo i hms ob ained mo e1503
ex eme execu ion imes wi h equal o less es e o 1504
han andom es ing. Ou app oach may be conside ed a1505
speci ic applica ion o he ideas o Wegene and la e au-1506
ho s o he domain o ea u e modelling. In his sense,1507
ou main con ibu ion is he de elopmen and con igu a-1508
ion o a no el e olu iona y algo i hm o deal wi h op i-1509
misa ion p oblems on ea u e models and i s applica ion1510
o pe o mance es ing in his domain.1511
Many au ho s con inued he wo k o Wegene e al.1512
in he applica ion o me aheu is ic sea ch echniques o1513
es non- unc ional p ope ies such as execu ion ime,1514
quali y o se ice, secu i y, usabili y o sa e y [2]. The1515
echniques used by he sea ch-based es ing communi y1516
include, among o he s, hill climbing, an colony op i-1517
misa ion, abu sea ch and simula ed annealing. In ou 1518
app oach, we used e olu iona y algo i hms inspi ed by1519
he wo k o Wegene e al. and hei p omising esul s in1520
a ela ed op imisa ion p oblem, i.e. gene a ion o inpu 1521
19
alues maximising he execu ion ime in eal ime sys-1522
ems. We ema k, howe e , ha he use o o he me a-1523
heu is ic echniques o he gene a ion o ha d ea u e1524
models is a p omising esea ch opic ha equi es u -1525
he s udy.1526
Gene icAlgo i hms(GAs)[1]a ea subclass o e olu-1527
iona y algo i hms in which solu ions a e encoded using1528
bi s ings. Howe e , i is di icul o encode he hie a -1529
chical s uc u e o ea u e models using his app oach1530
and he e o e we disca ded hei use. Gene ic P og am-1531
ming (GP) is ano he a ian o e olu iona y algo i hms1532
in which solu ions a e encoded as ees [54]. This en-1533
coding is commonly used o ep esen p og ams whose1534
abs ac syn ax can be na u ally ep esen ed hie a chi-1535
cally. C osso e in GP is applied on an indi idual by1536
swi ching one o i s b anches wi h ano he b anch om1537
ano he indi idual in he popula ion, i.e. indi iduals can1538
ha e di e en sizes. We iden i ied se e al ac o s ha 1539
make GPs unsui able o ou p oblem. Fi s , he classic1540
ee encoding does no conside c oss- ee cons ain s as1541
in ea u e models. As a esul , c osso e would p oba-1542
bly gene a e many dangling edges which may equi e1543
cos ly epai ing heu is ics. Second, and mo e impo -1544
an ly, c osso e in GP does no gua an ee a ixed size1545
o he solu ion which was a key cons ain in ou wo k.1546
These easons led us o design a cus om e olu iona y al-1547
go i hm, ETHOM, suppo ing he ep esen a ion o ea-1548
u e ees o ixed size wi h c oss- ee cons ain s.1549
7.3. Pe o mance e alua ion o CSP and SAT sol e s1550
CSP and SAT sol e s (he eina e , CP sol e s) use1551
algo i hms and echniques o Cons ain P og amming1552
(CP) o sol e complex p oblems om domains such as1553
compu e science, a i icial in elligence o ha dwa e de-1554
sign6. Theunde lying p oblems o CSP and SAT sol e s1555
a e NP-comple e and so CSP and SAT sol e s ha e an1556
exponen ial wo s case un ime. This makes e iciency1557
a c ucial ma e o hese ypes o ools. Hence, he e1558
exis a numbe o a ailable benchma ks o e alua e and1559
compa e he pe o mance o CP sol e s [27]. Also, se -1560
e al compe i ions a e held e e y yea o ank he pe -1561
o mance o he pa icipan s’ ools. As an example, 931562
sol e s ook pa in he SAT compe i ion7in 2013.1563
CP sol e s use h ee main ypes o p oblems o pe -1564
o mance e alua ion: p oblems om ealis ic domains1565
(e.g. ha dwa e design), andomly gene a ed p oblems1566
and ha d p oblems. Bo h andomly gene a ed and ha d1567
6A SAT p oblem can be ega ded a subclass o CSP wi h only
boolean a iables.
7h p://www.sa compe i ion.o g
p oblems a e au oma ically gene a ed and a e o en1568
o ced o ha e a leas one solu ion (i.e. be sa is iable).1569
The CP esea ch communi y ealised long ago ha he e1570
a e bene i s in using ha d p oblems o es he pe o -1571
mance o hei ools. In 1997, Cook and Mi chell [17]1572
p esen ed a su ey on he s a egies o ind ha d SAT1573
ins ances p oposed so a . In hei wo k, he au ho s1574
wa ned abou he impo ance o gene a ing ha d p ob-1575
lems o unde s anding hei complexi y and o p o id-1576
ing challenging benchma ks. Since hen, many o he 1577
con ibu ions ha e explo ed he gene a ion o ha d SAT1578
and CSP p oblems [5, 77].1579
A common s a egy o gene a e ha d CSP and SAT1580
p oblems is by exploi ing wha is known as he phase1581
ansi ion phenomenon [77]. This phenomenon es ab-1582
lishes ha o many NP-comple e p oblems he ha des 1583
ins ances occu be ween he egion in which mos p ob-1584
lems a e sa is iable and he egion in which mos p ob-1585
lems a e unsa is iable. This happens because o hese1586
p oblems he sol e has o explo e he sea ch space in1587
dep h be o e inding ou whe he he p oblem is sa is i-1588
able o no . CSP and SAT sol e s can be pa ame ically1589
guided o sea ch in he phase ansi ion egion enabling1590
he sys ema ic gene a ion o ha d p oblems. We a e no 1591
awa e o any wo k using e olu iona y algo i hms o he1592
gene a ion o ha d CP p oblems.1593
When compa ed o CP p oblems, he analysis o ea-1594
u e models di e s in se e al ways. Fi s , CSP and SAT1595
a e ela ed p oblems wi hin he cons ain p og amming1596
pa adigm. The analysis o ea u e models, howe e , is a1597
high-le el p oblem usually sol ed using qui e he e oge-1598
neous app oaches such as cons ain p og amming, de-1599
sc ip ion logic, seman ic web echnologies o ad-hoc al-1600
go i hms [10]. Also, CP sol e s ocus on a single anal-1601
ysis ope a ion (i.e. sa is iabili y) o which he e exis 1602
a numbe o well known algo i hms. In he analysis o 1603
ea u e models, howe e , mo e han 30 analysis ope a-1604
ions ha e been epo ed. In his scena io, we belie e1605
ha ou app oach may help he communi y o gene a e1606
ha d p oblems and s udy hei complexi y, leading o a1607
be e unde s anding o he analysis ope a ions and he1608
pe o mance o analysis ools.1609
We iden i ied wo main ad an ages in ou wo k when1610
compa ed o he sys ema ic gene a ion o ha d CP p ob-1611
lems. Fi s , ou app oach is gene ic and can be applied1612
o any ool, algo i hm o analysis ope a ion o he au-1613
oma ed ea men o ea u e models. Second, ou algo-1614
i hm is ee o explo e he whole sea ch space looking1615
o inpu models ha e eal pe o mance ulne abili ies.1616
In con as , CP ela ed wo k ocuses he sea ch o in-1617
pu s p oblem in a speci ic a ea ( he ansi ion phase e-1618
gion).1619
20
O e all, we conclude ha ela ed wo k in CP suppo 1620
ou app oach o he gene a ion o ha d ea u e mod-1621
els as a way o e alua e he pe o mance s eng hs and1622
weakness o ea u e model analysis ools.1623
8. Conclusions and u u e wo k1624
In his pape , we p esen ed ETHOM, a no el e o-1625
lu iona y algo i hm o sol e op imisa ion p oblems on1626
ea u e models and showed how i can be used o 1627
he au oma ed gene a ion o compu a ionally ha d ea-1628
u e models. Expe imen s using ou e olu iona y ap-1629
p oach on di e en analysis ope a ions and indepen-1630
den ools success ully iden i ied inpu models p oduc-1631
ing much longe execu ions imes and highe memo y1632
consump ion han andomly gene a ed models o iden-1633
ical o e en la ge size. In o al, mo e han 50 mil-1634
lion execu ions o analysis ope a ions we e pe o med1635
o con igu e and e alua e ou app oach. This is he1636
i s me aheu is ic-based s a egy o guide he sea ch o 1637
compu a ionally ha d ea u e models a he han sim-1638
ply using andomly gene a ed models. This app oach1639
will allow de elope s o ocus on he sea ch o ough1640
models o ealis ic size ha could e eal de iciencies in1641
hei ools a he han using huge andomly gene a ed1642
ea u e models ou o he scope o hei ools. Simi-1643
la ly, use s a e p o ided wi h mo e in o ma ion abou 1644
he expec ed beha iou o he ools in pessimis ic cases,1645
helping hem o choose he ool o echnique ha be e 1646
mee s hei needs. Con a y o gene al belie , we ound1647
ha model size has an impo an , bu no decisi e, e ec 1648
on pe o mance. Also, we ound ha he ha d ea u e1649
models gene a ed by ETHOM had simila p ope ies o1650
ealis ic models ound in he li e a u e. This means ha 1651
he long execu ion imes and high memo y consump ion1652
ound by ou algo i hm migh be ep oduced in eal sce-1653
na ios wi h he consequen nega i e e ec on he use .1654
In iew o he posi i e esul s ob ained, we expec his1655
wo k o be he seed o many o he esea ch con ibu-1656
ions exploi ing he bene i s o ETHOM in pa icula ,1657
and e olu iona y compu a ion in gene al, on he anal-1658
ysis o ea u e models. In pa icula , we en ision wo1659
main esea ch di ec ions o be explo ed by he commu-1660
ni y in he u u e, namely:1661
•Algo i hms de elopmen . The combina ion1662
o di e en encodings, selec ion echniques,1663
c osso e s a egies, mu a ion ope a o s and o he 1664
pa ame e s may lead o a whole new a ie y o e o-1665
lu iona y algo i hms o ea u e models o be ex-1666
plo ed. Also, he use o o he me aheu is ic ech-1667
niques (e.g. an colony op imisa ion) is a p omis-1668
ing opic ha need u he s udy. The de elop-1669
men o mo e lexible algo i hms would be desi -1670
able in o de o deal wi h o he ea u e modelling1671
languages (e.g. ca dinali y-based ea u e models)1672
o s ic e s uc u al cons ain s, e.g. enabling he1673
gene a ion o ha d models wi h a gi en pe cen -1674
age o manda o y ea u es. Also, he gene a ion o 1675
ea u e models wi h complex c oss- ee cons ain s1676
( hose in ol ing mo e han wo ea u es) emains1677
an open challenge ha we in end o add ess in ou 1678
u u e wo k.1679
•Applica ions. Fu he applica ions o ou algo-1680
i hm a e s ill o be explo ed. Some p omising ap-1681
plica ions a e hose dealing wi h he op imisa ion1682
o non– unc ional p ope ies in o he analysis ope -1683
a ions o e en di e en au oma ed ea men s, e.g.1684
e ac o ing ea u e models. The applica ion o ou 1685
algo i hm o minimisa ion p oblems is also an open1686
issue in which we ha e s a ed o ob ain p omising1687
esul s. Addi ionally, i would be nice o apply ou 1688
app oach o e i y he ime cons ain s o eal ime1689
sys ems dealing wi h a iabili y like hose o mo-1690
bile phones o con ex –awa e pe asi e sys ems.1691
Las , bu no leas , we plan o s udy he ha d ea-1692
u e models gene a ed and y o unde s and wha 1693
makes hem ha d o analyse. F om he in o ma ion1694
ob ained, mo e e ined applica ions and heu is ics1695
could be de eloped leading o mo e e icien ool1696
suppo o he analysis o ea u e models.1697
A Ja a implemen a ion o ETHOM is eady- o-use1698
and publicly a ailable as a pa o he open-sou ce1699
BeTTy F amewo k [14, 58].1700
Ma e ial1701
The p o o ype implemen a ion o ETHOM, ha d ea-1702
u e models gene a ed (in XML o ma ), s a is ical1703
esul s (in SPSS o ma ) and aw expe imen da a1704
a e a ailable a h p://www.lsi.us.es/~segu a/1705
iles/ma e ial/ESWA13/.1706
Acknowledgmen s1707
We would like o hank D . Don Ba o y, D . Ja ie 1708
Dolado, D . A naud Go lieb, D . And eas Me zge , D .1709
Jose C. Riquelme, D . Da id Ruiz and D . Ja ie Tuya1710
whose commen s and sugges ions helped us o imp o e1711
he a icle subs an ially. We would also like o hank1712
Jos´
e A. Galindo o his wo k in eg a ing ETHOM in o1713
he amewo k BeTTy.1714
21

This wo k has been pa ially suppo ed by he Eu o-1715
pean Commission (FEDER) and Spanish Go e nmen 1716
unde CICYT p ojec s SETI (TIN2009-07366) and1717
TAPAS (TIN2012-32273) and he Andalusian Go e n-1718
men p ojec s THEOS (TIC-5906) and COPAS (P12-1719
TIC-1867).1720
Appendix A. S a is ical analysis esul s1721
#Fea u es CTC (%)
10 20 30 40
200 0.53 0 0 0
400 0.28 0 0 0
600 0.36 0 0 0
800 0 0 0 0
1000 0.12 0 0 0
Table A.6: p- alues ob ained in Expe imen #1 using he Mann-
Whi ney-Wilcoxon es
1722 #Fea u es CTC (%)
10 20 30
50 0 0 0
100 0 0 0
150 0 0 0
200 0 0 0
250 0 0 0.85
Table A.7: p- alues ob ained in Expe imen #2 using he Mann-
Whi ney-Wilcoxon es
1723
Re e ences1724
[1] M. A enzelle , S. Wagne , S. Winkle , and A. Beham. Gene ic1725 Algo i hms and Gene ic P og amming: Mode n Concep s and1726 P ac ical Applica ions. Nume ical Insigh s. Taylo & F ancis,1727 2009.1728 [2] W. A zal, R. To ka , and R. Feld . A sys ema ic e iew o 1729 sea ch-based es ing o non- unc ional sys em p ope ies. In-1730 o ma ion and So wa e Technology, 51(6):957–976, 2009.1731 [3] AHEAD Tool Sui e. h p://www.cs.u exas.edu/use s/1732
schwa z/ATS.h ml, accessed July 2013.1733 [4] N. Ande sen, K. Cza necki, S. She, and A. Wasowski. E i-1734 cien syn hesis o ea u e models. In 16 h In e na ional So wa e1735 P oduc Line Con e ence, pages 106–115, 2012.1736 [5] C. Anso egui, R. Beja , C. Fe nandez, and C. Ma eu. Edge1737 ma ching puzzles as ha d SAT/CSP benchma ks. In P. S uckey,1738 edi o , P inciples and P ac ice o Cons ain P og amming, ol-1739 ume 5202 o Lec u e No es in Compu e Science, pages 560–1740 565. Sp inge Be lin /Heidelbe g, 2008.1741 [6] D. Ba o y. Fea u e models, g amma s, and p oposi ional o mu-1742 las. In So wa e P oduc Lines Con e ence (SPLC), olume 37141743 o Lec u e No es in Compu e Sciences, pages 7–20. Sp inge –1744 Ve lag, 2005.1745 [7] D. Ba o y, D. Bena ides, and A. Ruiz-Co ´
es. Au oma ed anal-1746 ysis o ea u e models: Challenges ahead. Communica ions o 1747 he ACM, Decembe :45–47, 2006.1748
[8] D. Bena ides, A. Ruiz-Co ´
es, and P. T inidad. Coping wi h au-1749 oma ic easoning on so wa e p oduc lines. In P oceedings o 1750 he 2nd G oningen Wo kshop on So wa e Va iabili y Manage-1751 men , No embe 2004.1752 [9] D. Bena ides, A. Ruiz-Co ´
es, and P. T inidad. Au oma ed ea-1753 soning on ea u e models. In 17 h In e na ional Con e ence on1754 Ad anced In o ma ion Sys ems Enginee ing (CAiSE), olume1755 3520 o Lec u e No es in Compu e Sciences, pages 491–503.1756 Sp inge –Ve lag, 2005.1757 [10] D. Bena ides, S. Segu a, and A. Ruiz-Co ´
es. Au oma ed anal-1758 ysis o ea u e models 20 yea s la e : A li e a u e e iew. In o -1759 ma ion Sys ems, 35(6):615 – 636, 2010.1760 [11] D. Bena ides, S. Segu a, P. T inidad, and A. Ruiz-Co ´
es. A i s 1761 s ep owa ds a amewo k o he au oma ed analysis o ea u e1762 models. In Managing Va iabili y o So wa e P oduc Lines:1763 Wo king Wi h Va iabili y Mechanisms, 2006.1764 [12] D. Bena ides, S. Segu a, P. T inidad, and A. Ruiz-Co ´
es. Using1765 Ja a CSP sol e s in he au oma ed analyses o ea u e models.1766 LNCS, 4143:389–398, 2006.1767 [13] T. Be ge , S. She, R. Lo u o, A. Wasowski, and K. Cza necki.1768 Va iabili y modeling in he eal: a pe spec i e om he ope a -1769 ing sys ems domain. In P oceedings o he IEEE/ACM In e na-1770 ional Con e ence on Au oma ed So wa e Enginee ing, pages1771 73–82. ACM, 2010.1772 [14] BeTTy F amewo k. h p://www.isa.us.es/be y, ac-1773 cessed July 2013.1774 [15] BigLe e . Bigle e so wa e gea s. h p://www.bigle e .1775
com/, accessed July 2013.1776 [16] P. Clemen s and L. No h op. So wa e P oduc Lines: P ac ices1777 and Pa e ns. SEI Se ies in So wa e Enginee ing. Addison–1778 Wesley, Augus 2001.1779 [17] S.A. Cook and D.G. Mi chell. Finding ha d ins ances o he1780 sa is iabili y p oblem: A su ey. In Sa is iabili y P oblem: The-1781 o y and Applica ions, olume 35 o Dimacs Se ies in Disc e e1782 Ma hema ics and Theo e ical Compu e Science, pages 1–17.1783 Ame ican Ma hema ical Socie y, 1997.1784 [18] A.E. Eiben and S.K. Smi . Pa ame e uning o con igu ing1785 and analyzing e olu iona y algo i hms. Swa m and E olu ion-1786 a y Compu a ion, 1(1):19 – 31, 2011.1787 [19] FaMa Tool Sui e. h p://www.isa.us.es/ ama/, accessed1788 July 2013.1789 [20] Fea u e Modeling Plug-in. h p://gp.uwa e loo.ca/ mp/,1790 accessed July 2013.1791 [21] J.A. Galindo, D. Bena ides, and S. Segu a. Debian packages1792 eposi o ies as so wa e p oduc line models. Towa ds au o-1793 ma ed analysis. In P oceedings o he 1s In e na ional Wo k-1794 shop on Au oma ed Con igu a ion and Tailo ing o Applica ions1795 (ACoTA), An we p, Belgium, 2010.1796 [22] R. Gheyi, T. Massoni, and P. Bo ba. A heo y o ea u e mod-1797 els in Alloy. In P oceedings o he ACM SIGSOFY Fi s Alloy1798 Wo kshop, pages 71–80, Po land, Uni ed S a es, no 2006.1799 [23] J. Guo, J. Whi e, G. Wang, J. Li, and Y. Wang. A gene ic algo-1800 i hm o op imized ea u e selec ion wi h esou ce cons ain s1801 in so wa e p oduc lines. Jou nal o Sys ems and So wa e,1802 84:2208–2221, Decembe 2011.1803 [24] A. Hemakuma . Finding con adic ions in ea u e models. In1804 Fi s In e na ional Wo kshop on Analyses o So wa e P oduc 1805 Lines (ASPL), pages 183–190, 2008.1806 [25] C. Hena d, M. Papadakis, G. Pe ouin, J. Klein, and Y.L. T aon.1807 Mul i-objec i e es gene a ion o so wa e p oduc lines. In1808 P oceedings o he 17 h In e na ional So wa e P oduc Line1809 Con e ence, SPLC ’13, pages 62–71, New Yo k, NY, USA,1810 2013. ACM.1811 [26] R. He adio-Gil, D. Fe nandez-Amo os, J.A. Ce ada, and1812 C. Ce ada. Suppo ing commonali y-based analysis o so wa e1813
22
p oduc lines. So wa e, IET, 5(6):496 –509, dec. 2011.1814 [27] H. Hoos and T. S u zle. SATLIB: An online esou ce o e-1815 sea ch on SAT. In I.P. an Maa en, H. Gen , and T. Walsh, ed-1816 i o s, Sa 2000: Highligh s o Sa is iabili y Resea ch in he Yea 1817 2000, pages 283–292. IOS P ess, 2000.1818 [28] IBM. SPSS 17 S a is ical Package. h p://www.spss.com/,1819 accessed No embe 2010.1820 [29] JaCoP. h p://jacop.osolp o.com/, accessed July 2013.1821 [30] Ja aBDD. h p://ja abdd.sou ce o ge.ne /, accessed1822 July 2013.1823 [31] M.F. Johansen, Ø. Haugen, and F. Fleu ey. An algo i hm o 1824 gene a ing -wise co e ing a ays om la ge ea u e models.1825 In 16 h In e na ional So wa e P oduc Line Con e ence, pages1826 46–55, 2012.1827 [32] K. Kang, S. Cohen, J. Hess, W. No ak, and S. Pe e son.1828 Fea u e–O ien ed Domain Analysis (FODA) Feasibili y S udy.1829 Technical Repo CMU/SEI-90-TR-21, SEI, 1990.1830 [33] A. Ka a as, H. Oguz ¨
uz¨
un, and A. Dog u. Global cons ain s1831 on ea u e models. In D. Cohen, edi o , P inciples and P ac ice1832 o Cons ain P og amming, olume 6308 o Lec u e No es in1833 Compu e Science, pages 537–551, 2010.1834 [34] C. Kas ne , S. Apel, and D. Ba o y. A case s udy implemen ing1835 ea u es using Aspec J. In SPLC ’07: P oceedings o he 11 h1836 In e na ional So wa e P oduc Line Con e ence, pages 223–1837 232, Washing on, DC, USA, 2007. IEEE Compu e Socie y.1838 [35] A. Kolmogo o . Sulla de e minazione empi ica di una legge di1839 dis ibuzione. G. Ins . I al. A ua i, 4:83, 1933.1840 [36] S.Q. Lau. Domain analysis o e-comme ce sys ems using1841 ea u e–based model empla es. mas e ’s hesis. Dep . o ECE,1842 Uni e si y o Wa e loo, Canada, 2006.1843 [37] F. Loesch and E. Ploede ede . Op imiza ion o a iabili y in1844 so wa e p oduc lines. In P oceedings o he 11 h In e na-1845 ional So wa e P oduc Line Con e ence (SPLC), pages 151–1846 162, Washing on, DC, USA, 2007. IEEE Compu e Socie y.1847 [38] R.E Lopez-He ejon and D. Ba o y. A s anda d p oblem o 1848 e alua ing p oduc -line me hodologies. In GCSE ’01: P oceed-1849 ings o he Thi d In e na ional Con e ence on Gene a i e and1850 Componen -Based So wa e Enginee ing, pages 10–24, Lon-1851 don, UK, 2001. Sp inge -Ve lag.1852 [39] R.E. Lopez-He ejon, F. Chicano, J. Fe e , A. Egyed, and1853 E. Alba. Mul i-objec i e op imal es sui e compu a ion o so -1854 wa e p oduc line pai wise es ing. In P oceedings o he 29 h1855 IEEE In e na ional Con e ence on So wa e Main enance, 2013.1856 [40] R.E. Lopez-He ejon, J.A. Galindo, D. Bena ides, S. Segu a,1857 and A. Egyed. Re e se enginee ing ea u e models wi h e olu-1858 iona y algo i hms: An explo a o y s udy. In Sea ch Based So -1859 wa e Enginee ing, olume 7515 o Lec u e No es in Compu e 1860 Science, pages 168–182. Sp inge Be lin Heidelbe g, 2012.1861 [41] H.B. Mann and D.R. Whi ney. On a es o whe he one o wo1862 andom a iables is s ochas ically la ge han he o he . Ann.1863 Ma h. S a ., 18:50–60, 1947.1864 [42] P. McMinn. Sea ch-based so wa e es da a gene a ion: a su -1865 ey. So wa e Tes ing Ve i ica ion and Reliabili y., 14(2):105–1866 156, 2004.1867 [43] M. Mendonca, M. B anco, and D. Cowan. S.P.L.O.T.: So wa e1868 P oduc Lines Online Tools. In Companion o he 24 h ACM1869 SIGPLAN In e na ional Con e ence on Objec -O ien ed P o-1870 g amming, Sys ems, Languages, and Applica ions (OOPSLA),1871 pages 761–762, O lando, Flo ida, USA, Oc obe 2009. ACM.1872 [44] M. Mendonca, D.D. Cowan, W. Malyk, and T. Oli ei a. Collab-1873 o a i e p oduc con igu a ion: Fo maliza ion and e icien algo-1874 i hms o dependency analysis. Jou nal o So wa e, 3(2):69–1875 82, 2008.1876 [45] M. Mendonca, A. Wasowski, and K. Cza necki. SAT–based1877 analysis o ea u e models is easy. In P oceedings o he In e -1878
na ional So wa e P oduc Line Con e ence (SPLC), 2009.1879 [46] M. Mendonca, A. Wasowski, K. Cza necki, and D.D. Cowan.1880 E icien compila ion echniques o la ge scale ea u e models.1881 In 7 h In e na ional Con e ence on Gene a i e P og amming1882 and Componen Enginee ing (GPCE), pages 13–22, 2008.1883 [47] A. Osman, S. Phon-Amnuaisuk, and C.K. Ho. Using i s o -1884 de logic o alida e ea u e model. In Thi d In e na ional1885 Wo kshop on Va iabili y Modelling in So wa e-in ensi e Sys-1886 ems (VaMoS), pages 169–172, 2009.1887 [48] J.A.. Pa ejo, A. Ruiz-Co ´
es, S. Lozano, and P. Fe nandez.1888 Me aheu is ic op imiza ion amewo ks: a su ey and bench-1889 ma king. So Compu ing - A Fusion o Founda ions, Me hod-1890 ologies and Applica ions, 16:527–561, 2012.1891 [49] L. Passos, M.No ako ic, Y. Xiong, T. Be ge , K. Cza necki, and1892 A. Wasowski. A s udy o non-boolean cons ain s in a iabili y1893 models o anembedded ope a ing sys em. InThi d In e na ional1894 Wo kshop on Fea u e-O ien ed So wa e De elopmen (FOSD),1895 SPLC ’11, pages 2:1–2:8. ACM, 2011.1896 [50] G. Pe ouin, S. Os e , S. Sen, J. Klein, B. Baud y, and Y. T aon.1897 Pai wise es ing o so wa e p oduc lines: compa ison o wo1898 app oaches. So wa e Quali y Jou nal, 20:605–643, 2012.1899 [51] R. Pohl, K. Lauen o h, and K. Pohl. A pe o mance compa ison1900 o con empo a y algo i hmic app oaches o au oma ed analy-1901 sis ope a ions on ea u e models. In 26 h In e na ional Con-1902 e ence on Au oma ed So wa e Enginee ing, pages 313–322.1903 IEEE, 2011.1904 [52] R. Pohl, V. S icke , and K. Pohl. Measu ing he s uc u al com-1905 plexi y o ea u e models. In 28 h In e na ional Con e ence on1906 Au oma ed So wa e Enginee ing, pages 454–464. IEEE, 2013.1907 [53] pu e:: a ian s. h p://www.pu e-sys ems.com/, accessed1908 July 2013.1909 [54] F. Ro hlau . Rep esen a ions o Gene ic and E olu iona y Al-1910 go i hms. Sp inge , 2nd edi ion, 2012.1911 [55] A.S. Sayyad, T. Menzies, and H. Amma . On he alue o use 1912 p e e ences in sea ch-based so wa e enginee ing: A case s udy1913 in so wa e p oduc lines. In P oceedings o he 2013 In e na-1914 ional Con e ence on So wa e Enginee ing, ICSE ’13, pages1915 492–501, Pisca away, NJ, USA, 2013. IEEE P ess.1916 [56] P. Schobbens, P. Heymans, J. T igaux, and Y. Bon emps. Fea-1917 u e Diag ams: A Su ey and A Fo mal Seman ics. In P oceed-1918 ings o he 14 h IEEE In e na ional Requi emen s Enginee ing1919 Con e ence (RE’06), Minneapolis, Minneso a, USA, Sep embe 1920 2006.1921 [57] S. Segu a. Au oma ed analysis o ea u e models using a omic1922 se s. In Fi s Wo kshop on Analyses o So wa e P oduc Lines1923 (ASPL), pages 201–207, Lime ick, I eland, Sep embe 2008.1924 [58] S. Segu a, J.A. Galindo, D. Bena ides, J.A. Pa ejo, and A. Ruiz-1925 Co ´
es. BeTTy: Benchma king and Tes ing on he Au oma ed1926 Analysis o Fea u e Models. In U.W. Eisenecke , S. Apel, and1927 S. Gnesi, edi o s, Six h In e na ional Wo kshop on Va iabil-1928 i y Modelling o So wa e-in ensi e Sys ems (VaMoS’12), pages1929 63–71, Leipzig, Ge many, 2012. ACM.1930 [59] S. Segu a, J.A. Pa ejo, R.M. Hie ons, D. Bena ides, and1931 A. Ruiz-Co ´
es. ETHOM: An e olu iona y algo i hm o 1932 op imized ea u e models gene a ion ( 1.3). Technical Re-1933 po ISA-2013-TR-01, Applied So wa e Enginee ing Resea ch1934 G oup, Se ille, Spain, 2013. h p://www.isa.us.es/1935
si es/de aul / iles/Ha dFMUsingEA_1.pd .1936 [60] S. S. Shapi o and M. B. Wilk. An analysis o a iance es 1937 o no mali y (comple e samples). Biome ika, 52(3/4):pp. 591–1938 611, 1965.1939 [61] S. She, R. Lo u o, T. Be ge , A. Wasowski, and K. Cza necki.1940 The a iabili y model o he linux ke nel. In Fou h In e na-1941 ional Wo kshop on Va iabili y Modelling o So wa e-in ensi e1942 Sys ems (VaMoS), Linz, Aus ia, Janua y 2010.1943
23
[62] S. She, R. Lo u o, T. Be ge , A. Wasowski, and K. Cza necki.1944 Re e se enginee ing ea u e models. In P oceeding o he 33 d1945 In e na ional Con e ence on So wa e Enginee ing, pages 461–1946 470. ACM, 2011.1947 [63] N. V. Smi no . Tables o es ima ing he goodness o i o em-1948 pi ical dis ibu ions. Annals o Ma hema ical S a is ic, 19:279,1949 1948.1950 [64] S. Sol ani, M. Asadi, D. Gase ic, M. Ha ala, and E. Baghe i.1951 Au oma ed planning o ea u e model con igu a ion based on1952 unc ional and non- unc ional equi emen s. In 16 h In e na-1953 ional So wa e P oduc Line Con e ence, pages 56–65, 2012.1954 [65] S.P.L.O.T.: So wa e P oduc Lines Online Tools. h p://1955
www.splo - esea ch.o g/, accessed July 2013.1956 [66] M. S ege , C. Tische , B. Boss, A. M¨
ulle , O. Pe le , W. S olz,1957 and S. Fe be . In oducing PLA a Bosch gasoline sys ems: Ex-1958 pe iences and p ac ices. In In e na ional So wa e P oduc Line1959 Con e ence (SPLC), pages 34–50, 2004.1960 [67] T. Th¨
um, D. Ba o y, and C. K¨
as ne . Reasoning abou edi s o1961 ea u e models. In In e na ional Con e ence on So wa e Engi-1962 nee ing, pages 254–264, 2009.1963 [68] Edwa d Tsang. Founda ions o Cons ain Sa is ac ion. Aca-1964 demic P ess, 1995.1965 [69] S. Voß. Me a-heu is ics: The s a e o he a . In ECAI ’00:1966 P oceedings o he Wo kshop on Local Sea ch o Planning and1967 Scheduling-Re ised Pape s, pages 1–23. Sp inge -Ve lag, Lon-1968 don, UK, 2001.1969 [70] H.H. Wang, Y.F. Li, J. Sun, H. Zhang, and J. Pan. Ve i ying1970 ea u e models using OWL. Jou nal o Web Seman ics, 5:117–1971 129, June 2007.1972 [71] S. Wang, S. Ali, and A. Go lieb. Minimizing es sui es in1973 so wa e p oduc lines using weigh -based gene ic algo i hms.1974 In P oceeding o he Fi een h Annual Con e ence on Gene ic1975 and E olu iona y Compu a ion Con e ence, GECCO ’13, pages1976 1493–1500, New Yo k, NY, USA, 2013. ACM.1977 [72] J. Wegene , K. G imm, M. G och mann, and H. S hame . Sys-1978 ema ic es ing o eal- ime sys ems. In P oceedings o he1979 Fou h In e na ional Con e ence on So wa e Tes ing and Re-1980 iew (Eu oSTAR), 1996.1981 [73] J. Wegene , H. S hame , B.F. Jones, and D.E. Ey es. Tes ing1982 eal- ime sys ems using gene ic algo i hms. So wa e Quali y1983 Con ol, 6(2):127–135, 1997.1984 [74] J. Whi e, B. Dough e y, and D. Schmid . Selec ing highly op-1985 imal a chi ec u al ea u e se s wi h il e ed ca esian la ening.1986 Jou nal o Sys ems and So wa e, 82(8):1268–1284, 2009.1987 [75] J. Whi e, B. Dough e y, D. Schmid , and D. Bena ides. Au-1988 oma ed easoning o mul i-s ep so wa e p oduc -line con ig-1989 u a ion p oblems. In P oceedings o he So wa e P oduc Line1990 Con e ence, pages 11–20, 2009.1991 [76] J. Whi e, D. Schmid , D. Bena ides P. T inidad, and Ruiz-1992 Co ´
es. Au oma ed diagnosis o p oduc -line con igu a ion e -1993 o s in ea u e models. In P oceedings o he 12 h So wa e1994 P oduc Line Con e ence (SPLC), Lime ick, I eland, Sep embe 1995 2008.1996 [77] K. Xu, F. Boussema , F. Heme y, and C. Lecou e. Random1997 cons ain sa is ac ion: Easy gene a ion o ha d (sa is iable) in-1998 s ances. A i icial In elligence, 171(8-9):514–534, 2007.1999 [78] H. Yan, W. Zhang, H. Zhao, and H. Mei. An op imiza ion s a -2000 egy o ea u e models’ e i ica ion by elimina ing e i ica ion-2001 i ele an ea u es and cons ain s. In ICSR, pages 65–75, 2009.2002 [79] W. Zhang, H. Yan, H. Zhao, and Z. Jin. A BDD–based app oach2003 o e i ying clone-enabled ea u e models’ cons ain s and cus-2004 omiza ion. In 10 h In e na ional Con e ence on So wa e Reuse2005 (ICSR), LNCS, pages 186–199. Sp inge , 2008.2006
24