The D upal F amewo k:
A Case S udy o E alua e Va iabili y Tes ing Techniques
Ana B. Sánchez, Se gio Segu a, and An onio Ruiz-Co és
Depa men o Compu e Languages and Sys ems
Uni e si y o Se ille, Spain
{anabsanchez,se giosegu a,a uiz}@us.es
ABSTRACT
Va iabili y es ing echniques sea ch o e ec i e bu man-
ageable es sui es ha lead o he apid de ec ion o aul s
in sys ems wi h high a iabili y. E alua ing he e ec i eness
o hese echniques in eal se ings is a mus bu challenging
due o he lack o a iabili y-in ensi e sys ems wi h a ail-
able code, au oma ed es s and aul epo s. In his pa-
pe , we p opose using he D upal amewo k as a case s udy
o e alua e a iabili y es ing echniques. Fi s , we ep e-
sen he amewo k a iabili y as a ea u e model. Then,
we epo on ex ensi e da a ex ac ed om he D upal gi
eposi o y and he D upal issue acking sys em. Among
o he esul s, we iden i ied 378 aul s in single ea u es and
11 aul s igge ed by he in e ac ion be ween wo o he
ea u es o D upal 7.23, epo ed du ing a one-yea pe iod.
These da a may gi e a new insigh in o he dis ibu ion o
aul s in a iabili y-in ensi e sys ems and he aul p open-
si y o ea u es. To show he easibili y o ou wo k, we
used he case s udy o e alua e he e ec i eness o a his o y-
based es case p io i iza ion c i e ion. Resul s sugges ha
his echnique could con ibu e o accele a e he de ec ion
o aul s o es sui es based on combina o ial es ing.
Keywo ds
Va iabili y, es ing, ea u e model, au oma ed es ing, es
case selec ion, es case p io i iza ion.
1. INTRODUCTION
Va iabili y de e mines he abili y o so wa e applica ions
o be con igu ed and cus omized. One o he mos p ominen
examples o a iabili y-in ensi e sys ems a e So wa e P od-
uc Lines (SPLs). An SPL is a amily o ela ed so wa e
p oduc s. Each p oduc ep esen s a speci ic combina ion
o ea u es o he SPL. Ins i u ions such as Gene al Mo o s,
NASA and Boeing a e using SPL echnology o dec ease
ime o ma ke and imp o e so wa e quali y [1]. Typically,
SPLs a e modelled by using a iabili y models such as ea-
u e models [3]. A ea u e model ep esen s he se o p od-
Pe mission o make digi al o ha d copies o all o pa o his wo k o
pe sonal o class oom use is g an ed wi hou ee p o ided ha copies a e no
made o dis ibu ed o p o i o comme cial ad an age and ha copies bea
his no ice and he ull ci a ion on he i s page. Copy igh s o componen s
o his wo k owned by o he s han ACM mus be hono ed. Abs ac ing wi h
c edi is pe mi ed. To copy o he wise, o epublish, o pos on se e s o o
edis ibu e o lis s, equi es p io speci ic pe mission and/o a ee.
VaMoS ’14 Nice, F ance.
Copy igh 2014 ACM 978-1-4503-2556-1/14/01 ...$15.00.
h p://dx.doi.o g/10.1145/2556624.2556638.
uc s o an SPL in e ms o ea u es and ela ionships among
hem (see Fig. 3).
The numbe o con igu a ions and dependencies in a i-
abili y models is po en ially huge. Fo ins ance, he Billing
ea u e model a ailable in he SPLOT eposi o y has 88 ea-
u es and 66% o hem a e connec ed by cons ain s, ep e-
sen ing mo e han 1 billion o po en ial p oduc s [15]. This
explosion o combina ions makes es ing a iabili y-in ensi e
sys ems a majo challenge. To add ess his p oblem, e-
sea che s ha e p oposed a ious echniques o educe he
cos o es ing in he p esence o a iabili y, including es
case selec ion and es case p io i iza ion echniques. Tes
case selec ion app oaches [8, 17, 24] selec an app op ia e
subse o he exis ing es sui e acco ding o some co e age
c i e ia. Tes case p io i iza ion app oaches [9, 19, 22, 24]
schedule es cases o execu ion in an o de ha a emp s
o inc ease hei e ec i eness a mee ing some pe o mance
goal, e.g. accele a e he de ec ion o aul s [19].
The numbe o wo ks on a iabili y es ing is g owing
apidly and hus he numbe o expe imen al e alua ions [6,
9, 16]. Howe e , i is ha d o ind eal a iabili y-in ensi e
sys ems wi h a ailable code, es cases, de ailed aul e-
po s and good documen a ion ha enable ep oducible ex-
pe imen s [20, 21]. As a esul , au ho s o en e alua e hei
es ing app oaches using a i icial a iabili y models [8, 9]
and simula ed aul s [6] which in oduce h ea s o alidi y
and weaken hei conclusions. A ela ed p oblem is he lack
o in o ma ion abou he dis ibu ion o aul s in a iabili y-
in ensi e sys ems, e.g. numbe and ypes o aul s, aul
se e i y, e c. This may be an obs acle o he design o new
es ing echniques since esea che s a e no ully awa e o
he ype o aul s hey a e looking o .
In he sea ch o eal a iabili y sys ems some au ho s
ha e explo ed he domain o open sou ce ope a ing sys ems
[7, 13, 21]. Howe e , hese wo ks mainly ocus on he a i-
abili y modelling pe spec i e and hus igno e ele an da a
o es e s such as he numbe o es cases o he dis ibu-
ion o aul s, i.e. how aul s a e dis ibu ed o e he di e -
en ea u es. Also, SPL2GO1is a ca alog o SPLs o which
sou ce code and a iabili y model (e.g. ea u e model) a e
publicly a ailable. Howe e , hese also lack o de ail da a
abou he aul s ound in he p og ams and i is up o he
use o inspec he code sea ching o es cases.
In o de o look o a eal a iabili y-in ensi e sys em wi h
a ailable code, we ollowed he s eps o p e ious au ho s
and looked in o he open sou ce communi y. In pa icu-
la , we ound he open sou ce D upal amewo k [4] o be a
1h p://spl2go.cs.o gu.de/
mo i a ing a iabili y-in ensi e sys em. D upal is a modu-
la web con en managemen amewo k w i en in PHP [4,
23]. D upal p o ides de ailed aul epo s including aul
desc ip ion, aul se e i y, ype, s a us and so on. Also,
mos o he modules o he amewo k include a numbe o
au oma ed es cases. The high numbe o he D upal com-
muni y membe s oge he wi h i s ex ensi e documen a ion
ha e also been s eng hs o choose his amewo k. D upal
is main ained and de eloped by a communi y o mo e han
630,000 use s and de elope s.
In his pape , we p opose using he D upal amewo k as
a mo i a ing case s udy o e alua e a iabili y es ing ech-
niques. We p opose mapping some o he main D upal mod-
ules o ea u es and ep esen he amewo k a iabili y us-
ing a ea u e model. The esul ing model has 28 ea u es, 27
c oss- ee cons ain s and ep esen s 96,768 di e en D upal
con igu a ions. Also, we epo on ex ensi e da a ex ac ed
om he D upal gi eposi o y and he D upal issue ack-
ing sys em e e ed o a pe iod o one yea . Fo each ea u e
unde s udy, we epo i s size, numbe o changes, numbe
o es cases and a ull aul epo . Faul s a e classi ied
acco ding o hei se e i y and he ea u e(s) ha igge
i . The s udy o aul s was eplica ed in wo consecu i e
D upal e sions, 7.22 and 7.23, o s udy he e olu ion in
he dis ibu ion o aul s. Among o he esul s, we iden i-
ied 390 aul s in o al, 11 aul s igge ed by he in e ac ion
be ween wo ea u es and a single aul caused by he in e -
ac ion among h ee ea u es in D upal 7.23. Fo he e al-
ua ion o ou wo k, we used he case s udy o e alua e he
e ec i eness o es case p io i iza ion based on his o ical o
aul s. Mo e speci ically, we used he in o ma ion abou he
aul s collec ed in D upal 7.22 o e ec i ely accele a e he
de ec ion o aul s o a pai wise sui e in D upal 7.23. In
summa y, his wo k con ibu es o gi e a new insigh in o
he dis ibu ion o aul s in a eal a iabili y-in ensi e sys-
em and he aul p opensi y o i s ea u es. Also, his may
be a aluable inpu o d i e he design and e alua ion o
a iabili y es ing app oaches.
The es o he pape is s uc u ed as ollows: Sec ion 2
in oduces he D upal amewo k. Sec ion 3 p esen s he
D upal ea u e model. The in o ma ion abou D upal ea-
u es (size, changes and es s) is p esen ed in Sec ion 4. The
desc ip ion abou aul s de ec ed in D upal 7.22 and 7.23
is p esen ed in Sec ion 5. Sec ion 6 summa izes he main
conclusions o ou s udy. Sec ion 7 shows a p elimina y
e alua ion o ou case s udy. Finally, we summa ize ou
conclusions in Sec ion 8.
2. THE DRUPAL FRAMEWORK
D upal is a highly modula open sou ce web con en man-
agemen amewo k implemen ed in PHP [4, 23]. This ool
can be used o build a a ie y o web si es including in e -
ne po als, e-comme ce applica ions and online newspape s
[23]. D upal is composed o a se o modules. A module is
a collec ion o unc ions ha p o ide ce ain unc ionali y
o he sys em. Ins alled modules in D upal can be enabled
o disabled. An enabled module is ac i a ed o be used by
he D upal sys em. A disabled module is deac i a ed and
adds no unc ionali y o he amewo k. The modules can be
classi ied in o co e modules and addi ional modules [4, 23].
The co e modules a e app o ed by he co e de elope s o he
D upal communi y and hey a e included by de aul in he
basic ins alla ion o D upal amewo k. Howe e , some o
Loca%za%on(Templa%ng(Syndica%on( Logging(
Basic(Con en (
Managemen (
Use (
Managemen (
Session(
Managemen ( URL(Aliasing(
Fo ums( WYSIWYG( E en (
Calenda s( Wo kg oups(
Image(
Galle ies( EEcomme ce( AdSense( Cus om(
Module(
Lib a y(o (Common(Func%ons(
Co e% Addi)onal%
Figu e 1: D upal co e and addi ional modules
hem can be enabled o disabled, while o he s mus always
be enabled. The co e modules a e esponsible o p o iding
he basic unc ionali y ha is used o suppo o he pa s o
he sys em. The D upal co e includes code ha allows he
sys em o boo s ap when i ecei es a eques , a lib a y o
common unc ions equen ly used wi h D upal, and mod-
ules ha p o ide basic unc ionali y like use managemen
and empla ing. Addi ional modules can be di ided in o con-
ibu ed modules and cus om modules. Con ibu ed modules
a e w i en by he D upal communi y and sha ed unde he
same GNU Public License (GPL) as D upal. Cus om mod-
ules a e hose c ea ed by he de elope o websi es ( hi d
pa ies). Fig. 1 depic s some popula co e and addi ional
D upal modules.
2.1 Module s uc u e
A he code le el, e e y D upal module is mapped o a di-
ec o y including he sou ce iles o he module. These iles
may include PHP iles, CSS s yleshee s, Ja aSc ip code,
es cases and help documen s. Also, e e y D upal mod-
ule mus include a .module ile and a .in o ile wi h me a
in o ma ion abou he module. Besides his, a module can
op ionally include he di ec o ies and iles o o he modules,
i.e. submodules. A submodule inc eases he unc ionali y o
he module con aining i .
A D upal .in o ile is a plain ex ile ha desc ibes he
basic in o ma ion equi ed o D upal o ecognize he mod-
ule. The name o his ile mus ma ch he name o he
module. This ile con ains a se o so-called di ec i es. A di-
ec i e is a p ope y name = alue. Some di ec i es can use
an a ay-like syn ax o decla e mul iple alues p ope ies,
name[] = alue. Any line ha begins wi h a semicolon (“;”)
is ea ed as a commen . Fo ins ance, Lis ing 1 desc ibes a
agmen o he iews.in o ile included in he Views module
o D upal 7.23:
Lis ing 1: F agmen o he ile iews.in o
name = Views
desc ip ion = C ea e cus omized lis s and que ies om you db.
package = Views
co e = 7.x
php = 5.2
s yleshee s[all][] = css/ iews.css
dependencies[] = c ools
; Handle s
iles[] = handle s/ iews_handle _a ea.inc
iles[] = handle s/ iews_handle _a ea_ esul .inc
iles[] = handle s/ iews_handle _a ea_ ex .inc
... mo e
; In o ma ion added by d upal.o g on 2013-04-09
e sion = "7.x-3.7"
co e = "7.x"
p ojec = " iews"
The s uc u e o .in o iles is s anda d ac oss all D upal 7
modules. The name and desc ip ion di ec i es speci y he
name and desc ip ion o he module ha will be displayed
in he D upal con igu a ion page. The package di ec i e
de ines which package o g oup o packages he module is
associa ed wi h. On he modules con igu a ion page, mod-
ules a e g ouped and displayed by package. The co e di-
ec i e de ines he e sion o D upal o which he module
was w i en. The php p ope y de ines he e sion o PHP
equi ed by he module. The iles di ec i e is an a ay wi h
he names o he iles o be loaded by D upal. Fu he mo e,
he .in o ile can op ionally include he dependencies ha
he module has wi h o he modules, i.e. modules ha mus
be ins alled and enabled o his module o wo k p ope ly.
In he example, he module Views depends on he module
C ools. The e exis some co e modules ha always ha e o
be ins alled and enabled, i.e. hose including he di ec i e
equi ed =TRUE.
2.2 Module es s
D upal modules can op ionally include a es di ec o y
wi h he es cases associa ed o he module. D upal de-
ines a es case as a class composed o unc ions (i.e. es s).
These es s a e pe o med h ough asse ions, a g oup o
me hods ha check o a condi ion and e u n a Boolean. I
i is TRUE, he es passes, i FALSE, he es ails. The e
exis h ee ypes o es s in D upal, uni , in eg a ion and
upg ade es cases. Uni es s a e me hods ha es an iso-
la ed piece o unc ionali y o a module, such as unc ions
o me hods. In eg a ion es s es how di e en componen s
(i.e. unc ionali y) wo k oge he . These es s may in ol e
any module o he D upal amewo k. In eg a ion es s usu-
ally simula e use in e ac ions wi h he g aphical use in-
e ace h ough HTTP messages. Acco ding o he D upal
documen a ion2, hese es s a e he mos common es s in
D upal. Upg ade es s a e used o de ec aul s caused by
he upg ade o a newe e sion o he amewo k, e.g. om
D upal 6.1 o 7.1. In o de o wo k wi h es s in D upal i
is necessa y o enable he SimpleTes module. This module
is a es ing amewo k mo ed in o co e in D upal 7. Sim-
pleTes au oma ically uns he es cases o all he ins alled
modules. Fig. 2 shows a snapsho o SimpleTes while un-
ning he es s (i.e. 328 es cases and 20,230 asse ions) o
D upal modules.
3. THE DRUPAL FEATURE MODEL
In his sec ion, we desc ibe he p ocess ollowed o model
D upal a iabili y using a ea u e model, depic ed in Fig. 3.
3.1 Fea u e ee
Acco ding o he D upal documen a ion, each module ha
is ins alled and enabled adds a new ea u e o he amewo k
[23]. Thus, we p opose modelling D upal modules as ea-
u es o he ea u e model. Also, when a module is ins alled,
new sub ea u es can be enabled adding ex a unc ionali y
o he module. These ea u es a e conside ed as child en
2h ps://d upal.o g/simple es
Figu e 2: Running D upal es s
ea u es o he module ha con ains hem. Fig. 3 shows he
D upal ea u es ha we e conside ed in ou s udy, 28 in o-
al including he ea u e oo . In pa icula , we i s selec ed
hose D upal co e modules ha mus be always enabled, 7
in o al, e.g. Node. In Fig. 3, hese ea u es appea wi h a
manda o y ela ion wi h he ea u es oo and Field. These
ea u es a e included in all D upal con igu a ions. A D upal
con igu a ion is a alid combina ion o ea u es ins alled and
enabled. Then, we andomly selec ed 20 modules wi hin he
es o he D upal co e modules (e.g. Image) and addi ional
modules (e.g. Views). All hese modules can be op ionally
ins alled and enabled and hus we e modelled as op ional
ea u es in he ea u e model.
Sub ea u es we e modelled conside ing as child en o o he
module hose submodules ha appea inside o i s module
di ec o y. These submodules p o ide ex a unc ionali y o
i s pa en module and hey ha e no meaning wi hou i .
As an example, he ea u e C ools p esen s se e al sub ea-
u es such as C ools access ulese ,C ools cus om con en
and Views con en . Excep ionally, he submodules o Node,
Blog,Book and Fo um, appea in sepa a e module olde s,
howe e , he desc ip ion o he modules in he D upal doc-
umen a ion indica es ha hese modules a e specializa ion-
s/ ypes o Node. Wi h espec o he ela ionships o and
al e na i e none o hem we e iden i ied among he ea u es
conside ed in Fig. 3.
3.2 C oss- ee cons ain s
We de ine he dependencies among modules as c oss- ee
cons ain s in he ea u e model. Cons ain s in ea u e
models a e ypically o he o m equi es o excludes. I
a ea u e A equi es a ea u e B, he inclusion o A in a con-
igu a ion implies he inclusion o B in such con igu a ion.
On he o he hand, i a ea u e A excludes a ea u e B, bo h
ea u es canno be pa o he same con igu a ion.
C oss- ee cons ain s we e iden i ied by manually inspec -
ing he dependencies di ec i e in he .in o ile o each mod-
ule. Fo each dependency, we c ea ed a equi es cons ain
in he ea u e model, 27 in o al. Fo ins ance, conside
he iews.in o ile depic ed in Lis ing 1. The ile indica es
ha Views depends on he C ools module, i.e. dependen-
cies[]=c ools. Thus, we es ablished a equi es cons ain be-
ween modules Views and C ools. No exclude cons ain s
we e iden i ied among he modules. In e es ingly, we ound
D upal 7.23
Node Fil e C ools
File
Op ions
Agg ega o
BookBlog
Fo um
C ools access
ulese
C ools cus om
con en
Views
con en
Field Field UI
Field SQL
s o age
Image
Lis
Numbe
Sys em Use Views
Views UI
Taxonomy Commen
Tex
Block
Tes ing
Fo um Requi es Taxonomy Op ions Requi es Field SQL s o age Views UI Requi es C ools
Fo um Requi es Op ions Field UI Requi es Field Taxonomy Requi es Op ions
Fo um Requi es Field Field UI Requi es Field SQL s o age Taxonomy Requi es Field
Fo um Requi es Field SQL s o age Image Requi es File Taxonomy Requi es Field SQL s o age
Fo um Requi es Commen Image Requi es Field Commen Requi es Tex
Fo um Requi es Tex Image Requi es Field SQL s o age Commen Requi es Field
Tex Requi es Field SQL s o age File Requi es Field Commen Requi es Field SQL s o age
Lis Requi es Field SQL s o age File Requi es Field SQL s o age
Lis Requi es Op ions Views con en Requi es Views
Numbe Requi es Field SQL s o age Views Requi es C ools
Figu e 3: D upal ea u e model
ha all modules in he same e sion o D upal a e expec ed
o wo k ine oge he . I a D upal module p esen s incom-
pa ibili ies wi h o he s, his is epo ed as a bug ha mus
be ixed. As a sani y check, we con i med he cons ain s
iden i ied using he JIT D upal module (Ja asc ip In oVis
Toolki ) which shows a g aphical ep esen a ion o he mod-
ules and hei ela ionships.
The a io o ea u es in ol ed in c oss- ee cons ain s o
he o al numbe o ea u es (CTCR) is 57.1%. The model
ep esen s 96,768 alid D upal con igu a ions.
4. DRUPAL FEATURES DATA
In his sec ion, we epo on he size, ecen changes and
es cases o he D upal ea u es shown in Fig. 3. These da a
a e o en used as good indica o s o he e o -p oneness o
a so wa e applica ion. Addi ionally, his may p o ide e-
sea che s and p ac i ione s wi h help ul in o ma ion abou
he cha ac e is ics o ea u es in a eal a iabili y-in ensi e
applica ion. In pa icula , o each D upal ea u e, we col-
lec ed he ollowing da a:
Fea u e size. This p o ides a ough idea o he complex-
i y o each ea u e and i s e o p oneness. The size o a
ea u e was calcula ed in e ms o i s numbe o Lines o
Code (LoC). LoC we e coun ed using he wc Linux com-
mand on each one o he sou ce iles included in he module
di ec o y associa ed o each ea u e. The command used is
shown below. Blank lines and commen s we e included in
he coun ing o simplici y. Tes iles we e excluded om
he coun ing. Table 1 depic s he numbe o LoC o each
ea u e. The sizes anged be ween 326 LoC ( ea u e C ools
cus om con en ) and 70,372 LoC ( ea u e Views). I is no e-
wo hy ha sub ea u es a e signi ican ly smalle han hei
espec i e pa en ea u es.
wc -l ‘ ind . -iname "module*" -o -iname "o he ile*"‘
Numbe o changes. Changes in he code a e likely o
in oduce aul s [24]. Thus, he numbe o changes in a
ea u e may be a good indica o o i s e o p oneness and
could help us o p edic aul s in he u u e. To ob ain he
numbe o changes made in each ea u e, we collec ed he
commi s om he gi eposi o y o D upal3. The sea ch was
na owed by ocusing on he changes pe o med du ing a
pe iod o one yea , om Sep embe 30 h 2012 o Sep embe
29 h 2013. Fi s , we cloned he en i e D upal 7.x eposi o y.
Then, we applied he console command showed below o ge
he numbe o commi s by module, e sion and da e. As
illus a ed in Table 1, he numbe o changes anged be ween
0 ( ea u e Blog) and 43 ( ea u e Views). The modules wi h
he highes numbe o commi s a e Views (43) and C ools
(24). I is no ewo hy ha hese a e he only addi ional
modules conside ed in Fig. 3, i.e. no included in D upal
co e.
gi log --p e y=oneline --a e ={2012-09-30}
--be o e={2013-09-29} 7.22..7.23 name_module | wc -l
Numbe o es s. Table 1 shows he o al numbe o es
cases and asse ions o each ea u e, ob ained om he ou -
pu o he SimpleTes module. In o al, D upal ea u es
include 328 es cases and 20,230 asse ions. In some cases
3h p://d upalcode.o g/p ojec /d upal.gi
such as C ools, he numbe o es s (7 es cases and 121 as-
se ions) seems low conside ing ha he size o he ea u e
is o e 9,000 LoC. I is also no ewo hy ha some o he
sub ea u es include no es cases.
Fea u e LoC Changes Tes s
TCs Asse s.
Agg ega o 4,384 1 12 985
Block 3,806 3 9 675
Blog 647 - 1 244
Book 2,734 1 1 531
Commen 6,547 1 14 3,287
C ools 9,267 24 7 121
C ools access ulese 385 1 - -
C ools cus om con en 326 - - -
Field 9,561 9 9 870
Field SQL s o age 1,428 2 1 94
Field UI 3,895 3 3 287
File 2,428 1 39 2,293
Fil e 5,206 3 9 958
Fo um 3,262 2 2 677
Image 6,570 3 13 811
Lis 638 1 3 232
Node 11,397 4 32 1,391
Numbe 632 1 1 87
Op ions 1,057 1 2 227
Sys em 23,478 16 58 2,138
Taxonomy 6,625 4 14 677
Tes ing 3,002 - 12 219
Tex 1,286 1 3 444
Use 10,117 12 23 1,355
Views 70,372 43 51 1,089
Views con en 3,468 1 - -
Views UI 2,462 2 9 538
To al 194,980 140 328 20,230
Table 1: Fea u e da a
5. FAULTS IN DRUPAL
In his sec ion, we epo he aul s de ec ed in he D u-
pal modules shown in Fig. 3. The collec ion o aul s was
ca ied ou on he D upal issue acking sys em4. To ge he
in o ma ion abou aul s, D upal o e s a web-based sea ch
ool o il e he aul s by se e i y, s a us, module and D u-
pal e sion. The sea ches on D upal issue acking we e
pe o med in he da e ange om Sep embe 30 h 2012 o
Sep embe 29 h 2013. We collec ed he aul s o wo consec-
u i e D upal e sions, 7.22 and 7.23. We conside ed bo h
e sions o achie e a be e unde s anding o he e olu ion
o a eal sys em and due o he exis ing in e es in ob aining
his o ical o aul s o es ing [5, 10].
The p ocess o collec ing aul s was manual. Fi s , we il-
e ed he aul s by he name o he module (i.e. label
name module.module in he issue acking sys em). Second,
we e ined he sea ch by he D upal e sions 7.23 and 7.22
and he da es p e iously men ioned. A o al o 453 aul s
ma ched he ini ial sea ch o D upal 7.23 and a o al o
481 aul s o D upal 7.22. In o de o iden i y he aul s
4h ps://d upal.o g/p ojec /issues

Module
D upal 7.22 D upal 7.23
Se e i y To al Se e i y To al
Mino No mal Majo C i ical Mino No mal Majo C i ical
Agg ega o - 6 1 - 7 - 7 - - 7
Block 2 12 2 1 17 2 9 2 1 14
Blog - 2 1 - 3 - 1 1 - 2
Book 1 5 - - 6 1 4 - - 5
Commen 2 16 2 1 21 2 13 2 2 19
C ools 7 130 20 8 165 7 130 20 8 165
Field UI 3 7 - - 10 3 4 - - 7
File 1 14 3 - 18 1 15 3 - 19
Fil e - 10 1 - 11 - 9 1 - 10
Fo um - 3 1 - 4 - 3 - - 3
Image - 9 1 1 11 - 9 1 1 11
Node 2 17 - 2 21 2 16 - 2 20
Numbe - 1 - - 1 - - - - -
Sys em 4 23 4 2 33 4 20 4 2 30
Taxonomy - 20 4 1 25 - 23 4 1 28
Tes ing - 4 2 1 7 - 4 2 1 7
Tex - 5 - - 5 - 4 - - 4
Use 4 15 - - 19 4 13 - - 17
Views - 2 - - 2 - 1 - - 1
Views Con en 1 18 2 - 21 1 18 2 - 21
To al 27 319 44 17 407 27 303 42 18 390
Table 2: Faul s in D upal 7.22 and 7.23
caused by he in e ac ion o se e al modules among all he
aul s ound, we i s ied a simila app oach o he one
p esen ed by A ho e al. [2] in he con ex o ope a ing
sys ems. This is, o each module, we sea ched o aul s
desc ip ions con aining he keywo ds “b eak”, “con lic ” o
“o e w i e”. Howe e , he sea ch did no e u n any esul
ela ed wi h in e ac ion among modules. Thus, we decided
o ollow a di e en app oach. Fo each module, we sea ched
o aul s desc ip ions and ags including he name (wi hou
dis inguishing be ween lowe and uppe case) o any o he
o he modules in Fig. 3. Fo ins ance, i a epo ed aul
in he module C ools included he name o he module Tax-
onomy in i s desc ip ion o ags, we selec ed he aul as a
candida e in eg a ion aul be ween C ools and Taxonomy.
Then, we manually checked which o he aul s ound we e
ac ually in eg a ion aul s. Se e al o he candida e in eg a-
ion aul s we e disca ded a e checking hem. A o al o
18 aul s ma ched in he sea ch o bo h e sions: 17 ou o
18 we e caused by he in e ac ion o wo modules and one o
hem by he in e ac ion o h ee modules. I is possible ha
some epo e s could ha e no included he in ol ed module
names in some in e ac ion aul s. As a esul , some o hese
aul s could ha e no been iden i ied.
Then, he sea ch was e ined o elimina e hose aul s ha
we e no accep ed by he D upal communi y, namely: du-
plica ed bugs,non ep oducible bugs and bugs wo king as
designed. The la e a e issues ha ha e been conside ed
no o be a bug because he epo ed beha iou was ei he
an in en ional pa o he p ojec , o he issue was caused
by cus omiza ions manually applied by he use . This le
390 alid aul s in D upal 7.23 and 407 in D upal 7.22.
Table 2 summa izes he aul s ound in bo h e sion o
D upal modules. Fo each module, he o al numbe o
aul s (single and in e ac ion aul s) is p esen ed classi y-
ing hem acco ding o hei se e i y le el. In e ac ion aul s
a e coun ed in he modules in which hey we e de ec ed.
Only he modules in which we ound aul s a e shown. We
may ema k ha he e exis modules such as C ools ha do
no disc imina e among e sions o D upal 7, i.e. hey only
speci y 7.x. This explains he same numbe and se e i y
o aul s ha appea in Table 2 o C ools. On he o he
hand, he e exis some aul s ha appea in D upal 7.22
and con inue in 7.23. Fo example, he same mino aul o
he module Book appea s in bo h e sions. This is no he
case o he module Agg ega o ha p esen s he same num-
be o aul s in 7.23 and 7.22 bu he aul s a e di e en .
Among he alid aul s ound, we iden i ied 12 in eg a ion
aul s. The ype o in e ac ion and he name o he mod-
ules causing he aul a e p esen ed in Table 3. Among he
in eg a ion aul s de ec ed, 11 we e caused by he in e ac-
ion be ween wo modules and jus one was caused by he
in e ac ion among h ee modules. All o hese aul s we e
iden i ied in bo h D upal e sions. I is no ewo hy ha 8
ou o 12 aul s we e caused by he in e ac ion o C ools wi h
o he module. C ools p o ides a se o APIs and ools o
modules o imp o e he de elope expe ience. Also, 6 ou
o 12 aul s we e igge ed by he in e ac ion o he module
Use wi h o he module. The module Use allows use s o
egis e , log in, and log ou and suppo s use oles and pe -
missions. Finally, since he p ocess o aul collec ion was
manual, he e exis s he possibili y ha some aul s could
ha e no been co ec ly iden i ied.
6. DISCUSSION
Nex , we p esen some o he conclusions o ou s udy:
Rela ion be ween aul s and ea u e size. We ound
ha 7 ou o he 10 la ges ea u es we e also among he
10 ea u es wi h a highe numbe o aul s. Simila ly, 8
ou o 10 smalles ea u es we e among he 10 ea u es wi h
less aul s. This sugges s ha he e exis s a ce ain co e-
la ion be ween he size o a ea u e and he p obabili y o
aul s on i . Howe e , we ound some excep ions. Fo in-
s ance, he la ges ea u e in D upal, Views, has a single
epo ed aul . Con e sely, he ea u e wi h a highes num-
be o aul s, C ools, is he six h la ges ea u e in D upal.
Thus, we conclude ha he size o a ea u e could be in e -
p e ed as a ough es ima ion o i s e o -p oneness.
Rela ion be ween aul s and changes. We ound ha 6
ou o he 10 ea u es wi h a highe numbe o changes we e
also among he 10 ea u es wi h a highe numbe o aul s.
In ac , he wo ea u es wi h he highes numbe o aul s,
C ools and Sys em, we e he second and hi d ea u es wi h
mo e changes du ing he las 12 mon hs espec i ely. We
also obse ed ha he ea u es wi h ew o no changes a all
we e among hose wi h a lowe numbe o aul s. This also
sugges s a co ela ion be ween he numbe o changes in a
ea u e and he p obabili y o aul s on i .
Rela ion be ween aul s and ype o ea u es. We
iden i ied 81 aul s in he co e ea u es o he D upal ea u e
model (81/7 = 11.5 aul s pe ea u e) and 309 aul s in he
op ional ea u es (309/20 = 15.4 aul s pe ea u e). Co e
ea u es ha e a lowe a io o aul s pe ea u e. We p esume
ha his is due o he ac ha co e ea u es a e included
in all D upal con igu a ions and hus hey a e be e sup-
po ed and mo e s able. Rega ding aul se e i y, a ound
78% o he aul s we e classi ied as no mal, 11% as majo ,
7% as mino and 4% as c i ical. We ound no co ela ion
be ween aul se e i y and he ypes o ea u es.
Rela ion be ween aul s and CTC (C oss-T ee Con-
s ain s). In [6], Baghe i e al. s udied se e al ea u e
model me ics and sugges ed ha hose ea u es in ol ed in
a highe numbe o CTCs a e mo e e o -p one. To explo e
his ac , we analyzed he ea u es in ol ed in cons ain s
in o de o s udy he ela ion be ween hem and hei aul
p opensi y. In pa icula , we ound ha only 4 ou o he 10
ea u es in ol ed in a highe numbe o CTCs we e among
he 10 ea u es wi h a highe numbe o aul s. In ac , he
h ee ea u es in ol ed in a highe numbe o cons ain s in
D upal, Field SQL s o age (10), Field (6) and Fo um (6)
had jus h ee aul s in o al. The e o e, we conclude ha
he co ela ion be ween ea u e in ol emen in CTCs and
aul p opensi y is no con i med in ou s udy.
Tes cases. We ound a clea ela ion be ween he size o
he ea u es and he numbe o es cases and asse ions in-
cluded as a pa o hei sou ce code. Hence, o ins ance,
he wo la ges D upal ea u es, Sys ems and View, we e
hose including a la ge numbe o es cases, 58 and 51
espec i ely. This seems easonable since la ge p og am e-
qui e a highe numbe o es s. Rega ding he dis ibu ion o
asse ions in he es cases esul s we e dispa a e. As an ex-
ample, ea u e Sys em includes 2,138 asse ions dis ibu ed
along 58 es cases while ea u e Commen includes 3,287
asse ions dis ibu ed along 14 es cases. We p esume his
is simply due o di e en de elope ’s p ac ices.
n-wise Modules in ol ed
2 Blog + Use
2 Commen + Use
2 C ools + Use
2 C ools + Tex
2 C ools + Use
3 C ools + Views con en + Fil e
2 C ools + Tex
2 C ools + Views
2 C ools + Taxonomy
2 C ools + Taxonomy
2 Sys em + Use
2 Use + Image
Table 3: In e a ion aul s in D upal
7. EVALUATION
P e ious s udies show ha he de ec ion o aul s in an
applica ion can be accele a ed by es ing i s hose compo-
nen s ha showed o be mo e e o -p one in p e ious e -
sions o he so wa e. This is e e ed o as his o y-based es
case p io i iza ion [5, 10]. In his sec ion, we use he D upal
case s udy o e alua e whe he his p io i iza ion echnique
could con ibu e o accele a e he de ec ion o aul s o a
pai wise es sui e. Pai wise is a well-known es case se-
lec ion app oach ha gene a es all possible combina ions o
pai s o ea u es based on he obse a ion ha mos aul s
o igina e om a single ea u e o by he in e ac ion o wo
ea u es [16]. In his scena io, we de ine a es case as a
con igu a ion o he D upal amewo k o be es ed, i.e. a
se o ea u es.
The e alua ion was pe o med in se e al s eps. Fi s ,
we ansla ed he D upal ea u e model o SPLX o ma
o make i machine-p ocessable [14]. Then, we seeded he
model wi h he aul s de ec ed in D upal 7.23, 390 in o al.
To his pu pose, we c ea ed a lis o aul y ea u e se s. Each
se simula ed aul s igge ed by n ea u es (n∈[1,3]). Fo
ins ance, he lis {{Node}{C ools,Use }} simula es a aul
in he ea u e Node and ano he aul caused by he in e -
ac ion be ween he C ools and Use ea u es. Second, we
used he SPLCAT ool [11] o gene a e a pai wise sui e o
he D upal ea u e model. As a esul , we ob ained a se
o 10 es cases (ou o 96,768) ha co e ed all he possible
pai s o ea u e combina ions. Then, we checked whe he
he pai wise sui e de ec ed he seeded aul s. We consid-
e ed ha a es case de ec s a aul i he es case includes
he ea u e(s) ha igge he aul . The pai wise es sui e
de ec ed all he seeded aul s.
Nex , we measu ed how as he aul s we e de ec ed by
he pai wise sui e calcula ing he APFD (A e age Pe cen -
age o Faul s De ec ed) me ic [18]. The APFD me ic mea-
su es he weigh ed a e age o he pe cen age o aul s de-
ec ed du ing he execu ion o he es sui e [18]. To o mally
illus a e APFD, le Tbe a es sui e which con ains n es
cases, and le Fbe a se o m aul s e ealed by T. Le TFi
be he posi ion o he i s es case in o de ing T’ o Twhich
e eals he aul i. The APFD me ic o he es sui e T’ is
gi en by he equa ion: APFD = 1−T F1+T F2+...+T Fn
n×m+1
2n.
APFD alue anges om 0 o 1.
The esul ing APFD alue o he pai wise sui e was 86.2%.
Then, we p io i ized he es sui e placing i s hose es
cases ha included he ea u es wi h mo e aul s in he p e-
ious e sion o D upal. Mo e speci ically, each es case
was gi en a p io i y alue equal o he sum o he aul s
de ec ed in i s ea u es in D upal 7.22. Then, he lis was
o de ed scheduling i s o es ing hose es cases wi h a
highe p io i y alue. The APFD me ic o he p io i ized
sui e was 94.2%, imp o ing up o 8 poin s he APFD o he
pai wise sui e (86.2%). This sugges s ha using aul his-
o y in o ma ion con ibu es o accele a e he de ec ion o
aul s o es sui es based on combina o ial es ing.
8. CONCLUSIONS
In his pape , we p esen ed he D upal amewo k as a
mo i a ing eal a iabili y-in ensi e sys em. In con as o
he ela ed wo k on a iabili y modelling in open-sou ce sys-
ems, we ocused on he es ing pe spec i e p o iding de-
ailed da a abou he aul dis ibu ion and aul s p open-
si y o i s ea u es. Also, we epo ed on how es cases a e
dis ibu ed and a anged in he amewo k p o iding help-
ul eedback o esea che s and p ac i ione s in he ield o
a iabili y. All hese da a may be a aluable asse o e alu-
a e a iabili y es ing echniques in a eal se up a he han
using andom a iabili y models and simula ed aul s. To
show he easibili y o ou wo k, we used he case s udy o
e alua e he e ec i eness o a his o y-based es case p i-
o i iza ion c i e ion. Resul s sugges ha his app oach in
combina ion wi h pai wise could con ibu e o accele a e he
de ec ion o aul s in a iabili y-in ensi e sys ems.
9. ACKNOWLEDGMENTS
This wo k was pa ially suppo ed by he Eu opean Com-
mission (FEDER), he Spanish and he Andalusian R&D&I
p og ammes (g an s TIN2009-07366 (SETI), TIN2012-32273
(TAPAS), TIC-5906 (THEOS), TIC-1867 (COPAS)).
10. REFERENCES
[1] S. Apel, A. on Rhein, P. Wendle , A. G ¨
oBlinge ,
and D. Beye . S a egies o p oduc -line e i ica ion:
case s udies and expe imen s. In In e na ional
Con e ence on So wa e Enginee ing, 2013.
[2] C. A ho, K. Suzaki, R. Di Cosmo, R. T einen, and
S. Zacchi oli. Why do so wa e packages con lic ? In
Con e ence on Mining So wa e Reposi o ies, 2012.
[3] D. Bena ides, S. Segu a, and A. Ruiz-Co ´es.
Au oma ed analyses o ea u e models 20 yea s la e :
A li e a u e e iew. In o ma ion Sys ems, 2010.
[4] D. Buy ae . D upal F amewo k.
h p://www.d upal.o g, accessed in Oc obe 2013.
[5] E. C. P. C is ian Simons. Reg ession es cases
p io i iza ion using ailu e pu sui sampling. In ISDA,
2010.
[6] F. Ensan, E. Baghe i, and D. Gase ic. E olu iona y
sea ch-based es gene a ion o so wa e p oduc line
ea u e models. In Con e ence on Ad anced
In o ma ion Sys ems Enginee ing (CAiSE’12), 2012.
[7] J. A. Galindo, D. Bena ides, and S. Segu a. Debian
packages eposi o ies as so wa e p oduc line models.
owa ds au oma ed analysis. In ACOTA, 2010.
[8] J. Guo, J. Whi e, G. Wang, J. Li, and Y. Wang. A
gene ic algo i hm o op imized ea u e selec ion wi h
esou ce cons ain s in so wa e p oduc lines. Jou nal
o Sys ems and So wa e, 2011.
[9] C. Hena d, M. Papadakis, G. Pe ouin, J. Klein,
P. Heymans, and Y. L. T aon. Bypassing he
combina o ial explosion: Using simila i y o gene a e
and p io i ize -wise es sui es o la ge so wa e
p oduc lines. Technical epo , 2012.
[10] Y.-C. Huang, C.-Y. Huang, J.-R. Chang, and T.-Y.
Chen. Design and analysis o cos -cognizan es case
p io i iza ion using gene ic algo i hm wi h es
his o y. In Compu e So wa e and Applica ions
Con e ence, 2010.
[11] M. F. Johansen, O. Haugen, and F. Fleu ey.
P ope ies o ealis ic ea u e models make
combina o ial es ing o p oduc lines easible. In
MODELS, 2011.
[12] M. F. Johansen, O. Haugen, F. Fleu ey, A. G.
Eldega d, and T. Sy e sen. Gene a ing be e pa ial
co e ing a ays by modeling weigh s on sub-p oduc
lines. In In e na ional Con e ence MODELS, 2012.
[13] R. Lo u o, S. She, T. Be ge , K. Cza necki, and
A. Wasowski. E olu ion o he linux ke nel a iabili y
model. In SPLC, 2010.
[14] M. Mendonca. E icien Reasoning Techniques o
La ge Scale Fea u e Models. PhD hesis, Uni e si y o
Wa e loo, 2009.
[15] M. Mendonca, M. B anco, and D. Cowan. S.p.l.o. . -
so wa e p oduc lines online ools. In OOPSLA, 2009.
[16] G. Pe ouin, S. Os e , S. Sen, J. Klein, B. Bud y, and
Y. le T aon. Pai wise es ing o so wa e p oduc
lines: compa ison o wo app oaches. So wa e Quali y
Jou nal, 2011.
[17] G. Pe ouin, S. Sen, J. Klein, B. Baud y, and
Y. le T aon. Au oma ed and scalable -wise es case
gene a ion s a egies o so wa e p oduc lines. In
Con e ence So wa e Tes ing, Ve i ica ion and
Valida ion, 2010.
[18] G. Ro he mel, R. Un ch, C. Chu, and M. Ha old.
P io i izing es cases o eg ession es ing. IEEE
T ans. So wa e Eng, 27:929–948, 2001.
[19] A. B. S´anchez, S. Segu a, and A. Ruiz-Co ´es. A
compa ison o es case p io i iza ion c i e ia o
so wa e p oduc lines. Technical epo , Uni e si y o
Se ille, Spain., 2013.
[20] S. Segu a and A. Ruiz-Co ´es. Benchma king on he
au oma ed analyses o ea u e models: A p elimina y
oadmap. In VaMoS, 2009.
[21] S. She, R. Lo u o, T. Be ge , A. Wasowski, and
k zysz o Cza necki. The a iabili y model o he linux
ke nel. In In e na ional Wo kshop on Va iabili y
Modelling o So wa e-in ensi e Sys ems, 2010.
[22] H. S ikan h, M. B. Cohen, and X. Qu. Reducing ield
ailu es in sys em con igu able so wa e: Cos -based
p io i iza ion. 2009.
[23] T. Tomlinson and J. K. VanDyk. P o D upal 7
de elopmen : hi d edi ion. 2010.
[24] S. Yoo and M. Ha man. Reg ession es ing
minimisa ion, selec ion and p io i isa ion: A su ey. In
So wa e Tes ing, Ve i ica ion and Reliabili y, 2012.