FaMa Test Suite v1.2: ISA Technical Report ISA-10-TR-01

FaMa Tes Sui e 1.2 Se gio Segu a, Da id Bena ides and An onio Ruiz-Co ´es {se giosegu a,bena ides,a uiz}@us.es Applied So wa e Enginee ing Resea ch G oup Uni e si y o Se ille, Spain Ma ch 2010 Technical Repo ISA-10-TR-01 This epo was p epa ed by he Applied So wa e Enginee ing Resea ch G oup (ISA) Depa men o compu e languages and sys ems A / Reina Me cedes S/N, 41012 Se ille, Spain h p://www.isa.us.es/ Copy igh c 2010 by ISA Resea ch G oup. Pe mission o ep oduce his documen and o p epa e de i a i e wo ks om his docu- men o in e nal use is g an ed, p o ided he copy igh and ’No Wa an y’ s a emen s a e included wi h all ep oduc ions and de i a i e wo ks. NO WARRANTY THIS ISA RESEARCH GROUP MATERIAL IS FURNISHED ON AN ’AS-IS’ BASIS. ISA RESEARCH GROUP MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIM- ITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTIBILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. Use o any adema ks in his epo is no in ended in any way o in inge on he igh s o he adema k holde Suppo : This wo k has been pa ially suppo ed by he Eu opean Commission (FEDER) and Spanish Go e nmen unde CICYT p ojec SETI (TIN2009-07366) and he Andalusian Go e nmen p ojec ISABEL (TIC-2533). Lis o changes Ve sion Da e Desc ip ion 1.0 Feb ua y 2009 Fi s elease 1.1 Decembe 2009 Re ined e alua ion wi h mu an s. New es cases added 1.2 Ma ch 2010 Tes cases o he ope a ion ValidP oduc upda ed. 3 FaMa Tes Sui e 1.2 Se gio Segu a, Da id Bena ides and An onio Ruiz-Co ´es Depa men o Compu e Languages and Sys ems A Reina Me cedes S/N, 41012 Se ille, Spain {se giosegu a, bena ides, a uiz}@us.es Abs ac : A Fea u e Model (FM) is a compac ep esen a ion o all he p oduc s o a so wa e p oduc line. Au oma ed analysis o FMs is apidly gaining impo ance: new ope a ions o analysis ha e been p oposed, new ools ha e been de eloped o suppo hose ope a ions and di e en logical pa adigms and algo i hms ha e been p oposed o pe o m hem. Implemen ing ope a ions is a complex ask ha easily leads o e o s in analysis solu ions. In his con ex , he lack o speci ic es ing mechanisms is becoming a majo obs acle hinde ing he de elopmen o ools and a ec ing hei quali y and eliabili y. In his pape , we p esen FaMa Tes Sui e, a se o implemen a ion–independen es cases o alida e he unc ionali y o FM analysis ools. This is an e icien and handy mechanism o assis in he de elopmen o ools, de ec ing aul s and imp o ing hei quali y. In o de o show he e ec i eness o ou p oposal, we e alua ed he sui e using mu a ion es ing as well as eal aul s and ools. Ou esul s a e p omising and di ec ly applicable in he es ing o analysis solu ions. We in end his pape o be a i s s ep owa d he de elopmen o a widely accep ed es sui e o suppo unc ional es ing in he communi y o au oma ed analysis o ea u e models. Key Wo ds: Tes sui e, unc ional es ing, ea u e models, au oma ed analysis, so wa e p oduc lines 1 In oduc ion So wa e P oduc Line (SPL) enginee ing is an app oach o de elop amilies o ela ed sys ems based on he usage o eusable asse s as a means o imp o e so wa e quali y while educing p oduc ion cos s and ime– o–ma ke [1]. P oduc s in so wa e p oduc lines a e speci ied in e ms o ea u es. A ea u e is de ined as an inc emen in p oduc unc ionali y [2]. Key o SPLs is o cap u e commonali ies (i.e. common ea u es) and a iabili ies (i.e. a ian ea u es) o he sys ems ha belong o he p oduc line. To his aim, ea u e models a e commonly used. A ea u e model [3, 4] is a compac ep esen a ion o all he p oduc s o a p oduc line in e ms o ea u es and ela ionships among hem (see Figu e 1). The au oma ed analysis o ea u e models deals wi h he au oma ed ex ac ion o in o ma ion om ea u e models. Typical ope a ions o analysis allow de e mining whe he a ea u e model is oid (i.e. i ep esen s no p oduc s), whe he i con ains e o s (e.g. ea u es ha canno be pa o any p oduc ) o wha is he numbe o p oduc s o he SPL ep esen ed by he model. Ca alogues wi h up o 30 di e en analysis ope a ions on ea u e models ha e been epo ed in he li e a u e [5, 6]. Analysis solu ions can be ca ego ized in hose using p oposi ional logic [7, 8, 9, 10, 11, 12, 13], cons ain p og amming [14, 15, 16], desc ip ion logic [17, 18] and ad-hoc algo i hms and echniques [19, 20, 21]. The e a e also a numbe o ools suppo ing hese analysis capabili ies such as AHEAD Tool Sui e [22], FaMa F amewo k [23], Fea u e Model Plug-in [24] and pu e:: a ian s [25]. The implemen a ion o analysis ope a ions is a ha d ask in ol ing complex da a s uc u es and algo i hms. This makes he de elopmen o analysis ools a om i ial and easily leads o e o s inc easing de elopmen ime and educing hei eliabili y. Gaining con idence in he absence o de ec s in hese ools is essen ial since he in o ma ion ex ac ed om ea u e models is used o suppo decisions all along he SPL de elopmen p ocess [2]. Howe e , he lack o speci ic es ing mechanisms in his con ex appea s as a majo obs acle o enginee s when ying o assess he unc ionali y and quali y o hei p og ams. Hence, i is known ha so wa e es ing accoun s o abou 50% o he o al cos o so wa e de elopmen [26]. So wa e es ing in ends o e eal aul s in he so wa e unde es [27, 28]. This is mainly done by checking he beha iou o he p og am wi h se o inpu -ou pu combina ions (i.e. es cases). The main challenge when es ing is o ind a balance be ween he numbe o es cases and hei 1 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 e ec i eness [28]. Hence, ying o be exhaus i e when es ing so wa e ools may easily inc ease he numbe o es cases o an unmanageable le el. Simila ly, using a educed numbe o inpu -ou pu combina ions may esul in a weak e ec i eness. In his pape , we p esen a se o implemen a ion–independen es cases o alida e he unc- ionali y o ea u e model analysis ools. Th ough he implemen a ion o ou es cases, aul s can be apidly de ec ed assis ing in he de elopmen o ea u e model analysis ools and imp o ing hei eliabili y and quali y. Fo i s design and e alua ion, we used popula echniques om he so wa e es ing communi y o assis us on he c ea ion o a ep esen a i e se o inpu -ou pu com- bina ions. These es cases can be used ei he in isola ion o as a sui able complemen o u he es ing me hods such as whi e–box es ing echniques [27, 29] o au oma ed es da a gene a o s [30]. As sugges ed by he es ing li e a u e, each es case was designed o e eal a single ype o aul . This allows use s o iden i y clea ly he sou ce o a aul once i has been de ec ed. B ie ly, we nex desc ibe he main cha ac e is ics o ou sui e: – Ope a ions es ed. Cu en e sion o ou sui e, called FaMa Tes Sui e, add esses 7 ou o 30 analysis ope a ions on ea u e models iden i ied in he li e a u e [5]. These we e selec ed o hei ex ended use in he communi y o au oma ed analysis and hei he e ogeneous na u e. – Tes ing echniques. Fou black-box es ing echniques we e used o design es cases, namely: equi alence pa i ioning, bounda y- alue analysis, pai wise es ing and e o guessing. A p e- limina y e alua ion o he es ing echniques o be used in ou app oach was p esen ed in [31]. – Tes cases. The sui e is composed by 192 es cases. Each es case is designed in e ms o he inpu s (i.e. ea u e models and some o he pa ame e s) and expec ed ou pu s o he analysis ope a ions unde es . – Adequacy. We e alua ed he e ec i eness o ou sui e using mu a ion es ing and eal aul s as ollows. Fi s ly, we gene a ed hund eds o aul y e sions (so-called mu an s) o h ee open sou ce analysis ools in eg a ed in o he FaMa amewo k. Then, we execu ed ou sui e agains hose aul y ools and check how many aul s we e de ec ed by ou es cases. As a esul , he sui e iden i ied 96.1% o he aul s showing he easibili y o ou p oposal. We hen e ined ou sui e un il ob aining a sco e o 100%. Ou e ined sui e also showed o be e ec i e in de ec ing mo i a ing aul s ound in he li e a u e and in a ecen elease o he FaMa amewo k. The emainde o he pape is s uc u ed as ollows: Sec ion 2 p esen s ea u e models, hei analyses and black-box es ing echniques in a nu shell. A de ailed desc ip ion o how we designed ou es cases is p esen ed in Sec ion 3. Sec ion 4 desc ibes he adequacy e alua ion and e inemen o he sui e. A summa y o he e ined sui e and a b ie discussion is p esen ed in Sec ion 5. Finally, we summa ize ou main conclusions and desc ibe ou u u e wo k in Sec ion 6. 2 P elimina ies This sec ion p o ides an o e iew o ea u e models, hei analysis and he black–box es ing echniques used in ou app oach. 2.1 Fea u e models A ea u e model [3, 4] ep esen s all he p oduc s o an SPL in a single model in e ms o ea u es and ela ionships among hem. I is o ganized hie a chically and is g aphically depic ed as a ea u e diag am. Figu e 1 shows a simpli ied example o a ea u e model ep esen ing an e-comme ce SPL. The model illus a es how ea u es a e used o speci y he commonali ies and a iabili ies o he on-line shopping sys ems ha belong o he p oduc line. The oo ea u e (i.e. E-Shop) iden i ies 2 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 E-Shop Sea ch Basic Ad anced Ca alogue In o Image P ice Desc ip ion Secu i y Medium High Paymen PC Bank D a O e s Mobile GUI C edi Ca d Visa Ame ican Exp ess Manda o y Op ional Al e na i e O Requi es Excludes Banne s Figu e 1: A sample ea u e model he SPL. The ela ionships be ween a pa en ea u e and i s child ea u es can be mainly di ided in o: Manda o y. A child ea u e has a manda o y ela ionship wi h i s pa en when he child is in- cluded in all p oduc s in which i s pa en ea u e appea s. Fo ins ance, e e y on-line shopping sys em in ou example mus implemen a ca alogue o p oduc s. Op ional. A child ea u e has an op ional ela ionship wi h i s pa en when he child can be op ionally included in all p oduc s in which i s pa en ea u e appea s. Fo ins ance, banne s is de ined as an op ional ea u e. Al e na i e. A se o child ea u es ha e an al e na i e ela ionship wi h hei pa en when only one ea u e o he child en can be selec ed when i s pa en ea u e is pa o he p oduc . In ou SPL, a shopping sys em may implemen high o medium secu i y policy bu no bo h in he same p oduc . O -Rela ion. A se o child ea u es ha e an o - ela ionship wi h hei pa en when one o mo e o hem can be included in he p oduc s in which i s pa en ea u e appea s. A shopping sys em can implemen se e al paymen modules: bank d a ,c edi ca d o bo h o hem. No ice ha a child ea u e can only appea in a p oduc i i s pa en ea u e does. The oo ea u e is a pa o all he p oduc s wi hin he SPL. In addi ion o he pa en al ela ionships be- ween ea u es, a ea u e model can also con ain c oss- ee cons ain s be ween ea u es. These a e ypically o he o m: Requi es. I a ea u e A equi es a ea u e B, he inclusion o A in a p oduc implies he inclusion o B in such p oduc . On-line shopping sys ems accep ing paymen s wi h c edi ca d mus imple- men a high secu i y policy. Excludes. I a ea u e A excludes a ea u e B, bo h ea u es canno be pa o he same p oduc . Shopping sys ems implemen ing a mobile GUI canno include suppo o banne s. Fea u e models we e i s in oduced as a pa o he FODA (Fea u e-O ien ed Domain Analy- sis) me hod back in 1990 [4]. Since hen, ea u e modelling has been widely adop ed by he so wa e p oduc line communi y and a numbe o ex ensions ha e been p oposed in a emp s o imp o e p ope ies such as succinc ness and na u alness. We e e he eade o [6] o a de ailed su ey on 3 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 he di e en ea u e modelling languages. 2.2 Au oma ed analyses o ea u e models The au oma ed analysis o ea u e models deals wi h he compu e –aided ex ac ion o in o ma- ion om ea u e models [5]. F om he in o ma ion ob ained, ma ke ing s a egies and echnical decisions can be de i ed [32]. The analysis o a ea u e model is gene ally pe o med in wo s eps: i) Fi s , he model is ansla ed in o a speci ic logic ep esen a ion such as a sa is iabili y p oblem [7, 8, 9, 10, 11, 12, 13], a cons ain sa is ac ion p oblem [14, 15, 16] o a knowledge base using desc ip ion logic [17, 18], ii) Then, o - he-shel sol e s a e used o au oma ically pe o m a a ie y o ope a ions on he logic ep esen a ion o he model. Ca alogues wi h up o 30 di e en analysis ope a ions on ea u e models ha e been epo ed in he li e a u e [5, 6]. Following, we summa ize some o he analysis ope a ions we will e e h ough he es o he pape . De e mining i a ea u e model is oid. This ope a ion akes a ea u e model as inpu and e u ns a alue in o ming whe he such ea u e model is oid o no [6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 33]. A ea u e model is oid i i ep esen s no p oduc s. Finding ou i a p oduc is alid. This ope a ion checks whe he an inpu p oduc (i.e. se o ea u es) belongs o he se o p oduc s ep esen ed by a gi en ea u e model o no [6, 7, 8, 9, 10, 14, 16, 18, 33]. As an example, P={E-Shop, Ca alogue, In o, Desc ip ion, Secu- i y, Medium, GUI, PC}is no a alid p oduc o he p oduc line ep esen ed by he model in Figu e 1 because i does no include he manda o y ea u e Paymen . Ob aining all p oduc s. This ope a ion akes a ea u e model as inpu and e u ns all he p od- uc s ep esen ed by he model [7, 9, 10, 14, 19, 20, 33]. Calcula ing he numbe o p oduc s. This ope a ion e u ns he numbe o p oduc s ep e- sen ed by a ea u e model [8, 10, 14, 19, 20, 21, 33]. The model in Figu e 1 ep esen s 2016 di e en p oduc s. Calcula ing a iabili y. This ope a ion akes a ea u e model as inpu and e u ns he a io be- ween he numbe o p oduc s and 2n−1 whe e n is he numbe o ea u es in he model [14, 33]. This ope a ion may be used o measu e he lexibili y o he p oduc line. Fo ins ance, a small ac o means ha he numbe o combina ions o ea u es is e y limi ed compa ed o he o al numbe o po en ial p oduc s. In Figu e 1, Va iabili y = 0.00048. Calcula ing commonali y. This ope a ion akes a ea u e model and a ea u e as inpu s and e- u ns a alue ep esen ing he p opo ion o alid p oduc s in which he ea u e appea s [14, 21, 33]. This ope a ion may be used o p io i ize he o de in which he ea u es a e o be de eloped and can also be used o de ec dead ea u es [15]. In Figu e 1, Commonali y(Banne s) = 25%. Dead ea u es de ec ion. This ope a ion akes a ea u e model as inpu and e u ns a se o dead ea u es (i any) [2, 8, 11, 15, 33, 20, 13]. A dead ea u e is a ea u e ha ne e appea s in any o he p oduc s ep esen ed by he ea u e model [15]. These a e caused by c oss- ee cons ain s. Some common cases o dead ea u e a e p esen ed in Figu e 4 (Sec ion 3.2.1). Some comme cial and open sou ce ools suppo ing he analysis o ea u e models a e he AHEAD Tool Sui e [22], FaMa F amewo k [23], Fea u e Model Plug-in [24] and pu e:: a ian s [25]. 4 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 2.3 Black-box es ing echniques So wa e es ing is pe o med by means o es cases. A es case is a se o es inpu s and ex- pec ed ou pu s de eloped o e i y compliance wi h a speci ic equi emen [29, 27]. Tes cases may be g ouped in o so-called es sui es. A a ie y o es ing echniques has been epo ed o assis on he design o e ec i e es cases, i.e. hose ha will ind mo e aul s wi h less e o and ime [26, 27, 29]. In his pape , we will ocus on he so called black-box es ing echniques. These echniques ocus on checking whe he a p og am does wha i is supposed o do based on i s spec- i ica ion [27]. No knowledge abou he in e nal s uc u e o implemen a ion o he so wa e unde es is assumed. We will e e o ou o hese echniques along he pape . B ie ly, hese a e: Equi alence pa i ioning. This echnique is used o educe he numbe o es cases o be de- eloped while s ill main aining a easonable es co e age (i.e. he deg ee o which he es cases e i ies he es equi emen s) [27, 29]. In his echnique, he inpu domain o he p og am is di- ided in o pa i ions (also called equi alence classes) in which he p og am is expec ed o p ocess he se o da a inpu in a simila (i.e. equi alen ) way. Acco ding o his es ing app oach, only one o a ew es cases o each pa i ion a e needed o e alua e he beha iou o he p og am o he co esponding pa i ion. Thus, selec ing a subse o es cases om each pa i ion is enough o es he p og am e ec i ely while keeping a manageable numbe o es cases. Bounda y alue analysis. This echnique is used o guide he es e when selec ing inpu s om equi alence classes [27, 29]. Acco ding o his echnique, p og amme s usually make mis akes wi h he inpu s alues loca ed on he bounda ies o he equi alence classes. This me hod guides he es e o selec hose inpu s loca ed on he “edges” o he equi alence pa i ions. Pai wise es ing (also called 2-wise es ing). This is a combina o ial so wa e es ing me hod ocusing on es ing all possible disc e e combina ions o wo inpu pa ame e s [29, 34]. Acco ding o he hypo hesis behind his echnique, mos common e o s in ol e one inpu pa ame e . The nex mos common ca ego y o e o s consis s o hose dependen on in e ac ions be ween pai s o pa ame e s and so on. Thus, he main goal o his echnique is o add ess he second mos common cases o e o s ( hose in ol ing wo pa ame e s) while keeping he numbe o es cases in a manageable le el. E o guessing. This is a so wa e es ing echnique based on he abili y o he es e o p edic whe e aul s a e loca ed acco ding o i s expe ience on he domain [29]. Using his echnique, es cases a e speci ically designed o exe cise ypical e o -p one poin s ela ed o he ype o sys em unde es . 3 Tes sui e design In his sec ion, we desc ibe how we designed he es cases ha compose ou sui e. These mainly a e inpu -ou pu combina ions speci ically c ea ed o e eal ailu es in he implemen a ions o analysis ope a ions on ea u e models. C ea ing es cases o e e y possible pe mu a ion o a p og am is imp ac ical and e y o en impossible; he e a e simply oo many inpu -ou pu combina ions [27]. Thus, as ecalled by P ess- man [28], he objec i e when es ing is o “design es s ha ha e he highes likelihood o inding mos e o s wi h a minimum amoun o ime and e o ”. To assis us in he p ocess, we e alua ed a numbe o echniques epo ed in he li e a u e [27, 29]. We ocused on black-box echniques since we wan ou es cases o ely on he speci ica ion o he analysis ope a ions a he han on speci ic implemen a ions. In pa icula , we ound he ou echniques desc ibed in Sec ion 2.3 o be e ec i e and gene ic enough o be applicable o ou domain. 5 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 Fo he design o he sui e we ollowed ou s eps, namely: i) iden i ica ion o he inpu s and ou pu s o he analysis ope a ions, ii) selec ion o ep esen a i e ins ances o each ype o inpu , iii) combina ion o p e ious ins ances in hose ope a ions ecei ing mo e han one inpu pa ame e , and i ) es cases epo . Following, we de ail how we ca ied ou hese s eps. 3.1 Iden i ica ion o inpu s and ou pu s The cu en e sion o ou sui e add esses 7 ou o 30 analysis ope a ions on ea u e models iden i- ied in he li e a u e [5] (de ailed in Sec ion 2.2). We selec ed hese ope a ions o i s ex ended use in he communi y o au oma ed analysis and hei he e ogeneous na u e. In o de o iden i y he ype o he inpu /ou pu pa ame e s o he ope a ions and a oid misunde s andings when in e - p e ing hei seman ics, we used he o mal de ini ion o he ope a ions p oposed by Bena ides [32]. Table 1 summa izes he ope a ions in e ms o hei inpu s and ou pu s. Fo he sake o simplici y, we assign an iden i ica o o each ope a ion o e e hem along he pape . As illus a ed, inpu s a e composed o ea u e models, p oduc s and ea u es. Ou pu s mainly comp ise collec ions o p oduc s and ea u es oge he wi h nume ic and boolean alues. ID ope a ion Ope a ion Inpu s Ou pu VoidFM Void FM FM Boolean alue ValidP oduc Valid p oduc FM, P oduc Boolean alue P oduc s P oduc s FM Collec ion o p oduc s #P oduc s Numbe o p oduc s FM Nume ic alue Va iabili y Va iabili y FM Nume ic alue Commonali y Commonali y FM, Fea u e Nume ic alue DeadFea u es Dead ea u es FM Collec ion o ea u es Table 1: Analysis ope a ions add essed in he sui e The ea u e modelling no a ion used in ou sui e co esponds o one showed in Sec ion 2.1 (he eina e e e ed as basic ea u e models). We selec ed his no a ion o i s simplici y and ex ended use in cu en ea u e model analysis ools and li e a u e. 3.2 Inpu s selec ion In his sec ion, we explain how we selec ed he inpu s o be used in ou es cases. Fo each ype o inpu (i.e. ea u e models, p oduc s and ea u es), we nex desc ibe he echniques used and how we applied hem. 3.2.1 Fea u e models We ound wo es ing echniques o be help ul o he selec ion o a sui able se o inpu ea u e models, namely: equi alence pa i ioning and e o guessing. We applied hem as ollows: Equi alence pa i ioning. The po en ial numbe o inpu ea u e models is limi less. To selec a ep esen a i e se o hese, we p opose di iding inpu ea u e models in o equi alence classes acco ding o he di e en ypes o ela ionships and cons ain s among ea u es, i.e. manda o y, op ional, o , al e na i e, equi es and excludes. This is a na u al pa i ion in ui i ely used in mos p oposals when de ining he mapping om a ea u e model o a speci ic logic pa adigm (e.g. cons ain sa is ac ion p oblem) [7, 9, 10, 13, 14, 15, 18, 32]. The e o e, acco ding o his echnique, i a ea u e model wi h a single manda o y ela ionship is co ec ly managed by an ope a ion, we could assume ha hose wi h mo e han one manda o y ela ionship would also be p ocessed success ully. 6 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 Reasone Mu an s Equi alen Disca ded Sa 4jReasone 262 27 47 Ja aBDDReasone 302 28 37 JaCoPReasone 185 46 3 To al 749 101 87 Table 4: Mu an s gene a ion esul s VoidFM and ValidP oduc p oduced he lowes sco es and highe numbe o ali e mu an s. We ound ha mu an s on hese ope a ions equi ed inpu models o ha e a e y speci ic pa e n in o de o be killed and he e o e we e ha de o de ec han mu an s in he es o ope a ions. A e age mu a ion sco es in he h ee easone s anged be ween 94.4% and 100%. In o al, ou es sui e was able o kill 1390 (96.1%) ou o he 1445 mu an s execu ed showing he e ec i eness o he sui e. Ope a ion Sa 4jReasone Ja aBDDReasone JaCoPReasone Mu an s Ali e Sco e Mu an s Ali e Sco e Mu an s Ali e Sco e VoidFM 55 20 63.6 75 12 84.0 8 0 100 ValidP oduc 109 4 96.3 129 7 94.6 61 0 100 P oduc s 86 1 98.8 130 2 98.5 37 0 100 #P oduc s 57 1 98.2 77 2 97.4 13 0 100 Va iabili y 82 1 98.8 104 2 98.1 36 0 100 Commonali y 109 1 99.1 131 2 98.5 66 0 100 DeadFea u es - - - - - - 80 0 100 To al 498 28 94.4 646 27 95.8 301 0 100 Table 5: Mu an s execu ion esul s 4.1.3 Re inemen As shown in p e ious sec ions, mu a ion es ing is an e ec i e means o measu e he e ec i eness o a es sui e. Howe e , in o ma ion p o ided by mu a ion es ing can also be used o guide he c ea ion o new es cases ha kill he emaining ali e mu an s and s eng hen he inal es sui e [41]. Following his app oach, we designed a numbe o es cases o kill emaining unde ec ed mu an s un il ob aining a sco e o 100% in he h ee FaMa easone s. A o al o 27 new es cases we e c ea ed and execu ed. Fo ins ance, ali e mu an s guided us o he c ea ion o a couple es cases o ensu e ha al e na i e ela ionships a e no p ocessed as o - ela ionships and ice– e sa (e.g. es cases VM-21 and VM-22 in Appendix A). Ou o he 27 es cases c ea ed, we selec ed hose es cases ha showed o be e ec i e in killing mu an s in a leas wo o he h ee subjec easone s and added hem o ou sui e. As a esul , 10 es cases we e added o he ini ial es sui e inc easing he numbe o hese un il 190. 4.2 E alua ion using eal aul s Fo a u he e alua ion o ou app oach, we checked he e ec i eness o ou ool in de ec ing eal aul s. In pa icula , we i s s udied a mo i a ing aul ound in he li e a u e. Then, we used ou es sui e o es he elease 1.0 alpha o he FaMa amewo k, de ec ing one de ec . These esul s a e nex epo ed. 13 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 4.2.1 Mo i a ing aul ound in he li e a u e Conside he wo k o Ba o y in SPLC’05 [7], one o he seminal pape s in he communi y o au oma ed analysis o ea u e models. The pape included a bug (la e ixed3) in he mapping o a ea u e model o a p oposi ional o mula. We implemen ed his w ong mapping in o a mock easone o FaMa and checked he e ec i eness o ou app oach in de ec ing he aul . Figu e 7 illus a es an example o he w ong ou pu caused by he aul . This mani es s i sel in al e na i e ela ionships whose pa en ea u e is no manda o y making easone s o conside as alid p oduc hose including mul iple al e na i e sub ea u es and excluding he pa en ea u e (P3). As a esul , he se o p oduc s e u ned by he ool is e oneously la ge han he ac ual one. Fo ins ance, he numbe o p oduc s e u ned by ou aul y ool when using he model in Figu e 1 as inpu is 3,584 (ins ead o he ac ual 2,016). No e ha his is a mo i a ing aul since i can easily emain unde ec ed e en when using an inpu wi h he p oblema ic pa e n. Hence, in he p e ious example (ei he wi h “secu i y” ea u e as manda o y o op ional), he mock ool co ec ly iden i ies he model as non oid (i.e. i ep esen s a leas one p oduc ), and so he aul emains la en . Secu i y Medium High P1={Secu i y,High} P2={Secu i y,Medium} P3={High,Medium} Figu e 7: W ong se o p oduc s ob ained wi h he aul y easone We implemen ed ou es cases using JUni and es ed ou aul y ool. The aul was de ec ed by ou es sui e in he ope a ions VoidFM,P oduc ,#P oduc s,Va iabili y and Commonali y emaining la en in he ope a ions ValidP oduc and DeadFea u es. These wo ope a ions equi ed a e y speci ic pa e n o e eal he aul no included in he inpu s o ou es sui e. This gi es an idea o he complexi y o es ing in his domain. We ound his aul su icien ly mo i a ing o ex end ou sui e wi h es cases ha de ec i in all he ope a ions. Thus, we designed wo new es cases o de ec he aul in he ope a ion ValidP oduc and DeadFea u es and added hem o ou es sui e esul ing in a o al o 192 es cases. 4.2.2 FaMa 1.0 alpha Finally, we e alua ed ou ool by ying o de ec aul s in a ecen elease o he FaMa F amewo k, FaMa 1.0 alpha. We execu ed he 192 es cases o ou e ined es sui e. Tes s e ealed one de ec . The aul a ec ed he ope a ions ValidP oduc and Commonali y in Sa 4jReasone . The sou ce o he p oblem was a bug in he c ea ion o p oposi ional clauses in he so-called s aged con igu a ions, a new ea u e o he ool. 5 Tes sui e summa y and discussion Table 6 summa izes he gene al aspec s o he e ined es sui e using he common e ms o he IEEE S anda d o So wa e Tes ing Documen a ion [35]. Fo each ope a ion, he numbe o es cases and he es ing echniques used a e p esen ed. Global inpu s cons ain s speci y cons ain s ha mus be ue o e e y inpu in he se o associa ed es cases. Fo he sake o simplici y, 3 p:// p.cs.u exas.edu/pub/p eda o /splc05.pd 14 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 wo main inpu cons ain s we e imposed, namely: i) inpu ea u e models mus be syn ac ically co ec (e.g. checking models o con o mance o a me amodel), and ii) all inpu pa ame e s o he analysis ope a ions mus be p o ided. An ope a ion is said o pass he es (so-called pass c i e ia) when all he es cases associa ed o ha ope a ion a e success ul. T ying o be exhaus i e when es ing ea u e model analyses ools can easily inc ease he numbe o es cases o an unmanageable le el. To keep a easonable balance be ween numbe o es cases and es co e age, we kep in mind a numbe o gene ic ecommenda ions om he es ing li e a u e, namely: i) we ga e p io i y o hose decisions educing he numbe o es cases, ii) we a oided edundancies by designing each es case o e eal a single ype o aul , and iii) we designed simple es cases whose ou pu could be wo ked ou manually o a oid es hemsel es o become e o -p one. We ema k ha he po en ial use s o he sui e a e e e y ool suppo ing he analysis o ea u e model. This can be au oma ed by simply implemen ing he es cases in he desi ed pla o m and execu ing hem. A comple e lis o he es cases ha compose he sui e is epo ed in Appendix A. To acili a e i s implemen a ion, inpu models used in he es cases a e also a ailable in XML o ma in he FaMa Tool Sui e Web si e4. Tes sui e iden i ie : FaMa Tes Sui e ( 1.2) Ope a ions es ed Tes cases Techniques used Void FM 24 EP, PT Valid p oduc 63 EP, PT, BVA P oduc s 21 EP, PT Numbe o p oduc s 21 EP, PT Va iabili y 21 EP, PT Commonali y 33 EP, PT, BVA Dead ea u es 9 EG To al 192 Global inpu cons ain s: - Inpu FMs mus be syn ac ically co ec - All inpu pa ame e s a e equi ed Ope a ion pass c i e ia: - Pass 100% o associa ed es cases Table 6: Gene al o e iew o he FaMa Tes Sui e (EP: Equi alence Pa i ioning, PT: Pai wise Tes ing, BVA: Bounda y-Value Analysis, EG: E o Guessing) 6 Conclusions and u u e wo k In his pape , we p esen a se o implemen a ion–independen es cases o alida e he unc ion- ali y o ools suppo ing he analysis o ea u e models. Th ough he implemen a ion o ou es cases, aul s can be apidly de ec ed assis ing in he de elopmen o ea u e model analysis ools and imp o ing hei eliabili y and quali y. These can be used ei he in isola ion o as a sui able complemen o u he es ing me hods such as whi e–box es ing echniques o au oma ed es da a gene a o s. Fo i s design, we used popula echniques om he so wa e es ing communi y o assis us on he c ea ion o a ep esen a i e se o inpu –ou pu combina ions. To e alua e i s e ec i eness, we applied mu a ion es ing on h ee open sou ce ea u e model analysis ools in e- g a ed in o he FaMa amewo k. These ools use di e en unde lying pa adigms and we e coded by di e en de elope s wha p o ides he necessa y he e ogenei y o he e alua ion. We ini ially ob ained an a e age mu a ion sco e o 96.1% and e ined ou sui e p og essi ely un il ge ing 100%. 4h p://www.isa.us.es/ ama/?FaMa Tes Sui e 15 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 Once e ined, ou sui e also showed o be e ec i e in de ec ing eal aul s ound in he li e a u e and in a ecen elease o FaMa. Bo h, he es sui e documen a ion and he inpu s models used in he es cases a e eady– o–use and a ailable a he Web Si e o he FaMa Tool Sui e. We in end his pape o be a i s e o owa d he de elopmen o a widely accep ed es sui e o suppo unc ional es ing in he communi y o au oma ed analysis o ea u e models. Se e al challenges emain o ou u u e wo k in wo main di ec ions, namely: –We in end o ex end ou sui e wi h new ope a ions and es ing echniques. We also plan o e alua e ou sui e wi h o he ools and in de elopmen scena ios using FaMa and epo ou expe iences o he communi y. –We also plan o explo e he bene i s ob ained when combining ou sui e wi h o he au oma ed es ing me hods. In a p e ious wo k [30], we p esen ed an au oma ed es da a gene a o o he analysis o ea u e models wi h p omising esul s. I gene a es andom ollow–up es cases based on he ela ions be ween inpu s ea u e models and hei s expec ed ou pu s (so-called me amo phic es ing). Howe e , i is known ha me amo phic es ing p oduce be e esul s when combined wi h o he es case selec ion s a egies ha gene a e he ini ial se o es cases. We in end o use ou sui e o guide he gene a ion o es cases in ou au oma ed es da a gene a o and s udy he gains in e iciency and e icacy. Ma e ial The ools, mu an s, and es cases used in ou e alua ion a e a ailable a h p://www.lsi.us. es/~segu a/ iles/ma e ial/ s_1_2/ Acknowledgmen s We would like o hank he e iewe s o he pape as well as Don Ba o y and Robe M. Hie ons whose commen s and sugges ions helped us o imp o e he pape subs an ially. We also hank Jose Galindo o his echnical suppo du ing he e alua ion o ou app oach. Re e ences 1. Clemen s P, No h op L. So wa e P oduc Lines: P ac ices and Pa e ns. SEI Se ies in So wa e Enginee ing. Addison–Wesley; 2001. 2. Ba o y D, Bena ides D, Ruiz-Co ´es A. Au oma ed Analysis o Fea u e Models: Challenges Ahead. Communica ions o he ACM. 2006;Decembe :45–47. 3. Cza necki K, Eisenecke UW. Gene a i e P og amming: Me hods, Techniques, and Applica ions. Addison–Wesley; may 2000. ISBN 0–201–30977–7. 4. Kang K, Cohen S, Hess J, No ak W, Pe e son S. Fea u e–O ien ed Domain Analysis (FODA) Feasi- bili y S udy. SEI; 1990. CMU/SEI-90-TR-21. 5. Bena ides D, Segu a S, Ruiz-Co ´es A. Au oma ed Analysis o Fea u e Models 20 Yea s La e : A Li e a u e Re iew. In o ma ion Sys ems. 2010;In p ess. 6. Schobbens P, P Heymans JCT, Bon emps Y. Gene ic seman ics o ea u e diag ams. Compu e Ne wo ks. 2007 Feb;51(2):456–479. 7. Ba o y D. Fea u e Models, G amma s, and P oposi ional Fo mulas. In: So wa e P oduc Lines Con e ence, LNCS 3714; 2005. p. 7–20. 8. Cza necki K, Kim P. Ca dinali y-Based Fea u e Modeling and Cons ain s: A P og ess Repo . In: P oceedings o he In e na ional Wo kshop on So wa e Fac o ies A OOPSLA 2005; 2005. . 9. Gheyi R, Massoni T, Bo ba P. A Theo y o Fea u e Models in Alloy. In: P oceedings o he ACM SIGSOFY Fi s Alloy Wo kshop. Po land, Uni ed S a es; 2006. p. 71–80. A ailable om: h p: //alloy.mi .edu/wo kshop/p og amme.h ml. 10. Mannion M, Cama a J. Theo em P o ing o P oduc Line Model Ve i ica ion. In: So wa e P oduc - Family Enginee ing (PFE). ol. 3014 o Lec u e No es in Compu e Science. Sp inge Be lin / Heidel- be g; 2003. p. 211–224. A ailable om: h p://www.sp inge link.com/con en /m0k40djlmmxx8 p/. 11. Mendon¸ca M, Wasowski A, Cza necki K. SAT–based analysis o ea u e models is easy. In: P oceedings o he So wa e P oduc Line Con e ence; 2009. . 16 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 12. an de S o m T. Gene ic Fea u e-Based So wa e Composi ion. In: So wa e Composi ion. ol. 4829 o LNCS. Sp inge ; 2007. p. 66–80. 13. Zhang W, Mei H, Zhao H. Fea u e-d i en equi emen dependency analysis and high-le el so - wa e design. Requi emen s Enginee ing. 2006 June;11(3):205–220. A ailable om: h p://www. sp inge link.com/con en / 1648q73m788q71x/. 14. Bena ides D, Ruiz-Co ´es A, T inidad P. Au oma ed Reasoning on Fea u e Models. LNCS, Ad anced In o ma ion Sys ems Enginee ing: 17 h In e na ional Con e ence, CAiSE 2005. 2005;3520:491–503. 15. T inidad P, Bena ides D, Du ´an A, Ruiz-Co ´es A, To o M. Au oma ed E o Analysis o he Ag- iliza ion o Fea u e Modeling. Jou nal o Sys ems and So wa e. 2008;81(6):883–896. 16. Whi e J, Schmid D, T inidad DBP, Ruiz-Co ´es. Au oma ed Diagnosis o P oduc -line Con igu a ion E o s in Fea u e Models. In: P oceedings o he 12 h So wa e P oduc Line Con e ence (SPLC’08). Lime ick, I eland; 2008. . 17. Fan S, Zhang N. Fea u e Model Based on Desc ip ion Logics. In: Knowledge-Based In elligen In o - ma ion and Enginee ing Sys ems; 2006. A ailable om: h p://dx.doi.o g/10.1007/11893004_145. 18. Wang H, Li YF, un J, Zhang H, Pan J. Ve i ying Fea u e Models using OWL. Jou nal o Web Seman ics. 2007 June;5:117–129. A ailable om: h p://dx.doi.o g/10.1016/j.websem.2006.11. 006. 19. an Deu sen A, Klin P. Domain–Speci ic Language Design Requi es Fea u e Desc ip ions. Jou nal o Compu ing and In o ma ion Technology. 2002;10(1):1–17. 20. an den B oek P, Gal ao I. Analysis o Fea u e Models using Gene alised Fea u e T ees. In: Thi d In e na ional Wo kshop on Va iabili y Modelling o So wa e-in ensi e Sys ems. No. 29 in ICB- Resea ch Repo . Essen, Ge many: Uni e si ¨a Duisbu g-Essen; 2009. p. 29–35. A ailable om: h p://www. amos-wo kshop.ne /p oceedings/VaMoS_2009_P oceedings.pd . 21. Fe nandez-Amo os D, He adio R, Ce ada J. In e ing In o ma ion om Fea u e Diag ams o P oduc Line Economic Models. In: P oceedings o he So wa e P oduc Line Con e ence; 2009. . 22. AHEAD Tool Sui e. h p://www.cs.u exas.edu/use s/schwa z/ATS.h ml;. Accessed No embe 2009. 23. Bena ides D, T inidad P, Segu a S, Ruiz-Co ´es A. FaMa F amewo k. h p://www.isa.us.es/ ama/;. 24. Fea u e Modeling Plug-in. h p://gp.uwa e loo.ca/ mp/;. Accessed No embe 2009. 25. pu e:: a ian s. h p://www.pu e-sys ems.com/;. Accessed No embe 2009. 26. Beize B. So wa e es ing echniques (2nd ed.). New Yo k, NY, USA: Van Nos and Reinhold Co.; 1990. 27. Mye s GJ, Sandle C. The A o So wa e Tes ing. John Wiley & Sons; 2004. 28. P essman RS. So wa e Enginee ing: A P ac i ione ’s App oach. 5 h ed. McG ap-Hill; 2001. 29. Copeland L. A P ac i ione ’s Guide o So wa e Tes Design. No wood, MA, USA: A ech House, Inc.; 2003. 30. Segu a S, Hie ons RM, Bena ides D, Ruiz-Co ´es A. Au oma ed Tes Da a Gene a ion on he Anal- yses o Fea u e Models: A Me amo phic Tes ing App oach. In: In e na ional Con e ence on So wa e Tes ing, Ve i ica ion and Valida ion. Pa is, F ance: IEEE p ess; 2010. In p ess. 31. Segu a S, Bena ides D, Ruiz-Co ´es A. Func ional Tes ing o Fea u e Model Analysis Tools. A Fi s S ep. In: 5 h So wa e P oduc Lines Tes ing Wo kshop (SPLiT 2008). SPLC’08. Lime ick, I eland; 2008. . 32. Bena ides D. On he Au oma ed Analyisis o So wa e P oduc Lines using Fea u e Models. A F ame- wo k o De eloping Au oma ed Tool Suppo . Uni e si y o Se ille; 2007. 33. T inidad P, Ruiz-Co ´es A. Abduc i e Reasoning and Au oma ed Analysis o Fea u e Models: How a e hey connec ed? In: Thi d In e na ional Wo kshop on Va iabili y Modelling o So wa e- In ensi e Sys ems. P oceedings; 2009. p. 145–153. A ailable om: h p://www. amos-wo kshop.ne / p oceedings/VaMoS_2009_P oceedings.pd . 34. G indal M, O u J, Andle SF. Combina ion es ing s a egies: a su ey. So wa e Tes ing, Ve i ica- ion and Reliabili y. 2005;15(3):167–199. A ailable om: h p://dx.doi.o g/10.1002/s .319. 35. D a IEEE S anda d o so wa e and sys em es documen a ion (Re ision o IEEE 829-1998); 2007. A ailable om: h p://ieeexplo e.ieee.o g/xpls/abs _all.jsp?a numbe =4432350. 36. DeMillo RA, Lip on RJ, Saywa d FG. Hin s on Tes Da a Selec ion: Help o he P ac icing P og am- me . IEEE Compu e . 1978;11(4):34–41. 37. Ce ina C, Fons J, Pelechano V. Moski Fea u e Modele . h p://www.p os.up .es/m m;. Accessed No embe 2009. 38. Sa 4j. h p://www.sa 4j.o g/;. Accessed No embe 2009. 39. Ja aBDD. h p://ja abdd.sou ce o ge.ne /;. Accessed No embe 2009. 40. JaCoP. h p://jacop.osolp o.com/;. Accessed No embe 2009. 41. Smi h BH, Williams L. On guiding he augmen a ion o an au oma ed es sui e ia mu a ion analysis. Empi ical So wa e Enginee ing. 2009;14(3):341–369. 42. Ma YS, O u J. Desc ip ion o Me hod-le el Mu a ion Ope a o s o Ja a, h p://cs.gmu.edu/ ~o u /muja a/mu opsMe hod.pd ; 2005. Accessed 2/10/2009. 43. JUni . h p://www.juni .o g/;. Accessed No embe 2009. 17 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 A Tes cases In his appendix, we p esen he es cases included in he cu en e sion o he FAMA Tes Sui e. Fo each ope a ion, associa ed es cases a e de ailed. A.1 Ope a ion Void Fea u e Model ID Desc ip ion Inpu Exp. Ou pu Deps. VM-1 Check whe he manda o y ela ionships a e co ec ly managed by he ope a ion. A B Non oid VM-2 Check whe he op ional ela ionships a e co - ec ly managed by he ope a ion. A B Non oid VM-3 Check whe he o – ela ionships a e co ec ly managed by he ope a ion. A B C Non oid VM-4 Check whe he al e na i e ela ionships a e co ec ly managed by he ope a ion. A B C Non oid VM-5 Check whe he ‘ equi es’ cons ain s a e co - ec ly managed by he ope a ion. A B C Non oid VM-2 VM-6 Check whe he ‘excludes’ cons ain s a e co - ec ly managed by he ope a ion. A B C Non oid VM-2 18 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VM-7 Check whe he he in e ac ion be ween manda o y and op ional ela ionships is co - ec ly p ocessed. A B D C E Non oid VM-1 VM-2 VM-8 Check whe he he in e ac ion be ween manda o y and o - ela ionships is co ec ly p ocessed. A B E F C D G Non oid VM-1 VM-3 VM-9 Check whe he he in e ac ion be ween manda o y and al e na i e ela ionships is co ec ly p ocessed. A B E F C D G Non oid VM-1 VM-4 VM-10 Check whe he he in e ac ion be ween manda o y ela ionships and ‘ equi es’ con- s ain s is co ec ly p ocessed. A BC Non oid VM-1 VM-5 VM-11 Check whe he he in e ac ion be ween manda o y ela ionships and ‘excludes’ con- s ain s is co ec ly p ocessed. A BC Void VM-1 VM-6 VM-12 Check whe he he in e ac ion be ween op- ional and o - ela ionships is co ec ly p o- cessed. A B E F C D G Non oid VM-2 VM-3 VM-13 Check whe he he in e ac ion be ween op- ional and al e na i e ela ionships is co ec ly p ocessed. A B E F C D G Non oid VM-2 VM-4 19 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VM-14 Check whe he he in e ac ion be ween o - and al e na i e ela ionships is co ec ly p o- cessed. A D E H I B C F G Non oid VM-3 VM-4 VM-15 Check whe he he in e ac ion be ween o - ela ionships and ‘ equi es’ cons ain s is co - ec ly p ocessed. A B C Non oid VM-3 VM-5 VM-16 Check whe he he in e ac ion be ween o - ela ionships and ‘excludes’ cons ain s is co - ec ly p ocessed. A B C Non oid VM-3 VM-6 VM-17 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘ equi es’ cons ain s is co ec ly p ocessed. A B C Non oid VM-4 VM-5 VM-18 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘excludes’ cons ain s is co ec ly p ocessed. A B C Non oid VM-4 VM-6 VM-19 Check whe he he in e ac ion be ween ‘ e- qui es’ and ‘excludes’ cons ain s is co ec ly p ocessed. A BC Non oid VM-2 VM-5 VM-6 VM-20 Check whe he he in e ac ion among h ee o mo e di e en ypes o ela ionships and con- s ain s is co ec ly p ocessed. A B D E C F G Non oid VM-1, ..., VM-19 20 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VM-21 Check whe he al e na i e ela ionships a e e oneously implemen ed as o - ela ionships B C D A E Void VM-1 VM-4 VM-9 VM-10 VM-17 VM-22 Check whe he o - ela ionships a e e o- neously implemen ed as al e na i e ela ion- ships B C D A E Non oid VM-1 VM-3 VM-8 VM-10 VM-15 VM-23 Check whe he al e na i e sub ea u es can be pa o a p oduc wi hou including i s pa en ea u e B C D A E Void VM-1 VM-2 VM-4 VM-5 VM-6 VM-7 VM-9 VM-10 VM-11 VM-13 VM-17 VM-18 VM-24 Check whe he a p oduc can e oneously in- clude mo e han one al e na i e ea u es B D G A H C F E Void VM-1 VM-4 VM-9 VM-10 VM-17 Table 7: Ope a ion oidFM. Tes cases A.2 Ope a ion Valid P oduc 21 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-1 Check whe he alid p oduc s a e co ec ly iden i ied in ea u e models wi h manda o y ela ionships. A B P={A,B} Valid VP-2 Check whe he non- alid p oduc s a e co - ec ly iden i ied in ea u e models wi h manda o y ela ionships. A B P={A} Non- alid VP-3 Check whe he alid p oduc s (wi h a mini- mum se o ea u es) a e co ec ly iden i ied in ea u e models wi h op ional ela ionships. A B P={A} Valid VP-4 Check whe he alid p oduc s (wi h a maxi- mum se o ea u es) a e co ec ly iden i ied in ea u e models wi h op ional ela ionships. A B P={A,B} Valid VP-5 Check whe he alid p oduc s (wi h a mini- mum se o ea u es) a e co ec ly iden i ied in ea u e models wi h o - ela ionships. A B C P={A,B} Valid VP-6 Check whe he alid p oduc s (wi h a maxi- mum se o ea u es) a e co ec ly iden i ied in ea u e models wi h o - ela ionships. A B C P={A,B,C} Valid 22 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-40 Check whe he alid p oduc s (wi h a mini- mum se o ea u es) a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘ equi es’ cons ain s. A B C P={A,C} Valid VP-5 VP-6 VP-7 VP-11 VP-12 VP-41 Check whe he alid p oduc s (wi h a maxi- mum se o ea u es) a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘ equi es’ cons ain s. A B C P={A,B,C} Valid VP-5 VP-6 VP-7 VP-11 VP-12 VP-42 Check whe he non- alid p oduc s (wi h a low numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘ equi es’ cons ain s. A B C P={A} Non- alid VP-5 VP-6 VP-7 VP-11 VP-12 VP-43 Check whe he alid p oduc s a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘excludes’ cons ain s. A B C P={A,B} Valid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 VP-44 Check whe he non- alid p oduc s (wi h a low numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘excludes’ cons ain s. A B C P={A} Non- alid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 29 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-45 Check whe he non- alid p oduc s (wi h a high numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining o - ela ionships and ‘excludes’ cons ain s. A B C P={A,B,C} Non- alid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 VP-46 Check whe he alid p oduc s a e co ec ly iden i ied in ea u e models con aining al e - na i e ela ionships and ‘ equi es’ cons ain s. A B C P={A,C} Valid VP-8 VP-9 VP-10 VP-11 VP-12 VP-47 Check whe he non- alid p oduc s (wi h a low numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining al e na i e ela ion- ships and ‘ equi es’ cons ain s. A B C P={A} Non- alid VP-8 VP-9 VP-10 VP-11 VP-12 VP-48 Check whe he non- alid p oduc s (wi h a high numbe o ea u es) a e co ec ly iden- i ied in ea u e models con aining al e na i e ela ionships and ‘ equi es’ cons ain s. A B C P={A,B,C} Non- alid VP-8 VP-9 VP-10 VP-11 VP-12 VP-49 Check whe he alid p oduc s a e co ec ly iden i ied in ea u e models con aining al- e na i e ela ionships and ‘excludes’ con- s ain s. A B C P={A,B} Valid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 30 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-50 Check whe he non- alid p oduc s (wi h a low numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining al e na i e ela ion- ships and ‘excludes’ cons ain s. A B C P={A} Non- alid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 VP-51 Check whe he non- alid p oduc s (wi h a high numbe o ea u es) a e co ec ly iden- i ied in ea u e models con aining al e na i e ela ionships and ‘excludes’ cons ain s. A B C P={A,B,C} Non- alid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 VP-52 Check whe he alid p oduc s (wi h a mini- mum se o ea u es) a e co ec ly iden i ied in ea u e models con aining ‘ equi es’ and ‘ex- cludes’ cons ain s. A BC P={A} Valid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 VP-53 Check whe he alid p oduc s (wi h a maxi- mum se o ea u es) a e co ec ly iden i ied in ea u e models con aining ‘ equi es’ and ‘ex- cludes’ cons ain s. A BC P={A,C} Valid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 VP-54 Check whe he non- alid p oduc s (wi h a high numbe o ea u es) a e co ec ly iden- i ied in ea u e models con aining ‘ equi es’ and ‘excludes’ cons ain s. A BC P={A,B,C} Non- alid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 31 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-55 Check whe he alid p oduc s (wi h a mini- mum se o ea u es) a e co ec ly iden i ied in ea u e models con aining h ee o mo e di - e en ypes o ela ionships and cons ain s. A B D E C F G P={A,B,D} Valid VP-1, ..., VP-54 VP-56 Check whe he alid p oduc s (wi h a max- imum se o ea u es) a e co ec ly iden i ied in ea u e models con aining h ee o mo e di - e en ypes o ela ionships and cons ain s. A B D E C F G P={A,B,C,E, F,G} Valid VP-1, ..., VP-54 VP-57 Check whe he non- alid p oduc s (wi h a low numbe o ea u es) a e co ec ly iden i ied in ea u e models con aining h ee o mo e di - e en ypes o ela ionships and cons ain s. A B D E C F G P={A,B} Non- alid VP-1, ..., VP-54 VP-58 Check whe he non- alid p oduc s (wi h a high numbe o ea u es) a e co ec ly iden- i ied in ea u e models con aining h ee o mo e di e en ypes o ela ionships and con- s ain s. A B D E C F G P={A,B,C,D, E,F,G} Non- alid VP-1, ..., VP-54 VP-59 Check whe he non- alid p oduc s excluding he oo ea u e a e co ec ly iden i ied. A B P={B} Non- alid VP-3 VP-4 32 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. VP-60 Check whe he non- alid p oduc s including non-exis en ea u es a e co ec ly p ocessed by he ope a ion. A B P={A,H} Non- alid VP-3 VP-4 VP-61 Check whe he child ea u es in an al e na- i e ela ionships can e oneously be pa o a p oduc wi hou hei pa en ea u e. B C D A E P={A,D,E} Non Valid VP-1 ... VP-4 VP-8 ... VP-18 VP-22 ... VP-28 VP-32 ... VP-35 VP-46 ... VP-51 VP-62 Check whe he a p oduc can e oneously in- clude mo e han one al e na i e ea u es B D G A H C F E P={A,B,E,G,H} Non Valid VP-1 VP-2 VP-8 VP-9 VP-10 VP-22 ... VP-27 VP-46 VP-47 VP-48 VP-63 Check whe he mul iple al e na i e ea u es can be e oneously pa o a p oduc when no including hei non-manda o y pa en ea u e A B C D P={A,C,D} Non Valid VP-3 VP-4 VP-8 VP-9 VP-10 VP-32 VP-33 VP-34 VP-35 Table 8: Ope a ion ValidP oduc . Tes cases A.3 Ope a ion All P oduc s 33 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. P-1 Check whe he manda o y ela ionships a e co ec ly managed by he ope a ion. A B {A,B} P-2 Check whe he op ional ela ionships a e co - ec ly managed by he ope a ion. A B {A},{A,B} P-3 Check whe he o – ela ionships a e co ec ly managed by he ope a ion. A B C {A,B},{A,C}, {A,B.C} P-4 Check whe he al e na i e ela ionships a e co ec ly managed by he ope a ion. A B C {A,B},{A,C} P-5 Check whe he ‘ equi es’ cons ain s a e co - ec ly managed by he ope a ion. A B C {A},{A,C}, {A,B,C} P-2 P-6 Check whe he ‘excludes’ cons ain s a e co - ec ly managed by he ope a ion. A B C {A},{A,B}, {A,C} P-2 P-7 Check whe he he in e ac ion be ween manda o y and op ional ela ionships is co - ec ly p ocessed. A B D C E {A,B},{A,B,D}, {A,B,C,E}, {A,B,C,D,E} P-1 P-2 34 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. P-8 Check whe he he in e ac ion be ween manda o y and o - ela ionships is co ec ly p ocessed. A B E F C D G {A,B,C,E}, {A,B,C,F}, {A,B,C,E,F}, {A,B,D,E,G}, {A,B,C,D,E,G}, {A,B,D,F,G}, {A,B,C,D,F,G}, {A,B,D,E,F,G}, {A,B,C,D,E,F,G} P-1 P-3 P-9 Check whe he he in e ac ion be ween manda o y and al e na i e ela ionships is co ec ly p ocessed. A B E F C D G {A,B,D,F}, {A,B,D,E}, {A,B,C,F,G}, {A,B,C,E,G} P-1 P-4 P-10 Check whe he he in e ac ion be ween manda o y ela ionships and ‘ equi es’ con- s ain s is co ec ly p ocessed. A BC {A,B,C} P-1 P-5 P-11 Check whe he he in e ac ion be ween manda o y ela ionships and ‘excludes’ con- s ain s is co ec ly p ocessed. A BC None P-1 P-6 35 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. P-12 Check whe he he in e ac ion be ween op- ional and o - ela ionships is co ec ly p o- cessed. A B E F C D G {A,D},{A,C}, {A,C,D}, {A,C,G}, {A,B,D,F}, {A,B,D,E}, {A,C,D,G}, {A,B,C,F}, {A,B,C,E}, {A,B,D,E,F}, {A,B,C,F,G}, {A,B,C,E,G}, {A,B,C,D,F}, {A,B,C,D,E}, {A,B,C,F,E}, {A,B,C,D,F,G}, {A,B,C,D,E,G}, {A,B,C,E,F,G}, {A,B,C,D,E,F}, {A,B,C,D,E,F,G} P-2 P-3 P-13 Check whe he he in e ac ion be ween op- ional and al e na i e ela ionships is co ec ly p ocessed. A B E F C D G {A,C},{A,D}, {A,D,G}, {A,B,C,E}, {A,B,C,F}, {A,B,D,E}, {A,B,D,F}, {A,B,D,E,G}, {A,B,D,F,G} P-2 P-4 36 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. P-14 Check whe he he in e ac ion be ween o - and al e na i e ela ionships is co ec ly p o- cessed. A D E H I B C F G {A,D,C}, {A,C,E,H}, {A,C,E,I}, {A,B,D,G}, {A,B,D,F}, {A,C,D,E,H}, {A,C,D,E,I}, {A,B,D,F,G}, {A,B,E,G,H}, {A,B,E,F,H}, {A,B,E,G,I}, {A,B,E,F,I}, {A,B,D,E,G,H}, {A,B,E,F,G,H}, {A,B,D,E,F,H}, {A,B,D,E,G,I}, {A,B,E,F,G,I}, {A,B,D,E,F,I}, {A,B,D,E,F,G,H}, {A,B,D,E,F,G,I} P-3 P-4 P-15 Check whe he he in e ac ion be ween o - ela ionships and ‘ equi es’ cons ain s is co - ec ly p ocessed. A B C {A,C},{A,B,C} P-3 P-5 P-16 Check whe he he in e ac ion be ween o - ela ionships and ‘excludes’ cons ain s is co - ec ly p ocessed. A B C {A,B},{A,C} P-3 P-6 P-17 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘ equi es’ cons ain s is co ec ly p ocessed. A B C {A,C} P-4 P-5 P-18 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘excludes’ cons ain s is co ec ly p ocessed. A B C {A,B},{A,C} P-4 P-6 37 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. P-19 Check whe he he in e ac ion be ween ‘ e- qui es’ and ‘excludes’ cons ain s is co ec ly p ocessed. A BC {A},{A,C} P-2 P-5 P-6 P-20 Check whe he he in e ac ion among h ee o mo e di e en ypes o ela ionships and con- s ain s is co ec ly p ocessed. A B D E C F G {A,B,D}, {A,B,C,D,F}, {A,B,C,E,F}, {A,B,C,E,F,G} P-1, ..., P-19 P-21 Check whe he a p oduc can e oneously in- clude mo e han one al e na i e ea u es. B D G A H C F E None P-1 P-4 P-9 P-10 P-17 Table 9: Ope a ion P oduc s. Tes cases A.4 Ope a ion Numbe o P oduc s ID Desc ip ion Inpu Exp. Ou pu Deps. NP-1 Check whe he manda o y ela ionships a e co ec ly managed by he ope a ion. A B 1 NP-2 Check whe he op ional ela ionships a e co - ec ly managed by he ope a ion. A B 2 NP-3 Check whe he o – ela ionships a e co ec ly managed by he ope a ion. A B C 3 38 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-2 Check whe he op ional ela ionships a e co - ec ly managed by he ope a ion. A B Fea u e=B 50% C-3 Check whe he o – ela ionships a e co ec ly managed by he ope a ion. A B C Fea u e=B 66% C-4 Check whe he al e na i e ela ionships a e co ec ly managed by he ope a ion. A B C Fea u e=B 50% C-5 Check whe he ‘ equi es’ cons ain s a e co - ec ly managed by he ope a ion. Inpu ea- u e has minimum commonali y. A B C Fea u e=B 33% C-2 C-6 Check whe he ‘ equi es’ cons ain s a e co - ec ly managed by he ope a ion. Inpu ea- u e has maximum commonali y. A B C Fea u e=C 66% C-2 C-7 Check whe he ‘excludes’ cons ain s a e co - ec ly managed by he ope a ion. A B C Fea u e=B 33% C-2 45 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-8 Check whe he he in e ac ion be ween manda o y and op ional ela ionships is co - ec ly p ocessed. Inpu ea u e has minimum commonali y. A B D C E Fea u e=E 50% C-1 C-2 C-9 Check whe he he in e ac ion be ween manda o y and op ional ela ionships is co - ec ly p ocessed. Inpu ea u e has maximum commonali y. A B D C E Fea u e=B 100% C-1 C-2 C-10 Check whe he he in e ac ion be ween manda o y and o - ela ionships is co ec ly p ocessed. Inpu ea u e has minimum com- monali y. A B E F C D G Fea u e=F 66% C-1 C-3 C-11 Check whe he he in e ac ion be ween manda o y and o - ela ionships is co ec ly p ocessed. Inpu ea u e has maximum com- monali y. A B E F C D G Fea u e=B 100% C-1 C-3 C-12 Check whe he he in e ac ion be ween manda o y and al e na i e ela ionships is co ec ly p ocessed. Inpu ea u e has mini- mum commonali y. A B E F C D G Fea u e=G 50% C-1 C-4 46 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-13 Check whe he he in e ac ion be ween manda o y and al e na i e ela ionships is co ec ly p ocessed. Inpu ea u e has maxi- mum commonali y. A B E F C D G Fea u e=B 100% C-1 C-4 C-14 Check whe he he in e ac ion be ween manda o y ela ionships and ‘ equi es’ con- s ain s is co ec ly p ocessed. A BC Fea u e=B 100% C-1 C-5 C-6 C-15 Check whe he he in e ac ion be ween manda o y ela ionships and ‘excludes’ con- s ain s is co ec ly p ocessed. A BC Fea u e=B 0% C-1 C-7 C-16 Check whe he he in e ac ion be ween op- ional and o - ela ionships is co ec ly p o- cessed. Inpu ea u e has minimum common- ali y. A B E F C D G Fea u e=G 40% C-2 C-3 C-17 Check whe he he in e ac ion be ween op- ional and o - ela ionships is co ec ly p o- cessed. Inpu ea u e has maximum common- ali y. A B E F C D G Fea u e=C 80% C-2 C-3 47 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-18 Check whe he he in e ac ion be ween op- ional and al e na i e ela ionships is co ec ly p ocessed. Inpu ea u e has minimum com- monali y. A B E F C D G Fea u e=E 33% C-2 C-4 C-19 Check whe he he in e ac ion be ween op- ional and al e na i e ela ionships is co ec ly p ocessed. Inpu ea u e has maximum com- monali y. A B E F C D G Fea u e=D 66% C-2 C-4 C-20 Check whe he he in e ac ion be ween o - and al e na i e ela ionships is co ec ly p o- cessed. Inpu ea u e has minimum common- ali y. A D E H I B C F G Fea u e=C 25% C-3 C-4 C-21 Check whe he he in e ac ion be ween o - and al e na i e ela ionships is co ec ly p o- cessed. Inpu ea u e has maximum common- ali y. A D E H I B C F G Fea u e=E 80% C-3 C-4 C-22 Check whe he he in e ac ion be ween o - ela ionships and ‘ equi es’ cons ain s is co - ec ly p ocessed. Inpu ea u e has minimum commonali y. A B C Fea u e=B 50% C-3 C-5 C-6 C-23 Check whe he he in e ac ion be ween o - ela ionships and ‘ equi es’ cons ain s is co - ec ly p ocessed. Inpu ea u e has maximum commonali y. A B C Fea u e=C 100% C-3 C-5 C-6 48 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-24 Check whe he he in e ac ion be ween o - ela ionships and ‘excludes’ cons ain s is co - ec ly p ocessed. A B C Fea u e=C 50% C-3 C-7 C-25 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘ equi es’ cons ain s is co ec ly p ocessed. Inpu ea u e has min- imum commonali y. A B C Fea u e=B 0% C-4 C-5 C-6 C-26 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘ equi es’ cons ain s is co ec ly p ocessed. Inpu ea u e has max- imum commonali y. A B C Fea u e=C 100% C-4 C-5 C-6 C-27 Check whe he he in e ac ion be ween al e - na i e ela ionships and ‘excludes’ cons ain s is co ec ly p ocessed. A B C Fea u e=B 50% C-4 C-7 C-28 Check whe he he in e ac ion be ween ‘ e- qui es’ and ‘excludes’ cons ain s is co ec ly p ocessed. Inpu ea u e has minimum com- monali y. A BC Fea u e=B 0% C-2 C-5 C-6 C-7 49 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. C-29 Check whe he he in e ac ion be ween ‘ e- qui es’ and ‘excludes’ cons ain s is co ec ly p ocessed. Inpu ea u e has maximum com- monali y. A BC Fea u e=C 50% C-2 C-5 C-6 C-7 C-30 Check whe he he in e ac ion among h ee o mo e di e en ypes o ela ionships and con- s ain s is co ec ly p ocessed. Inpu ea u e has minimum commonali y. A B D E C F G Fea u e=G 25% C-1, ..., C-29 C-31 Check whe he he in e ac ion among h ee o mo e di e en ypes o ela ionships and con- s ain s is co ec ly p ocessed. Inpu ea u e has maximum commonali y. A B D E C F G Fea u e=B 100% C-1, ..., C-29 C-32 Check whe he non-exis en ea u es a e co - ec ly managed. A B Fea u e=C 0% C-2 C-33 Check whe he a p oduc can e oneously in- clude mo e han one al e na i e ea u e. B D G A H C F E 0% C-1 C-4 C-12 C-13 C-14 C-25 C-26 Table 12: Ope a ion Commonali y. Tes cases A.7 Ope a ion Dead Fea u es 50 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. DF-1 Check whe he dead ea u es caused by an ‘ex- cludes’ cons ain s be ween a manda o y ea- u e and an al e na i e child ea u e a e co - ec ly de ec ed. A BC D E D DF-2 Check whe he dead ea u es caused by he a ‘ equi es’ cons ain s be ween a manda o y ea u e and an al e na i e child ea u e a e co ec ly de ec ed. The pa en ea u e in he al e na i e ela ionship is manda o y. A BC D E E DF-3 Check whe he dead ea u es caused by an ‘ex- cludes’ cons ain s be ween a manda o y ea- u e and one o he child ea u es o an o - ela ionship a e co ec ly de ec ed. A BC D E D DF-4 Check whe he dead ea u es caused by an ‘ex- cludes’ cons ain be ween a manda o y and an op ional ea u e a e co ec ly de ec ed. A BC C DF-5 Check whe he dead ea u es caused by an ‘ex- cludes’ cons ain be ween wo manda o y ea- u es a e co ec ly de ec ed. A BC A,B,C DF-6 Check whe he dead ea u es caused by a ‘ equi es’ cons ain be ween wo al e na i e child ea u es a e co ec ly de ec ed. A B C B 51 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01 ID Desc ip ion Inpu Exp. Ou pu Deps. DF-7 Check whe he dead ea u es caused by an ‘ex- cludes’ cons ain be ween a pa en and i s child ea u e a e co ec ly de ec ed. A B A,B DF-8 Check whe he dead ea u es caused by an ‘ex- cludes’ and a ‘ equi es’ cons ain s be ween wo op ional ea u es a e co ec ly de ec ed. A BC B DF-9 Check whe he dead ea u es caused by he a ‘ equi es’ cons ain s be ween a manda o y ea u e and an al e na i e child ea u e a e co ec ly de ec ed. The pa en ea u e in he al e na i e ela ionship is op ional. A BC D E E Table 13: Ope a ion DeadFea u es. Tes cases 52 FaMa Tes Sui e 1.2 ISA Technical Repo ISA-10-TR-01