A general approach to Software Product Line testing

A gene al app oach o So wa e P oduc Line es ing El i a G. Ruiz1, Jon Aye di2, José A. Galindo1, Ai o A ie a2, Goiu ia Saga dui2, and Da id Bena ides1 1Uni e sidad de Se illa, Dep . Lenguajes y Sis emas In o má icos, A . Reina Me cedes s/n Se illa - España, {eg uiz, jagalindo, bena ides}@us.es 2Mond agon Unibe si a ea, Dep . de Elec ónica e In o má ica, Goi u 2, Mond agon - España [email p o ec ed], [email p o ec ed], [email p o ec ed] Abs ac . Va iabili y is a cen al concep in So wa e P oduc Lines (SPLs). I has been ex ensi ely s udied how he SPL pa adigm can im- p o e bo h he e iciency o a company and he quali y o p oduc s. Ne - e heless, his b ings se e al challenges when es ing an SPL, which a e mainly caused by he po en ially huge amoun o p oduc s ha can be de i ed om an SPL. Di e en s udies p oposing me hods o es ing SPLs exis . Fu he mo e, he e a e seconda y s udies e iewing and map- ping he li e a u e o he exis ing p oposals. Howe e , he e is a lack o sys ema ic guidelines o p ac i ione s and esea che s wi h he di e en s eps equi ed o pe o m a es ing s a egy o an SPL. In his pape , we p esen a i s p elimina y e sion o a u o ial ha summa izes he exis ing p oposals o he SPL es ing a ea. To he bes o ou knowledge, he e is no simila a emp in exis ing li e a u e. Ou goal is o discuss his u o ial wi h he communi y and en ich i o p o ide a mo e solid e sion o i in he u u e. Keywo ds: So wa e p oduc lines, So wa e es ing, So wa e eusabil- i y. 1 In oduc ion So wa e p oduc lines and a iabili y in ensi e sys ems bene i s om a se o echniques, ools and me hods ha a e used o de elop a se o di e en p oduc s ha sha e some commonali ies [26]. The conc e e unc ionali y ha a ies ac oss p oduc s in he SPL is encapsula ed using an abs ac ion known as ea u e. Fea u e models a e used o encode common and a ying pa s o SPLs [17]. In he li e a u e, we ind eal examples encoding a la ge numbe o p oduc s. Fo example, he Linux Ke nel [24] wi h mo e han 6,000 ea u es o Debian packaging sys ems [11] wi h mo e han 27,000. The la ge amoun o p oduc s ha an SPL can encode, makes i s analysis a ime–consuming and e o p one ask. Then, esea che s p oposed he use o au oma ed analysis echniques [5] o a se o ac i i ies in which es ing is usually one o he mos ele an [12]. SPL es ing ep esen s a new challenge o so wa e es ing p ac i ione s and esea che s [23]. When es ing SPL, each p oduc sha es some common unc- ionali y wi h one o mo e p oduc s, while di e ing in a leas one ea u e. SPLs add es ing complexi y because hey equi e es ing a se o p oduc s a he han a single p oduc . These p oduc s, howe e , sha e common unc ionali y o a i ac s, enabling he euse o some es s ac oss he en i e SPL. Acco ding o [23], se e al s a egies can be used o es SPL p oduc s. These es ing s a egies can be summa ized as ollows: i) es ing p oduc by p oduc , ii)inc emen al es ing, and iii) eusable asse ins an ia ion. Tes ing p oduc by p oduc is a s a egy ha es s all p oduc s one by one, as i hey we e no pa o an SPL. Wi h his s a egy he es p ocess co e s all possible in e ac ions be ween ea u es bu g ows exponen ially in cos as a unc ion o he numbe o ea u es in he SPL. Inc emen al es ing is a s a egy ha s a s by es ing he i s de eloped p oduc and c ea es new uni es s o each new ea u e added. Using his s a egy, he commonali ies in he SPL a e exploi ed o educe es ing e o . Howe e , when a new ea u e is in oduced, all he in e ac ions be ween he new ea u e and he old ones ha e o be es ed, which can be challenging o la ge SPLs. Reusable asse ins an ia ion elies on da a cap u ed in he domain analysis s age o SPL c ea ion o de elop a se o abs ac es cases ha co e all ea u es (bu no necessa ily con igu a ions) in he SPL. These abs ac es s cases a e mapped o conc e e equi emen s in he applica ion enginee ing s age. These las wo es ing s a egies a e designed o educe he SPL combina o ial explosion in es ing cos as a unc ion o he ea u e coun . Wi hin SPL enginee ing, wo di e en p ocesses can be dis inguished: (1) domain enginee ing and (2) applica ion enginee ing. Domain enginee ing is he p ocess o de eloping he pla o m o building p oduc s and de ining he com- monali ies and he a iabili y o he p oduc line. Applica ion enginee ing is he p ocess o de i ing speci ic applica ions by using he pla o m de ined in domain enginee ing and binding he a iabili y o sa is y he needs o each pa icula applica ion [22]. In [15] au ho s p opose an ideal pa h o ollow when i comes o SPL es ing, which is a W es ing model o SPLs ha conside s componen , in eg a ion and sys em es ing o bo h domain and applica ion enginee ing. This pa adigm maps e e y sub-p ocess o ei he domain o applica ion enginee ing. Howe e , and due o he complexi y o he es ing p ocesses when a iabili y is conside ed, in many so wa e p ojec s he e is no such a clea di ision o asks. I is possible o ind some es ing p ocesses (i.e., componen es ing) ha can be s a ed in he domain enginee ing phase and con inued du ing he applica ion enginee ing phase – In ac , i is ecommended o adap he pa adigm o he necessi ies o each SPL. To he bes o ou knowledge, he e is a lack o sys ema ic guidelines ha p e en s he p ac i ione s om hese peculia i ies. Mo i a ed by his, we ha e o mula ed a i s app oach o wha could be a lexible – ye s ill sys ema ic – app oach. The app oach is based on he p inciples s a ed by [23], and comple ed 2 wi h di e en s a egies s a ed o he SPL es ing li e a u e, like mappings and e iews [8,10,28]. Ou goal is o begin a discussion a ound he model wi h he communi y in o de o en ich i and p o ide a solid e sion ha can be used as a e e ence o new and senio SPL de elope s. The emainde o his pape is s uc u ed as ollows: Sec ion 2 p esen s he in o ma ion ega ding he so wa e de eloped o his a icle. Sec ion 3 p esen s backg ound in o ma ion on di e en SPL heo e ical es ing app oaches. Ou p oposed app oach is desc ibed in Sec ion 4 and de ailed in Sec ion 5. Finally in Sec ion 6 we p esen concluding ema ks and lessons lea ned. 2 Running example Online Shop Ca alog Paymen Secu i y P oduc Sea ch Bank Accoun E-coins C edi Ca d High Low {1..3} C edi Ca d implies High Manda o y Op ional O Al e na i e Fig. 1. Online shop ea u e model [29]. Figu e 1 shows he ea u e model o he online shop example ha we will use h oughou his pape . I is a con igu able online shop sys em wi h di e en capabili ies which a e he sys em’s a iabili y poin s, as p oposed in [29]. One o he mos common me hods o modelling a iabili y in indus y consis in using ea u e models [6], in which a iabili y poin s a e mapped in o ea u es and hen ep esen ed in a hie a chical diag am ha depic s he ela ionships be ween ea u es. The ea u e model om Figu e 1 shows ha all online shops mus ha e a ca alog lis ing all he a ailable p oduc s, a se o paymen me hods, and a secu i y le el. Fu he mo e, an online shop can op ionally ha e a sea ch ea u e which allows use s o ind p oduc s mo e easily. Fu he down he hie - a chy, we can see ha he e a e h ee possible paymen me hods, a leas one o which needs o be selec ed: bank accoun , e-coins, and c edi ca d. No e ha he {1..3} ca dinali y anno a ion is edundan in his case, since he o pa en -child ela ionship equi es ha a leas one o mo e sub- ea u es a e selec ed. Finally, 3 he secu i y le el o he online shop mus be ei he high o low, since he al e - na i e pa en -child ela ionship manda es ha exac ly one o he sub- ea u es has o be selec ed. In addi ion o he pa en al ela ionships be ween ea u es, ea u e models may also ha e addi ional c oss– ee cons ain s, which a e p oposi ional o mulas ha u he educe he amoun o alid con igu a ions. In ou example, he “C edi Ca d implies High” cons ain makes he ea u e High o be manda o y when he ea u e C edi Ca d is selec ed. Taking all o his in o accoun , we can de i e a o al o 20 alid online shop a ian s. Fo mo e in o ma ion abou ea u e modeling heo y, e e o [4,5]. O de Summa y S a P oduc Sea ch Ca alog oCa alog addToCa iewP oduc De ails E-coins E-coins P oduc De ails Paymen Choice Bank Accoun Bank Accoun C edi Ca d C edi Ca d Paymen Valida ion P oduc Selec ion Ca Con en Checkou Fea u e Links P oduc Sea ch iewP oduc De ails oCa alog sea chP oduc sea chP oduc iewCa Con en oCa alog cancelO de alidPaymen in alidPaymen oPaymen Choice emo eP oduc F omCa iewO de Summa y selec BankAccoun selec ECoins selec C edi Ca d alida ePaymen alida ePaymen alida ePaymen oCa alog Fig. 2. Online shop 150% model. One way o modeling he beha io o a so wa e sys em is by employing a s a e machine model. This model could also be used o gene a e es cases i we we e using Model-Based Tes ing (MBT) echniques. In he case o SPLs, he so-called 150% model can be buil , which is a domain enginee ing asse ha ep esen s he beha io o he whole p oduc line. 150% models in eg a e all he a iabili y, i.e., he a iabili y ela ed o he whole p oduc line in o one single model [2]. When a speci ic p oduc a ian is selec ed, he a iabili y o he 150% model is bound, o ming he 100% model (i.e., he model speci ic o ha con igu a ion) [2]. This app oach is no only conside ed o models bu can also be used o gene ic code. Figu e 2 shows he 150% s a e machine model o he online shop example, whe e he links o he ea u es om he ea u e model ha e been ep esen ed wi h colo s. No e ha in many cases he 150% model 4 i sel may no be a alid p oduc a ian , since i is no always alid o selec all he ea u es on a single p oduc . Analogously o he 150% s a e machine model shown, o he domain engi- nee ing asse s can be gene a ed, such as use case diag ams, class diag ams, and e en he so wa e sou ce code i sel . These asse s should also be linked o he ea u e model so ha hey can be au oma ically eused o di e en p oduc s. In o de o demons a e how o manage an SPL p ojec , we ha e implemen ed a simple, Ja a based e sion o ou online shops example using Fea u eIDE [27], a ool which can be used o de elop SPLs using he ea u e-o ien ed so wa e de elopmen pa adigm. 3 Top-down s Bo om-up app oach P oduc -Speci ic Tes Sui es Tes Execu ion Top-Down Tes Gene a ion P oduc Selec ion Fea u e model Reusable Componen s Domain Enginee ing Asse s Applica ion Enginee ing Asse s P oduc s Domain Enginee ing Applica ion Enginee ing Fig. 3. SPL es ing Top-Down app oach. P oduc Line Tes Sui e P oduc -Speci ic Tes Sui es Tes Execu ion Bo om-Up Tes Gene a ion P oduc Selec ion Fea u e model Reusable Componen s Domain Enginee ing Asse s Applica ion Enginee ing Asse s P oduc s Domain Enginee ing Applica ion Enginee ing Fig. 4. SPL es ing Bo om-Up app oach. The e a e wo main app oaches a he SPL es ing: (1) he op-down app oach and (2) he bo om-up app oach [29], which a e also e e ed o as p oduc - cen e ed and p oduc line-cen e ed espec i ely in some publica ions [19]. On he one hand, he op-down app oach consis s in selec ing and gene a ing he desi ed p oduc a ian s i s , and hen gene a ing a es cases o each de i ed p oduc indi idually. On he o he hand, he bo om-up app oach consis s in gene a ing a da abase o gene ic es cases o he whole SPL based on he domain enginee ing asse s, such as he 150% model. La e , a iabili y o hese es cases is bound o es indi idual p oduc a ian s. Figu es 3 and 4 show an o e iew o hese app oaches. 5 oCa alog, sea chP oduc , iewP oduc De ails, sea chP oduc , oCa alog, iew- P oduc De ails, addToCa , oCa alog, iewCa Con en , emo eP oduc F om- Ca , oCa alog, iewCa Con en , iewO de Summa y, cancelO de , oCa a- log, iewCa Con en , iewO de Summa y, oPaymen Choice, (selec BankAc- coun OR selec ECoins OR selec C edi Ca d), alida ePaymen , alidPaymen Example 1. Bo om-up es case o online shops. We de ine a es case as a speci ica ion o inpu s used o so wa e es ing. In his pape , hey will consis o sequences o e en s ha igge ansi ions in ou example s a e machine model. As an example o he bo om-up app oach, i we wan ed o gene a e a p oduc line es sui e o ob ain a high ansi ion co e age o he online shops example, we could come up wi h he es case shown in Example 1, whe e he colo s ep esen he same ea u e links as in he Figu e 2 model. oCa alog, sea chP oduc , iewP oduc De ails, sea chP oduc , oCa alog, iew- P oduc De ails, addToCa , oCa alog, iewCa Con en , emo eP oduc F om- Ca , oCa alog, iewCa Con en , iewO de Summa y, cancelO de , oCa - alog, iewCa Con en , iewO de Summa y, oPaymen Choice, selec ECoins, alida ePaymen , alidPaymen Example 2. De i ed bo om-up es case o online shops. A e p oduc selec ion, we can de i e his es case in o p oduc -speci ic es cases o e e y p oduc , allowing us o euse i . No e ha his es case only co e s one o he paymen me hods, e en i mul iple a e selec ed. I , o ins ance, we selec ed a p oduc wi h {P oduc Sea ch, ECoins, LowSecu i y}, i s de i ed p oduc -speci ic es case would be Example 2. As o he op-down app oach, he p oduc -speci ic es sui e will be gene - a ed a e selec ing he p oduc a ian s, so we can jus le e age exis ing so wa e es ing echniques and ools. I has been obse ed by some au ho s ha he bo om-up app oach seems o scale be e han he op-down app oach in e ms o es execu ion cos (con- side ing he o al es case coun , numbe o s eps and numbe o con igu a- ions) [19]. Fu he mo e, conside ing ha he op-down app oach is simila o he adi ional es ing sys em, his pape will be mo e ocused in he bo om-up app oach. 4 P oposed P ocess O e iew This p oposal is based on he s a egy Design es asse s o euse [8]. This means ha es plans and es cases a e c ea ed as soon as possible, usually in domain enginee ing. Ne e heless, applica ion enginee ing es s a e s ill needed, 6 so i is also impo an o encou age he euse o hose p oduc -speci ic es cases de ined in applica ion enginee ing om one o ano he p oduc . In e ac ion Tes ing P oduc Tes ing E olu ion Tes ing Tes Op imiza ion (Op ional) Tes Execu ion Fea u e Model Consis ency Checking Asse Tes ing P oduc Line Tes Gene a ion Domain Enginee ing Applica ion Enginee ing P oduc Line Tes Gene a ion P oduc Line Tes Gene a ionSampling and P io i iza ion Fig. 5. SPL es ing p ocess imeline p oposal. Figu e 5 shows an o e iew o he ac i i ies in ol ed in he es ing p ocess o an SPL and hei mapping in o domain o applica ion enginee ing. I is possible o di ide he SPL es ing wo k low in ou di e en p ocedu es: Asse Tes ing, In e ac ion Tes ing,P oduc Tes ing and E olu ion Tes ing. These p ocedu es a e so ed in ch onological o de in Figu e 5, bu in high a iabili y en i onmen s i is usually impossible o a oid mixing hem. One example is how in [23] i is desc ibed how i is no necessa y o de elop and es e e y asse be o e s a ing o es a p oduc by applying es ing p io i iza ion. The e a e SPL es ing p ocesses ha a e a ec ed and modi ied by all o he a o emen ioned p ocedu es. They a e ep esen ed by whi e boxes in Figu e 5, and he a ows ep esen he dependency be ween hem. In o de o achie e an op imal es execu ion i s ly we need o ensu e ha e e y ea u e selec ed (o added o he SPL due o la e p oduc needs) is consis en wi h he o iginal ea u e model – Fea u e model consis ency checking [1]. A e ha , p io i iza ion o es s should be es ablished o e e y single p ocedu e. Mos imes, and because o he a iabili y o he SPL, sampling echniques ha e o be applied in o de o achie e good co e age o in e ac ion es ing be ween asse s. This is e e ed o as Sampling and P io i iza ion. 7 Once we ha e a clea iew o which in e ac ions and p ocesses should be p io i ized on he SPL es ing, a gene ic es plan is buil . Named as P oduc Line Tes Gene a ion, his p ocess is hea ily in luenced by bo h domain and applica ion enginee ing, and depends on he al eady es ed asse s. I is essen ial o uni y and euse es asse s ( es cases, es scena ios and es esul s) as much as possible om one p oduc o ano he . To achie e his, p oduc -speci ic c ea ed es asse s a e s o ed in he gene ic SPL es ing plan as hey appea , as his has been p o en o impac he e o educ ion [8]. Finally, Tes Op imiza ion can op ionally be pe o med on applica ion engi- nee ing o achie e an op imal Tes Execu ion o e e y p ocedu e. 5 P ocedu es In his sec ion, e e y sub-p ocess om Figu e 5 is de ine. Fu he mo e, some glimpses abou di e en echniques e e y p ocess a e gi en. 5.1 Asse Tes ing Asse s a e de ined as a i ac s buil o he de elopmen o di e en p oduc s o he same SPL [3]. In he li e a u e hey a e also e e ed o as domain a i ac s o co e asse s. These asse s can be es ed independen ly h ough uni es ing making use o adi ional so wa e echniques [8]. Resul s om hese es s a e aluable o e e y p oduc de i ed om a SPL. This p ocess is usually pe o med in domain enginee ing, al hough he e a e some asse s ha canno be es ed un il he e is a p oduc [23]. Depending on he pe spec i e o he SPL de elopmen , di e en app oaches can be made. Acco ding o [23], i is highly ecommended o es commonali ies i s . Commonali ies a e hose asse s conside ed co e in he SPL and ha will be p esen in e e y p oduc . This will be use ul in case ha a e e ence applica ion [21] wan s o be used a he in e ac ion es ing p ocedu e. Once commonali ies a e es ed, he es ing o a iabili y-a ec ed asse s can begin. In o de o achie e an op imal co e age o in e ac ions, he e can be a p io i iza ion o which a iable asse s need o be es ed i s . This will allow in e ac ion es ing o s a ea lie in he es ing p ocess. Example 3 shows a es case o he P oduc De ails class in ou online shops example, which checks he p esence o absence o he P oduc Sea ch op ion in he use menu. E en i P oduc De ails is a co e asse ha is always p esen , his pa icula es case canno be execu ed un il a p oduc is selec ed because he ansi ion o P oduc Sea ch may o may no exis . 5.2 In e ac ion Tes ing Asse es ing is no enough o achie ing a high quali y SPL. In SPLs, in e ac- ion be ween asse s causes ailu es, bugs and inconsis encies ha can only be 8 1public oid es Sea chT ansi ion() { 2// Selec menu op ion 3inpu . p in ln ("2"); // 2. BACK TO CATALOG 4// Run P oduc De ails 5p oduc De ails. un(); 6Lis < S ing > lines = ou pu . lines () . collec () ; 7// Check ou pu 8boolean sea chP oduc P esen = alse ; 9 o ( S ing line : lines ) { 10 i ( line . equals (" 3. SEARCH PRODUCT ")) { 11 asse False(sea chP oduc P esen ); 12 sea chP oduc = ue; 13 } 14 } 15 // #i P oduc Sea ch 16 asse T ue(sea chP oduc P esen ); 17 // #else 18 asse False(sea chP oduc P esen ); 19 // # endi 20 } Example 3. Asse es o he P oduc De ails class. de ec ed when ce ain ea u e combina ions a e p esen . Fo his eason, in e ac- ion es ing is used when es ing SPLs. Va iabili y is con olled on his s age o SPL es ing [7], and esul s will be aluable o e e y di e en p oduc s de i ed om he SPL. In e ac ion es ing includes in eg a ion es ing – which belongs o domain enginee ing [8] – and also binding es ing [23], which belongs o ap- plica ion enginee ing. In high a iabili y scena ios i is impossible o achie e a ull co e age o e e y in e ac ion be ween asse s. This is caused by he numbe o p oduc s an SPL can ha e, ha g ows exponen ially as he numbe o ea u es inc eases. The e o e, sampling is necessa y o es as many di e en in e ac ions as possible while a oiding exhaus i e es ing [28]. The idea behind sampling is o de i e a subse o all possible p oduc s ha collec i ely co e he beha io o he SPL and e eal mos o he aul s by only hem [28]. To sample a p oduc subse om he en i e SPL, se e al app oaches ha e been p oposed. Va shosaz e al., p oposed a axonomy o classi y hese ap- p oaches [28]. This axonomy included (1) inpu da a o he sampling app oach (e.g., ea u e model), (2) ype o algo i hm used o sampling p oduc s (e.g., om simple g eedy-based algo i hm o mo e sophis ica ed popula ion-based al- go i hms) and (3) ype o co e age employed (e.g., ea u e-in e ac ion co e age). Ou o he scope o his pape , he classi ica ion also included he e alua ion echnique and he ype o applica ion o he app oach, among which mos o hem we e ocused on es ing. 9