Generation of test cases from functional requirements. A survey

Gene a ion o es cases om unc ional equi emen s. A su ey Ja ie J. Gu ié ez, Ma ía J. Escalona, Manuel Mejías, Jesús To es Depa men o Compu e Languages and Sys ems Uni e si y o Se ille {ja ie j, escalona, iso o, j o es}@lsi.us.es Abs ac : One o he majo quali y c i e ia o a so wa e sys em is how well i ul ils he needs o use s o cus ome s. One echnique o e i y and imp o e he g ade o ul ilmen is sys em es ing. Sys em es cases migh be de i ed om he equi emen s o he sys em unde es . This pape p esen s he esul s o a su ey among 13 app oaches o d i e he gene a ion o es cases om unc ional equi emen s. 1. In oduc ion So wa e es ing is becoming mo e complex day by day. This complexi y en o ces using echniques and me hods o assu e so wa e quali y. One o his me hods is sys em es ing. The main goal o sys em es ing is o e i y ha equi emen s a e success ully implemen ed in o sys em unde es . In o de wo ds, sys em es ing assu es ha so wa e sys em does wha i is expec ed o do. The main a e ac s o ob ain sys em es cases a e he own equi emen s o he sys em unde es [Be olino04]. This p ocess migh be d i en by sys ema ic me hods and ools. Since 2003, we ha e been s udying and compa ing exis ing app oaches o de i e es cases om he unc ional speci ica ion o he sys em unde es . A p elimina y abs ac o ou su ey was p esen ed in SV04 [Gu ie ez04]. In his pape we p esen a mo e comple e su ey which alida es conclusions exposed in [Gu ie ez04] and ex ends hem. Nowadays, use cases a e widely used o de ine unc ional equi emen s [Escalona04], so bo h e ms a e synonymous in his pape . This wo k is o ganized as ollows. Sec ion 2 desc ibes al eady analyzed app oaches. Sec ion 3 iden i ies he sol ed and unsol ed aspec s in he p ocess o gene a ion o es cases. Finally, sec ion 4 exposes conclusions and u u e wo k. 2. S a e o he a The e a e se e al app oaches o de i e es cases om unc ional equi emen s when hose equi emen s a e exp essed in a o mal and p ecise no a ion, like Z o algeb aic speci ica ions. Howe e , mos o he so wa e indus y wo ks wi h equi emen s in na u al languages. Templa e and abula models, like he ones in oduced by [Cockbu n00] o [Escalona04], a e widely used o de elop all kind o sys ems. This su ey is ocused on app oaches ha s a om unc ional equi emen s exp essed in na u al language, gene ally as use cases. Ac ually, we only know wo epo s ha analyse and compa e app oaches o gene a e sys em es cases om equi emen s. The i s epo [Denge 03] (called Denge om he e on) analyses 12 app oaches. The second epo is his su ey i sel (called Gu ie ez om he e on), which analyzes 13 app oaches (lis ed in able 1). Fi e o hose app oaches a e also analyzed in Denge epo (Id 1, 3, 7, 8 and Ca ego y Pa i ion in 11). Addi ional app oaches included in Denge epo a e lis ed in able 2. Those app oaches a e qui e simila o he ones in able1, so, hey ha e been omi ed. Id Yea Ti le Re e ences 1 1997 So wa e Requi emen s and Accep ance Tes ing. [Hsia95], [Hsia97] 2 2000 Au oma ed Tes Case Gene a ion om Dynamic Models [F öhlich00] 3 2000 Ex ended Use Case Tes Design Pa e n [Binde 00] 4 2001 Requi emen Base Tes ing [Mogyo odi02], [Mogyo odi03] 5 2002 A UML-Based App oach o Sys em Tes ing [Labiche02] 6 2002 Tes Cases om Use Cases [Heumann02] 7 2002 Tes ing F om Use Cases Using Pa h Analysis Technique [Na esh02] 8 2002 Use Case De i ed Tes Cases. [Wood02] 9 1999 Scena io-Based Valida ion and Tes o So wa e [Glinz99] , [Ryse 03] 10 2003 Requi emen s by Con ac [Nebu 03] 11 2003 PLUTO & Ca ego y Pa i ion Me hod [Os and88], [Be olino03], [Be olino04] 12 2004 De i a ion o Domain Tes Scena ios om Ac i i y Diag ams. [Rude 04], [Rude 04-2] 13 2004 Requi emen s o Tes ing in a Na u al Way [Boddu04] Table 1: App oaches analyzed in Gu ie ez Repo . Ou p e ious pape includes ou app oaches. Only wo o hem (id 6 and id 9 in able 1) ha e been included in he inal e sion o he su ey. An app oach omi ed is UML-Based S a is ical Tes Case Gene a ion due we ha e ocused only on unc ional e i ica ion. Ano he omi ed app oach is AGEDIS [Ha man04], since a u he in es iga ion and con e sa ion wi h hei au ho s shows us ha i is ocused on design models and ha i equi es he sys em buil . Howe e , AGEDIS pape s a e a good sou ce o ideas. Yea Au ho s Ti le Re e ence 1998 Meye S. Sand oss R. Applying Use-Case Me hodology o SRE and Sys em Tes ing. STAR Wes Con e ence. 1999 Ca pen e P.B. Ve i ica ion o Requi emen s o Sa e y-C i ical so wa e. SIGAda'99. Redondo Beach CA. USA. 1999 Colla d R. Tes Design: De eloping Tes Cases om Use Cases. So wa e Tes ing & Quali y Enginee ing Magazine. 1999 Cunning S.J. Rozenbli J.W. Tes Scena io Gene a ion F om a S uc u ed Requi emen Speci ica ion. IEEE Enginee ing o Compu e Based Sys ems. 2000 Hindel B. Hehn U. Cons uc ing Tes Cases om De i ed Requi emen s. Eu oSPI. 2002 Blacbu n M. Busse R. Nauman A. In e ace-D i en Model-Based Tes Au oma ion. In e na ional Con e ence on So . Tes ing Analysis & Re iew 2002 Pudipeddi H.V. Unde s anding, Designing and Tes ing Use Cases. h p://www.s ickyminds.com Table 2: App oaches analyzed in Denge Repo . App oaches analyzed in Gu ie ez epo can be di ided in o h ee g oups (shown in igu e 1) due o hei no a ion o modelling sys em beha iou . The i s g oup de i es es cases di ec ly om equi emen s in na u al language. The second g oup builds a model om he unc ionali y and de i es es cases om ha model. A e ac s used o exp ess he beha iou o he sys em a e lis ed in able 4. The hi d g oup is based on he Ca ego y-Pa i ion echnique. Two app oaches join echniques om he wo g oups. App oach [Boddu04] builds s a e machines om equi emen s in na u al language as a i s s ep o de i e es cases. App oach [Rude 04] builds an ac i i y diag am anno a ed wi h ca ego ies and pa i ions. Figu e 1: Techniques o modelling he beha iou o he sys em. The e a e also di e ences among he a e ac s needed o gene a e es cases. Some app oaches s a om use cases desc ibed wi h UML Use Case diag am [UML03] and comple ed wi h empla es in na u al language. O he app oaches include hei own elici a ion phase, so hei s a ing poin is a se o needs om he use s. One app oach needs use cases and addi ional in o ma ion, like class diag ams and sequence diag ams. A classi ica ion is shown in igu e 2. Figu e 2: A e ac s needed o gene a e es cases. The esul s ob ained om he app oaches ha e been classi ied in o 2 g oups ( igu e 3). The i s g oup deno es ha de i ed es cases a e exp essed like ac ions o e he sys em in na u al language o decision ables (like [Binde 00]). Those es cases mus be implemen ed by hand. The second g oup allows us o ob ain es sc ip s. Howe e , hey used comme cial o expe imen al ools, no a ailable. In some cases, he Ca ego y-Pa i ion me hod can gene a e execu able esul s, as desc ibed in [Balce 89]. Figu e 3: Resul s ob ained om each app oach. The e a e se e al app oaches ha men ion a suppo ing ool. A classi ica ion is shown in igu e 4. The only ool eely a ailable ha we ha e ound is a suppo ing ool o [Nebu 03] app oach. Many o he analyzed app oaches include he gene a ion o expec ed esul s. Howe e , in all o he app oaches, his ac i i y is de eloped by hand, using he c i e ion o he es e s. Figu e 4: Suppo ing ools. Denge and Gu ie ez analyses ha e simila conclusions al hough Gu ie ez epo is wo yea s olde han he Denge epo . The main conclusion o Denge epo is ha app oaches wo k a a heo e ical le el wi hou desc ibing how o gene a e execu able es cases. Denge epo exposes ha he e is oom o mo e app oaches o sol e he oubles and he lacks de ec ed in he exi ing ones. The conclusions a Gu ie ez epo exposes ha new app oaches ha e he same lacks han he ones analyzed in Denge . The e is a lack o documen a ion, case s udies, and a ailable suppo ing ools. These ac s make ha d he applica ion o heses app oaches o e eal p ojec s. 3. Resol ed and un esol ed aspec s A e analyzing and compa ing app oaches lis ed in able 1, we ha e ex ac ed which aspec s a e ea ed sa is ac o ily and which aspec s need a u he de elopmen . Those aspec s a e lis ed in able 3 and desc ibed in he ollowing pa ag aphs. Resol ed aspec s Un esol ed aspec s • Building beha iou al mode. • Suppo ing ools. • Two le els in es cases de i a ion. • De ini ion o equi emen s • Documen a ion and p ac ical cases. • No measu e o e ec i eness and quali y. • Lack o sys ema iza ion and au oma ion. • Tes case implemen a ion. • Lack o empi ical s udies. Table 3: Resol ed and un esol ed aspec iden i ied. As seen in igu e 1, en app oaches de elop a beha iou al model o exp ess he unc ionali y expec ed in he sys em unde es . The no a ions used a e lis ed in able 4. Re e ence Beha iou al model [Hsia97] Scena io ees and ini e s a e machines. [F öhlich00] Fini e s a e machines. [Mogyo odi03] Cause-e ec diag ams. [Labiche02] UML ac i i ies and sequence diag ams. [Na esh02] Ac ion- low diag am. [Nebu 03] Use cases anno a ed wi h p econdi ion, pos -condi ions and in a ian s. [Ryse 03] Own Use cases execu ion model (simila o FSM). [Os and88], [Be olino04] Ca ego ies, choices and es ic ion. [Rude 04] UML ac i i y diag am anno a ed wi h ca ego ies, choices and es ic ion. [Boddu04] Fini e s a e machine. Table 4: Techniques o gene a e beha iou al model. The use o beha iou al models allows sys ema ize and au oma e he p ocess. A widely used echnique in es ing is called model-based es ing [Be olino04-2]. Tes ing app oaches based on model-based es ing migh also be applied o de i e sys em es cases om beha iou al models. The exis ence o suppo ing ools can be classi ied as a esol ed bu also as an un esol ed aspec . Se en o he app oaches analyzed ha e a suppo ing ool. This ac is a g ade o hei ma u i y and imp o es hei au oma ion. Howe e , an un esol ed aspec is o allow he access o hese ools by ee o open-sou ce licences o , a leas , in es iga ion licences, like AGEDIS p ojec . An aspec disco e ed, is ha i is necessa y o de i e wo le els o es cases. I is well known in code es ing ha es sepa a e componen s o agmen s o code es ed wi h uni es cases do no gua an ee he p ope ope a ion o he sys em [Cohen04]. Thus, an addi ional in eg a ion es ing phase which e i ies ha each componen wo ks success ully in collabo a ion wi h o he componen s is needed. This philosophy can be also applied o sys em es ing. Sys em es ing can be di ided in o wo s ages. The i s s age e i ies he beha iou o each use case in isola ion (simila o code uni es ing). The second s age e i ies ha use cases wo k p ope ly oge he and sa is y unc ional equi emen s (simila o code in eg a ion es ing). Bo h s ages can be combined in o de o ob ain es cases ha combine se e al equi emen s and se e al execu ion pa hs o each equi emen . Howe e , he e a e only 2 app oaches (id 6 and 10 in able 1) ha include dependences o use cases and de i e es cases ha in ol e se e al use cases. So, his s age has o be s udied deeply o new app oaches. An impo an aspec is how o de ine a use case o de i e a se o es cases o m i . Mos o he app oaches indica e he in o ma ion ha a use case has o include o be use ul in es case de i a ion. As he e is no a s anda d widely accep o de ine use cases, his aspec is esol ed enough. The e a e also se e al aspec s ha we es ima e ha hey a e s ill un esol ed. Those aspec s a e lis ed in Table 3. The i s aspec is he lack o documen a ion. All app oaches ha e li le o incomple e echnical documen a ion. Many app oaches do no e e o any p ac ical applica ion o ealis ic case s udy. Bo h ac s obs uc eal applica ion o hose app oaches. An aspec al eady o go en by all app oaches is o p opose me ics and ools o e alua e he quali y o gene a ed es cases. We assume ha all app oaches s a om he assump ion ha , i hei s eps a e success ully applied, a se o es cases wi h he maximum co e age a e ob ained. Since he e a e se e al decisions ha es ing eam has o adop , a me ic o e alua e how hose decisions a ec o co e age is needed. This aspec is also ela ed o he absence o empi ical s udies abou e ec i eness o each app oach. None o he au ho s demons a ed wi h expe imen s ha hei app oach is be e han andom es ing o han using common sense. Co e age c i e ion in he app oaches analyzed (excep in [Binde 00]) is mainly based in a combina ional explo a ion o all possible scena ios om a use case. Some app oaches, like [Rude 04], [Nebu 03] o [Na esh02], include se e al co e age app oaches (like all- ansi ion o all-s a es) o minimize he combina ional explosion. The implemen a ion o es cases is also an aspec b ie ly (o omi ed) desc ibed in he analyzed app oaches. The wo app oaches ([Rude 04] and [Boddu04]) ha de i e execu able es sc ip s need ools no a ailable. As seen in igu e 3, many app oaches jus only gene a e ex ual desc ip ion o es cases in na u al language. Addi ional and non- i ial s eps ha e o be pe o med o implemen hose es cases in o execu able sc ip s o code. 4. Conclusions The main conclusion is ha he e a e enough app oaches o acqui e p ecise ideas on how o de i e es cases. Howe e , he e is s ill no a comple e and in eg a ed app oach ha desc ibes he whole p ocess. F om his su ey, i is concluded ha he mos -used echnique is model- based es ing. This echnique is being used mo e and mo e e e y day in he so wa e indus y. Model-based es ing is success ully applied and documen ed when i is applied o design models o when models a e ex ac ed om code. Ou u u e wo k is o de elop a model-based es ing app oach om unc ional equi emen . This app oach has o be able o de i e es cases om ea ly de elopmen phases, allowing imp o e quali y equi emen s and minimizing ime and e o used in sys em es ing [Dahls ed 05]. The inc easing numbe o analyzed app oaches, om 4 o 13, has con i med he p elimina y conclusions p esen ed in [Gu ie ez04]. None o he app oaches is he de ini e one. Wi h hei s udy as a whole se , we ha e exposed i s esol ed and un esol ed aspec s, so he e is a gasp o a new app oach which uses he esol ed aspec and co ec s he un esol ed ones. Numbe Task 1 Build beha iou al model. 2 De i e es scena ios om one use case. 3 De i e es scena ios om se e al use cases. 4 Gene a e es alues. 5 Ob ain es scena ios. 6 Reduc ion he numbe o es s cases wi hou los o co e age. 7 Measu e o co e age. 8 Gene a e expec ed esul s. 9 So es cases o maximize a selec ed c i e ion ( es scena io p io i ize). 10 Build es sc ip s o execu able es code. Table 5: Tasks o gene a e es cases om use cases. In pape [Gu ie ez04], we exposed, as u u e wo k, he de elopmen o a new me hodology. Nowadays, we ha e de ined he scope o he new app oach om he conclusions ob ained om Gu ie ez epo . The scope o ou app oach is lis ed in able 5. Ac i i y App oach 1 App oaches lis ed in able 4. 2 All app oaches 3 [Labiche02], [Nebu 03] 4 App oaches based in CP me hod [Be olino04], [Rude 04], [Os and88] 5 [Labiche02] 6 [Na esh02] 7 [Binde 00] 8 No app oaches 9 [Na esh02] 10 No app oaches Table 6: App oaches ha men ioned s eps lis ed in able 5. Compa ing he aspec s exposed in sec ion 3 wi h asks lis ed in Table 5, we ha e de ined he g ade o ma u i y and co e age o each ask om exi ing app oaches. Resul s a e lis ed in able 6. Some p elimina y pape s and se e al cases o s udy ha e been de eloped. Howe e , none o hem ha e been published ye . Re e ences [Balce 89] Balce M. J., Hasling W. M., Os and T. J. 1989. Au oma ic Gene a ion o Tes Sc ip s om Fo mal Tes Speci ica ions. ACM, pp 210- 218. USA. [Be olino03] Be olino, A., Gnesi, S. 2003. Use Case-based Tes ing o P oduc Lines. ESEC/FSE’03. Helsinki, Finland. [Be olino04] Be olino, A., Gnesi, S. 2004. PLUTO: A Tes Me hodology o P oduc Families. Lec u e No es in Compu e Science. Sp inge - Ve lag Heidelbe g. 3014 / 2004. pp 181-197. [Be olino04-2] Be olino A., Ma che i E., Faedo A. 2004. In oducing a Reasonably Comple e and Cohe en App oach o Model-based Tes ing. TACoS'04 P elimina y Ve sion. [Be olino04] Be olino, A., Gnesi, S. 2004. PLUTO: A Tes Me hodology o P oduc Families. Lec u e No es in Compu e Science. Sp inge - Ve lag Heidelbe g. 3014 / 2004. pp 181-197. [Binde 00] Binde R. V. 2000. Tes ing Objec -O ien ed Sys ems. Addison- Wesley. USA. [Boddu04] Boddu R., Guo L., Mukhopadhyay S. 2004. RETNA: F om Requi emen s o Tes ing in Na u al Way. 12 h IEEE In e na ional Requi emen s Enginee ing RE’04. [Cockbu n00] Cockbu n, A. 2000. W i ing E ec i e Use Cases. Addison- Wesley 1s edi ion. USA. [Cohen04] Cohen, F. 2004. Ja a Tes ing and Design. F om Uni Tes ing o Au oma ed Web Tes s. P en ice Hall. USA. [Dahls ed 05] Dahls ed , Å. (2005) Guidelines Rega ding Requi emen s Enginee ing P ac ices in o de o Facili a e Sys em Tes ing. 11 h In e na ional Wo kshop on Requi emen s Enginee ing. Po o, Po ugal. [Denge 03] Denge C., Medina M. 2003. Tes Case De i ed om Requi emen Speci ica ions. F aunho e IESE Repo . [Escalona04] Escalona M.J. 2004. Modelos y écnicas pa a la especi icación y el análisis de la Na egación en Sis emas So wa e. Ph. Eu opean Thesis. Depa men o Compu e Language and Sys ems. Uni e si y o Se ille. Se ille, Spain. [F öhlich00] F öhlich, P, Link, J. 2000. Au oma ed Tes Case Gene a ion om Dynamic Models. ECOOP 2000. pp. 472-491.