A Case Study for Generating Test Cases from Use Cases

209 ACase Study for Generating Test Cases from Use Cases Javier J. Gutierrez, Maria J. Escalona, Manuel Mejias, Jesus Torres, Arturo H. Centeno Department of Computer and Software Languages University of Sevilla Spain {javierj, escalona, risotto, jtorres}@lsi. us. es AbstractThe verification of the correct implementation of use eases is avital task in software development and quality assurance. Although there are many works describing how to generate test eases from use cases, there are very few ease studies and empirical results of their application and effectiveness. This paper introduces afirst ease study that test the correct implementation of use cases in aweb system and acommand line system, analyses the results and exposes that generation of use cases has asuccessful about 80%. Index Terms-Test objective, Use case, Automatic generation, Testing tools, Case study, Empirical evaluation. I. INTRODUCTION Nowadays, use cases are awidely used technique to define the functional requirements of software systems. Several authors, like Cockburn [6], Ben Achour [4] or Escalona [9] [10], propose how to define use cases with UML Use Case Diagrams, which describes the relations between use cases and actor and between use cases and other use cases, in combination with templates for describing the behaviour of every use case and their preconditions, post-conditions, performance, priority, stability, etc. Templates are writing in narrative English with few or none formalism. Two reports [7] [11] discovered two main gaps in the generation of test cases from use cases: lack of automatism and absence of empirical evaluation. The automatism of scenarios analysis written in natural language (first gap) has been resolved in our previous works [13] [14] using language patterns and regular expressions for extracting information from use case templates. This paper is focused in the second gap. Few empirical results about functional system testing have been published. So, the main goal of this paper and its original contribution is the execution of two cases studies to measure and evaluate the effectiveness of scenario analysis technique. The case study not only generates test cases, but implements and executes them to evaluate their effectiveness. The scenario analysis technique is acommon technique for generating test cases from use cases. It identifies the scenarios from ause case and generates test cases from them. As mentioned before, the main contribution of our previous papers are the automatism of this technique. This paper is organised as followed: section II describes the technique for generating test cases from use cases using use case scenarios. Then, section III describes systems under test and the preparation of the case study. Section IV describes the results of the case study. Section Vintroduces other related works. Finally, section VI exposes conclusions and ongoing works. II. AN OVERVIEW OF SCENARIO ANALYSIS As seen in section I, ause case is mainly defined by natural language and it is mainly composed of steps. In this paper, those steps are grouped in amain sequence, an alternative sequence and an erroneous sequence. An alternative sequence defines the steps that may be realised as an optional alternative to astep of the main sequence, while an erroneous sequence defines the steps that may be used if astep from the main sequence meets an error and is unable to exercise its behaviour. An example of use case, taken from the case study, is showed in table 2. The textual template is codified as aXML file to improve automation. However, the natural language is often too ambiguous and generic to be automatically processed. Thus, the first task is to translate the behaviour of ause case into amore formal model. An UML Activity diagram has been chosen to define the behaviour of ause case. An Activity diagram allows indicating if an action is performed by the system or 210 by an external action; it includes different execution flows; it does not need to expose information about the implementation of the system or its external interfaces. An example is shown in figure 1. The algorithms used to extract information from ause case and to generate an activity diagram may be consulted in previous papers [13] and [15]. They have been implemented in an open-source software tool called TestGen (available in www.lsi.us.es/-javierj/). The result of this tool is shown in figure 1according to the use case oftable 1. Table 1. Use case example <useCase id="Search link by description"> <description> A use case searches a set of links by their description and shows the results. </description> <mainSequence> <step id="l"> The visitor asks the system for searching links by description. </step> <step id="2"> The system asks for the description. </step> <step id="3"> The visitor introduces de description. </step> <step id="4"> The system searches for links which match up with the description introduced by the visitor. </step> <step id="S"> The system shows the found results. </step> </mainSequence> <alternativeSteps> <astep id="3.1"> At any time, the visitor may cancel the search, then the use case ends. </astep> <astep id="4.1"> If the visitor introduces an empty description, then the system searches for all the stored links and step 5 is repeated. </astep> </alternativeSteps> <errorSteps> <estep id="4.2"> If the system finds any error performing the search, then an error message is shown and this use case ends. </estep> <estep id="4.3"> If the result is empty, then the system shows a message and this use case ends. </estep> </errorSteps> </useCase> . <<Vlsltar» l .. The v~ asks the systemfoneerthtlg ~ bydestrfl:tion . .«vlstDr» 3. The vsltor Introduces I pIeCe d the desalptiOn ofthe seerched Inks. . .«9fstem» 4~2.an errorn1!~ S stown .. the systemflnds any error 0rrrt1g the search. <<system» 4.1. ~system seard1esforalstored lila, . . «system>>. 5• .1. the 'sYster:nshcPNs s' rTeSsage. the visitor htrociJces an enpty piece of 2~----------":iIll «system» 4. The system searches for the links which rretches l4' wth the fntrOdLO!d desaiption •. Not(the vstor tltroduces an el11tY piece ri teet) «~Stem» S'"(he systemshiws the results found. Fig 1. Activity diagram automatically generated. 211 Then, use case scenarios are derived from the activity diagram. The TestGen tool implements the all-nodes, alltransitions, and all-scenarios criteria to select scenarios. For the all-nodes criterion, the TestGen tool selects the paths that go across ahigher number of actions until all the actions of the activity diagrams have been traversed at least once. For the all-transitions criterion, the TestGen tool selects the path that traverse ahigher number of object-flow edges until all of them have been crossed at least once. If the activity diagram has not got any loops, as in figure 1, the all-scenarios criterion selects the paths that go through all output object-flow edges from decision nodes at least once. If the activity diagram has got some loops, the all-scenarios criterion selects the paths that go through all output objectflow edges from decision nodes and all combinations among loops at least once. Table 2shows an example of the paths and use case scenarios that have been obtained after applying the all-scenarios criterion in the activity diagram of figure 1 (each path is atest case). The numbers in each test case indicates the activities and decisions traversed from the activity diagram. Table 2. Paths and a use case scenario. Use case: Search link by description The All-Scenarios Criterion Test cases (Tc): 7 1 : 1, 2, 001, End. 2: 1, 2, 001, 3, 002, 4.1, 003, 4.2, End. 3: 1, 2, 001, 3, 002, 4.1, 003, 004, 5.1, End. 4 : 1, 2, 001, 3, 002, 4.1, 003, 004, 5, End. 5: 1, 2, 001, 3, 002, 4, 003, 4.2, End. 6: 1, 2, 001, 3, 002, 4, 003, 004, 5.1, End. 7: 1, 2, 001, 3, 002, 4, 003, 004, 5, End. Use case scenario 1: 1: The visitor asks the system for searching links by description. 2: The system asks for the description. 001: The visitor cancels the search then the use case ends. End. Second row in table 2describes the steps performed by the test case number 1. ITI. CASE STUDY SETTINGS The goal of this case study is to measure de effectiveness of the test cases generated from use cases using scenario analysis. For this reason, mutant systems, with different behaviour than the one described in its use cases are generated. Effectiveness is measured with the number of mutant killed (this means, different behaviour detected) by the test cases. A. Systems under test Two systems have been tested in the case study: aCRUD web system (WEB) and adesktop system with acommandline interface (CML). Not all use cases have been tested. The statistics of the tested use cases of both systems are resumed in table 3. The WEB system allows to maintenance an on-line link catalogue. The four use cases under test for WEB system are: add new link, search links by description (showed in table 1), show recent links and view details of alink. Table 3. Use cases under test. WEB CML Use cases 43 Total number of steps 14 11 Total number of alternative 13 9 steps The CML system is asimple annotation application. The three use cases under test for CML system are: add anew note, erase all notes and list all notes. Implementation details of both systems are described in table 4. Table 4. Implementatioo details. WEB CML Modules 20 I Lines of code 4630 76 WEB system was developed in Java using Struts 2.0 framework. Modules include Java classes and JSP pages. CML was developed also in Java, as astand-alone class, using standard packages only. B. Mutating use cases Aset of mutant versions of the two systems under test were codified. Those mutant versions were able to run successfully, but they exhibit adifferent behaviour than the one specified in their use cases. Faults were introduced into systems using mutant operators. No classic code mutant operators were used, due our objective is not to change the source code (we do not test code). Our objective was mutant use cases for describing adifferent behaviour. So, the first step was to generate anew set of mutant operators for use cases. These mutant operators were obtained from the use cases fault model introduce by Binder [5]. The complete list of mutant operators used is showed in table 5. Mutant operators were applied over use cases to generate mutant systems. Amutant system is acomplete system with only one fault, which generates adifferent behaviour. Table 6 shows mutants obtained for each use case of the WEB system. 212 Table 7shows mutants obtained for each use case of the CML Table 8. Examples ofmutating a use case. system. Mutants Even little use cases may have abig number of mutants and variations. An example of mutants obtained after apply the mutant operator 3(sudden end of the use case) over the use case described in table I(Search link use case of the WEB system) is showed in table 8. Table 5. Mutant operators for use cases. Step 4: Instead of performing the query, the use case ends. Step 4.1 If description is empty, then the use case ends. 10 11 12 10 11 12 lII-- I-- 0 0 8 I-- - ,-' I- - ~ --, 2 2 m- .- - II II 0n II 0 0 - - i - - 10 -' HJ-il 5 6 ~' 4 n' , fTID 0 0 0 n 10 15 20 30 25 Fig. 2. Mutants obtained for each mutant operator (CML system). 18 17 16 14 12 10 35 Figures 2and 3describe the number of mutants obtained from each mutant operator to the CML system (figure 2) and the WEB system (figure 3). Fig. 3. Mutants obtained for each mutant operator (WEB system). Table 6. Mutants for WEB system. Id Mutation operator ILogical operator from acondition of an alternative or erroneous step replacement. 2Condition of an alternative or erroneous step always evaluated to true or false. 3 A sudden end step. 4Astep deletion. 5Anew step performed by the system addition. 6Incorrect data admission or validation rules replacement. 7Incorrect or incomplete information showed by the system. 8An operation that may fail, always works correctly. 9An operation that has not an erroneous step attached fails. 10 Information showed to the actor has fewer elements. 11 Step performer replacement. Use cases Mutants obtained Add new list 34 Search links 33 List recent links IS View details of alink 10 Avg. mutants per use 92/4 = 23 case IV. CASE STUDY RESULTS Table 7. Mutants for CML system. Use cases Mutants obtained Delete all notes 15 Add new note 18 List notes 20 Avg. mutants per use 53/3 =17'7 case A. Test case generation Test cases were generated (before obtaining mutant systems) using the supporting tool, TestGen, developed in the cited previous works. Table 9enumerates the test cases obtained after applying three coverage criteria (described in section II) over the activity diagram generated from each use case of the WEB system. Table 9. Test cases for WEB use cases. Use cases AlINodes AlIScenarios AlITransitions Add new list 3 10 5 Search links 67 5 List recent 3 3 3 links View details of 13 3 alink Total: 13 23 16 Table 10 enumerates the test cases obtained after applying the three coverage criteria over the activity diagram generated from each use case of the CML system. In both tables (9 and 10), the littlest number of test cases is obtained with the allnodes criterion and the bigger number is obtained with the all-scenarios criterion. Table 10. Test cases for CML use cases. Use cases AIlNodes AlIScenarios AlITransitions Delete allnotes 244 Add new note 183 List notes 4 4 4 Total: 7 16 11 B. Test case codification andexecution Taking every scenario as atest case, the number of scenarios calculated was 23 for WEB system and 16 for CML system. Then, test cases were codified with the help of two test harness described next. Test cases for all-nodes and alltransitions are subsets of the test cases for all-scenarios. The test harness selected for the WEB system was JWebUnit (jwebunit.sourceforge.com). This tool interacts with the web system under test in the same way than aweb browser. It also includes the mnit tool for validating results. For the CML system, an ad-hoc console test harness was developed. This test harness redirects the standard input and output. This one allows simulating aset of inputs from auser and evaluating the output of the system using the mnit tool too. After that, use case scenarios were codified in Java and executed over all the mutant versions generated in section TILB. Results for the test cases obtained with the all-scenarios criteria are listed in table 11. Table 11. Results using all-scenarios aiteria. WEB CML Killed mutants 76 45 Effectiveness 76/92 =82'6% 45/53 =84'9% 213 Results for the test cases obtained with the all-nodes criteria are listed in table 12. Table 12. Results using all-nodes criteria. WEB CML Killed mutants 60 45 Effectiveness 60 /92 =65'2% 45/53 =84'9% Results for the test cases obtained with the all-transitions criteria are listed in table 13. Table 13. Results using all-transitions criteria. WEB CML Killed mutants 72 45 Effectiveness 72 /92 =78'3% 45/53 =84'9% Results for the three criteria are the same in the CML system. This mct seems to indicate than it is possible to reduce .the number of test cases from 16 to 7by selecting the minimum number of test cases for each use case in table 9. The implementation of the all-nodes criterion tends to find the biggest path, this means, the path that traverse abigger number of activities and decisions and, therefore, the test case that verify more steps. So, this fact suggested that size matters when testing use cases. This means that big paths, and test cases that exercise much of the steps of ause case, are better than little paths. However, results from WEB systems invalidate these ideas. As showed in tables 12 and 13, areduction of the number of test cases executed implies areduction of the effectiveness (mutants detected), even when all steps of the use cases are exercised at least once (all-nodes) and all execution flows of the activities diagrams are traversed al least once (alltransitions). This fact is provoked for implementation details of the WEB system. As mentioned, this system has been codified using Struts framework. This tool imposes aconcrete way of work, with object caching, session mechanisms, etc. So, if an operation where executed successfully the first time, second and other times the results are already available and operation is not repeated. This mct justifies the decrement of effectiveness when reducing the test case number and it suggests that abig number of use cases with the same test steps executed in different sequences are useful. V. RELATED WORKS There are several papers and approaches about the testing of use cases defined in atextual tabular notation. An extensive list of references may be found in [7] and [II] reports. In next paragraphs, some of the most relevant approaches are summarised. 214 TDEIUML approach, [8], expresses ause case as aUML activity diagram and uses the Category-Partition method [12] to generate test cases. However, the approach does not indicate if the activity diagram may be generated automatically from the use cases, nor the format in which the use cases must be defined. TOTEM [18], Requirement-based Contract [17] and the CowSuite [2] approaches expressed ause case as an UML sequence diagram. The sequences of messages are expressed as regular expressions and are combined between them to generate test cases. We found some problems using sequence diagrams. It is very difficult to express alternative or erroneous sequences in the same diagram. Information about architecture and internal implementation, like classes and messages are also needed, so it cannot be applied in the early phases of the development. Other approaches work directly with natural language, like references [16] and [19]. All of them propose asimple combinational explosion among all scenarios in ause case. These approaches are quite simple and omit many important aspects, like coverage, test values, expected results or test implementation. As mentioned in introduction, there are very few works that describes case studies. One of them is [3]. In this paper describes the testing of areal E-Ticket system for the Netherlands. For testing, atest scenario was generated (by hand) from each use case scenario, then, each test scenario was instantiate into atest case with concrete test values. Results were satisfactory, discovering and resolving all critical issues and about 50% of medium ones. VI. CONCLUSIONS It is not the same mutating test cases for generating mutant systems than testing real system with real faults. Some studies, like [1] exposes that real system and faults are less elaborated than mutant faults. However, the results described in this paper indicates that it may be valuable apply the use case scenario analysis to real systems. Results from section IV.B suggested that it is not only important to exercise all steps from ause case but exercise them in different combinations. Each step is codified into test code once and, then, used in every test case that exercises the step, so it is easily to generate abig amount of test cases with little codification effort. Moreover, results in section IV.B also exposes that the automatic generation of software testing may detect avaluable amount of errors. However, case studies and experiments are also hard to perform due the generation and implementation of mutants is not automatic. The biggest amount of time dedicated to this case study have been spent in developed mutant systems and executing test cases over mutant systems, instead ofgenerating ofcodifying test cases. The main ongoing work is to repeat this case study with the operational variable analysis technique [5] and compare results. VII. ACKNOWLEDGES This work is supported by the Ministry of Science and Education under the National Program for Researching, Development and Innovation, project QSimTec (TIN200767843-C06-03) and REPRIS (TIN2005-24792-E). REFERENCES [1] Briand L.C. Labiche Y. 2005. Is Mutation an Appropiate Tool for Testing Experiments? International Conftrence of Software Engineering ICSE'05. St. Louis, Missouri, EEUU. [2] Basanieri F. Bertolino A. Marchetti E. 2002. The Cow_Suite Approach to Planning and Deriving Test Suites in UML Projects. Lecture Notes In Computer Science 2460 pp. 383-397. [3] Roubtsov S. Heck P. 2006. Use Case-Based Acceptance Testing ofa Large Industrial System: Approach and Experience Report. TAlC-PART06. Windsor, UK. [4] Ben Achour C. 1998. Writing and Correcting Textual Scenarios for System Design. Natural Language and Information Systems Worlcshop. Vienna, Austria. [5] Binder R. V. 2000. Testing Object-OrientedSystems. Addison-Wesley. USA. [6] Cockburn, A. 2000. Writing Effective Use Cases. Addison-Wesley 1st edition. USA. [7] Denger, C. Medina M. 2003. Test Case Derived from Requirement SPecifications. Fraunhofer lESE Report. Germany. [8] Ruder A. 2004. UML-based Test Generation and Execution. Rilclcblick Meeting. Berlin. Germany. [9] Escalona M.J. 2004. Models and Techniquesfor the Specification and Analysis of Navigation in Software Systems. Ph. European Thesis. Department of Computer Language and Systems. University of Seville. Seville, Spain. (10] Escalona M.J. Gutierrez J.J. Villadiego D. Le6n A. Torres A.H. 2006. Practical Experiences in Web Engineering. 15th International Conference On Information Systems Development. Budapest, Hungary, 31 August2 September (11] Gutierrez, J.1., Escalona M.J., Mejias M., Torres, J. 2004. Comparative Analysis of Methodological Proposes to Systematic Generation of System Test Cases. 3° Workshop on System Testing and Validation. Paris. France. (12] Ostrand T. J., Balcer M. J. 1988. Category-Partition Method Communications of the ACM. 676-686. (13] Gutierrez J.J. Escalona M.1. Mejias M. Torres J. 2006. Derivation of test objectives automatically. Fifteenth International Conference On Information Systems Development (lSD06). Budapest, Hungary, 31 August - 2 September, 2006 [14] Gutierrez J.J. Escalona M.1. Mejias M. Torres J. 2006. Towards a Complete Approach to Generate System Test Cases. IeEIS Doctoral Consortium. Oaphos, Cyprus. (15] Gutierrez J.J. Escalona M.J. Mejias M. Torres J. 2006. Modelos Y Algoritmos Para La Generaci6n De Objetivos De Prueba. Jornadas sobre Ingenieria del Software yBases de Datos JISBD. Sitges. Spain. [16] Heumann, J. 2002. Generating Test Cases from Use Cases. Journal of Software Testing Professionals. EEUU. [17] Nebut C. Fleury F. Le Traon Y. Jezequel J. M. 2006. Automatic Test Generation: A Use Case Driven Approach. IEEE Transactions on Software Engineering Vol. 32. 3. March. (18] Labiche Y., Briand, L.C. 2002. A UML-Based Approach to System Testing, Journal of Software and Systems Modelling (SoSyM) Vol. 1 No.1 pp. 10-42. [19] Naresh, A. 2002. Testing From Use Cases Using PathAnalysis Technique. International Conference On Software Testing Analysis &Review. EEUU