Improving NDT with Automatic Test Case Generation

Improving NDT with Automatic Test Case Generation J. Gutiérrez , M. J. Escalona , M. Mejías , F. Domínguez , and C. R. Cutilla Abstract The model-driven development defi nes the software development process as a set of iterations to create models and a set of transformations to obtain new models. From this point of view, this paper presents the enhancement of a modeldriven approach, called navigational development techniques (NDT), by means of new models and transformations in order to generate test cases. It also states some conclusions from the research work and practical cases in which this approach was used. 1 Introduction Model-driven development (MDD hereinafter) is a software engineering paradigm focused on creating and exploiting domain models [ 1 ]. Navigational development techniques (NDT) [ 2 ] is a development framework which follows MDD. Therefore, NDT describes the full software development cycle indicating which models generate in each phase and how to derive the models of a phase from those of a previous phase. Projects developed with NDT start with a goal-oriented phase of require-ments and apply use cases for defi ning requirements and transformations so as to generate the following models. NDT was initially defi ned to deal with Web development requirements, but it has evolved in the last years and nowadays it offers a complete support for the complete life cycle. NDT covers viability study, requirements treatment, analysis, design, construction, or implementation as well as maintenance and test processes. Additionally, it supports a set of processes to bear out project management and quality assurance. Figure 1 depicts the development process as defi ned in NDT. This development process is independent from the development life cycle, so any life cycles (e.g., cascade or iterative) may be applied. Testing is always a mandatory task (Fig. 1 ) regardless of the type of project. Thus, there is a need of incorporating techniques in order to defi ne test cases. Moreover, due to the MDD nature of NDT, these techniques should be described in terms of the proper elements of MDD, mainly models and transformations from models to models. This paper describes how NDT has been extended to incorporate functional system test cases [ 3 ]. These test cases verify that the system under test commits the behavior defi ned in its functional requirement. NDT models the functional requirements as use cases; thus, both terms will be used as synonyms in this paper. Fig. 1 NDT development process overview This paper is organized as follows. After this introduction, Sect. 2 offers an overview of the existing techniques dealing with generating functional test cases from functional system requirements defi ned as use cases. Section 3 describes the techniques used to generate test cases. Section 4 summarizes the extension of NDT to incorporate those techniques from a MDD perspective. Section 5 presents practical applications for NDT enrichment with test case generation. Finally, Sect. 6 states the conclusions and ongoing work. 2 Related Work There are several approaches generating functional test cases specifi cally from a functional requirements model defi ned as use cases by means of MDD. A survey about this issue, which updates the original survey published in [ 4 ], has been published in [ 5 ] at the end of 2011. Some specifi c approaches studied in Escalona’s survey are described in next paragraphs. Frölich et al. [ 6 ] introduce an approach describing how to translate a functional requirement from natural language into a state-chart diagram in a systematic way as well as how to generate a set of functional test cases from that diagram. Naresh [ 7 ] presents an approach dealing with translating a functional requirement from natural language into a fl ow diagram and performing a path coverage technique to generate test cases. Mogyorodi [ 8 ] introduces an approach describing functional requirements as cause-effect graphs which generates test cases from diagrams. Boddu et al. [ 9 ] present an approach divided into two blocks: the fi rst one describes a natural language analyzer generating a state machine from functional requirements, and the second one shows how to create test cases from such state machine. Ruder’s [ 10 ] approach starts with functional requirements written in natural language. The result is a set of functional test cases obtained from a coverage criterion based on combinations that support Boolean propositions. Binder’s book [ 11 ] describes the application of the category-partition method over use cases. The categories are any point in which the behavior of the use case may be different between two realizations of the use case. This application is named the extended use case pattern. Finally, Ibrahim et al. [ 12 ] offer a tool, called GenTCase, which generates test cases automatically from a use case diagram enriched with each use case tabular text description. Escalona’s survey claims that there is no defi nitive approach that closes the problem of generating functional text cases automatically in a satisfactory way, what implies a lack of evolution among the existing approaches. Thus, there are some aspects to be improved, like the use of standards for inputs and outputs, the application of standards and more formal methods to describe the process itself, the need for empirical results or the measure of the possible automation, and a profi table tool supporting, among others. Conclusions of Denger’s survey go in the same line. 3 Techniques for Test Cases Generation Two techniques have been identifi ed for generating test cases from use cases, from the surveys cited in previous sections: round-strip strategy and extended use cases (terminology defi ned by Binder in [ 11 ]). Below, these techniques will be described in depth. The round-strip strategy consists in the application of a classic algorithm of path fi nding over a state machine. The behavior described in a functional requirement may be managed as a graph or as state machine despite its concrete syntax. Hence, a path searching allows identifying all the different paths across the behavior. Each path will be a scenario designed together with the system. Each scenario is a potential test case for testing the right implementation of such scenario in the system under test. Generation of test cases from state machines is a widely described topic in research literature. Previous section presented several references about this topic in the specifi c use cases context, like [ 6 , 7 , 9 ]. Figure 2(a) shows an example of the round-strip strategy using the behavior of a use case defi ned as an activity diagram. The extended use case pattern consists in applying the category-partition method [ 13 ] to use cases. The category-partition method is a technique based on identifying categories and partitions and then generating combinations among such partitions (Fig. 2b ). In the context of functional requirements, a category is any point for which the functional requirement defi nes an alternative behavior (Fig. 2b ). Besides, a partition is defi ned as a subset of the domain of the condition evaluated in the category which decides whether a concrete piece of behavior is executed or not. Once all categories and partitions are identifi ed, a combination among them is performed and each combination becomes a potential test case. The previous section presented several references about this topic in the specifi c context of use cases, like [ 10 ] or [ 11 ]. Figure 2b shows an example of the category-partition method (as described in [ 11 ]) using the same behavior as Fig. 2a . Fig. 2 Examples of round-strip strategy ( a ) and extended use cases ( b ) techniques 4 Extension of NDT This section describes the work carried out to extend NDT after including the two techniques presented in the previous section. Section 4.1 defi nes the information involved. Then, Sect. 4.2 defi nes how to apply both techniques to obtain the target test artifacts from the functional requirements. Finally, Sect. 4.3 offers an overview on the application of Sects. 4.1 and 4.2 results. 4.1 Concepts and Metamodels Due to the model-driven nature of NDT, the concepts involved in generating functional test cases should be identifi ed and defi ned as metamodels. A metamodel defi nes the concept in terms of its attributes and its relationships with other concepts [ 1 ]. Four metamodels were designed. The fi rst one (Fig. 3 ) defi nes the necessary elements from functional requirements to generate test cases. These elements constitute a subset of functional requirements. Therefore, it only involves the elements used for test cases generation. This metamodel may be applied with other frameworks apart from NDT. The functional requirement metamodel (Fig. 3 ) includes classic elements of functional requirement defi ned as use cases , Step or Actor, among others, widely described in the literature. The second metamodel (Fig. 4 ) defi nes the concepts resulting from the roundstrip technique (Fig. 2a ). Each path is called test scenario (element TestScenario in Fig. 4 ) and the traverse/crossed steps are classifi ed into actions (element ActionFromTestScenario in Fig. 4 ) when performed by an external actor or into verifi cations (element Verifi cationFromTestScenario in Fig. 4 ) when performed by the system and, therefore, is suitable to introduce an assert during the test. The third metamodel (Fig. 5 ) defi nes the concepts resulting from the categorypartition method. Categories are modeled using the element OperationalVariable class Functional Requirement Metamodel SystemActor name: String description: String FunctionalRequirement name: String description: String priority: String notes: String [0..1] Step action: String mainStep: Boolean ExecutionOrder target 1 in * 1 out * executor 0..1 interaction 1..* 1 step 1..* {ordered} source Fig. 3 Metamodel for functional requirements (as named in [ 11 ]), whereas partitions are modeled through the element Partition. The element Instance points out an evaluation of an operational variable, for example, A or B cells in Fig. 2b , and allows distinguishing it from other evaluations of the same operational variable, in case the behavior of the functional requirement has loops. A Quantum element models a value transfer from a partition to an instance. A combination (a row in Fig. 2b ) is modeled using the element Instancecombination . Finally, the last metamodel introduces artifacts that combine the results of the two previous techniques in the same model. This last metamodel does not introduce any new information. However, it offers glue elements to represent the information through a common artifact (called test case), the steps from a functional requirement as well as a combination of partitions. Figure 6 shows the tracing relation between the four metamodels. Tracing enables knowing which test artifacts have been generated for each functional requirement. class Test scenarios TestScenario name: String description: String notes: String [0..1] StepFromTestScenario body: String TestActor name: String description: String VerificationFromTestScenario ActionFromTestScenario testScenarioStep 1..* {ordered} scenario 1..* executor 1 interaction 1..* {complete, disjoint} Fig. 4 Metamodel for test scenarios class Test Values OperationalVariable name: String description: String domain: String [0..1] comments: String [0..1] InstanceCombination description: String [0..1] DataPartition name: String description: String [0..1] rangeOfValues: String [0..1] Instance Quantum partition 1..* subdomain specific * general 0..1 instance 1..* {ordered} operationalVariable quantums 1..* combination 1..* quantum 1..* instance 1 quantum 1..* dataPartition 1 Fig. 5 Metamodel for test values Some additional elements from the metamodels have been omitted. These elements introduce additional concepts like preconditions and packages. The four former metamodels have been added to the set of metamodels managed and supported by NDT as part of its MDD development process. 4.2 Relations and Transformations Section 4.1 described the concepts involved in the improvement of NDT to generate functional system test cases. This section goes one step beyond and describes how to apply the two techniques presented in Sect. 2 (round-strip and extended use cases) using the information from the functional requirements metamodels (in the previous section) as source and the information from the testing metamodels as target. The process of applying both techniques is defi ned according to the identifi cation of a set of relations between source concepts (functional requirements) and target concepts (test scenarios and operational variables combinations), as observed in Fig. 7 . The task of identifying these relations consists in detecting how to build up one target element, for example, a test case, by means of the source elements and their information. Next paragraphs provide an overview of the three relations (named T1, T2, and T3 in Fig. 7 ) defi ned to create test scenarios, combinations of operational variables, and test cases from functional requirements. Relation T1 involves functional requirements and the round-strip strategy. As it was represented in Fig. 2a , the functional requirement behavior may be modeled as a state machine; the concept Step from Fig. 3 models the states; and the concept Execution Order models the transitions. Thus, a classic coverage criterion may be selected to traverse/cross the functional requirement and generate test scenarios. The all-loops criterion, in which all combinations among loops are traversed at least once, is the one selected to extend NDT. Test scenarios steps are generated from all the functional requirements steps. Action (element ActionFromTestScenario ) and verifi cation (element Verifi cationFromTestScenario ) classifi cations depend on whether there is a relation with a system actor. Finally, test actors are generated from actors, which, due to their attributes, are the same ones. class Dependencies Functional Requirements Metamodel Test Scenario Metamodel Test Values Metamodel Test Case Metamodel «trace»«trace» «trace» «trace»«trace» Fig. 6 Tracing relationships among metamodels Relation T2 in Fig. 7 involves functional requirements and the categorypartition method. Operational variables are created from steps that have more than one output transition (modeled as an ExecutionOrder element). The outputs of the steps generate the different partition. Again, combinations may be calculated using several criteria, from calculating all possible solutions or calculate just a subset. Relation T3 (Fig. 7 ) combines both techniques results. Test scenarios and combinations of operational variables merge using test cases. 4.3 Application On one hand, previous metamodels and transformations do not impose a concrete representation of the involved elements (functional requirements, test scenarios, operational variables, and test cases), but on the other hand, working directly with the metamodels object may be diffi cult, as shown in Figs. 8 and 9 . NDT does not impose a concrete syntax for requirements, allowing the defi nition of use cases by means of either a model defi ned in UML or a text template. As it can be observed in Fig. 8 , several concrete syntaxes may be used for defi ning functional requirements. The “?” indicates that any other syntaxes or formats plus the indicated one may be used. Thus, the fi rst step to apply the generation of functional test cases from functional requirements deals with defi ning a process for extracting a functional requirement model in accordance with the metamodel introduced in Sect. 4.1 . This process depends on the specifi c syntax and it is out of the scope of this paper. Some previous work in this line was published in [ 14 ]. T1 T2 T3 Fig. 7 Transformations among models ? Use cases (concrete notation)Functional requirement metamodel UC-01. Añadir nuevo enlace. No 1 El visitante solicita añadir un nuevoenlace. 2 El sistema solicita la información del enlace(SR-02). 3 El visitanteintroduce la información del nuevo enlace. 4 El sistema almacena el nuevoenlace. 3.1.i El visitantecancela la operaci termina. 3.2.i Si el visitantedesea cambiar la categoría caso de uso “Cambiar categoría” y este caso de uso continua. 4.1.p Si el nombre del enlace o su URL est án vacíos la información del enlace. Nuevo enlace añadido al sistema. Por defecto, el sistema selecciona la categoría “Top” enlace. Nombre Precondición Post-condición Notas Secuencia principal Error/ Secuencias alternatives Fig. 8 Concrete syntaxes for functional requirements model