A general approach to Software Product Line testing

A general approach to Software Product Line testing Elvira G. Ruiz1, Jon Ayerdi2, José A. Galindo1, Aitor Arrieta2, Goiuria Sagardui2, and David Benavides1 1Universidad de Sevilla, Dept. Lenguajes y Sistemas Informáticos, Av. Reina Mercedes s/n Sevilla - España, {egruiz, jagalindo, benavides}@us.es 2Mondragon Unibertsitatea, Dept. de Electrónica e Informática, Goiru 2, Mondragon - España [email protected], [email protected], [email protected] Abstract. Variability is a central concept in Software Product Lines (SPLs). It has been extensively studied how the SPL paradigm can improve both the efficiency of a company and the quality of products. Nevertheless, this brings several challenges when testing an SPL, which are mainly caused by the potentially huge amount of products that can be derived from an SPL. Different studies proposing methods for testing SPLs exist. Furthermore, there are secondary studies reviewing and mapping the literature of the existing proposals. However, there is a lack of systematic guidelines for practitioners and researchers with the different steps required to perform a testing strategy of an SPL. In this paper, we present a first preliminary version for a tutorial that summarizes the existing proposals of the SPL testing area. To the best of our knowledge, there is no similar attempt in existing literature. Our goal is to discuss this tutorial with the community and enrich it to provide a more solid version of it in the future. Keywords: Software product lines, Software testing, Software reusability. 1 Introduction Software product lines and variability intensive systems benefits from a set of techniques, tools and methods that are used to develop a set of different products that share some commonalities [26]. The concrete functionality that varies across products in the SPL is encapsulated using an abstraction known as feature. Feature models are used to encode common and varying parts of SPLs [17]. In the literature, we find real examples encoding a large number of products. For example, the Linux Kernel [24] with more than 6,000 features or Debian packaging systems [11] with more than 27,000. The large amount of products that an SPL can encode, makes its analysis a time–consuming and error prone task. Then, researchers proposed the use of automated analysis techniques [5] for a set of activities in which testing is usually one of the most relevant [12]. SPL testing represents a new challenge for software testing practitioners and researchers [23]. When testing SPL, each product shares some common functionality with one or more products, while differing in at least one feature. SPLs add testing complexity because they require testing a set of products rather than a single product. These products, however, share common functionality or artifacts, enabling the reuse of some tests across the entire SPL. According to [23], several strategies can be used to test SPL products. These testing strategies can be summarized as follows: i)testing product by product, ii)incremental testing, and iii)reusable asset instantiation. Testing product by product is a strategy that tests all products one by one, as if they were not part of an SPL. With this strategy the test process covers all possible interactions between features but grows exponentially in cost as a function of the number of features in the SPL. Incremental testing is a strategy that starts by testing the first developed product and creates new unit tests for each new feature added. Using this strategy, the commonalities in the SPL are exploited to reduce testing effort. However, when a new feature is introduced, all the interactions between the new feature and the old ones have to be tested, which can be challenging for large SPLs. Reusable asset instantiation relies on data captured in the domain analysis stage of SPL creation to develop a set of abstract test cases that cover all features (but not necessarily configurations) in the SPL. These abstract tests cases are mapped to concrete requirements in the application engineering stage. These last two testing strategies are designed to reduce the SPL combinatorial explosion in testing cost as a function of the feature count. Within SPL engineering, two different processes can be distinguished: (1) domain engineering and (2) application engineering. Domain engineering is the process of developing the platform for building products and defining the commonalities and the variability of the product line. Application engineering is the process of deriving specific applications by using the platform defined in domain engineering and binding the variability to satisfy the needs of each particular application [22]. In [15] authors propose an ideal path to follow when it comes to SPL testing, which is a W testing model for SPLs that considers component, integration and system testing for both domain and application engineering. This paradigm maps every sub-process to either domain or application engineering. However, and due to the complexity of the testing processes when variability is considered, in many software projects there is not such a clear division of tasks. It is possible to find some testing processes (i.e., component testing) that can be started in the domain engineering phase and continued during the application engineering phase – In fact, it is recommended to adapt the paradigm to the necessities of each SPL. To the best of our knowledge, there is a lack of systematic guidelines that prevents the practitioners from these peculiarities. Motivated by this, we have formulated a first approach to what could be a flexible – yet still systematic – approach. The approach is based on the principles stated by [23], and completed 2 with different strategies stated of the SPL testing literature, like mappings and reviews [8,10,28]. Our goal is to begin a discussion around the model with the community in order to enrich it and provide a solid version that can be used as a reference for new and senior SPL developers. The remainder of this paper is structured as follows: Section 2 presents the information regarding the software developed for this article. Section 3 presents background information on different SPL theoretical testing approaches. Our proposed approach is described in Section 4 and detailed in Section 5. Finally in Section 6 we present concluding remarks and lessons learned. 2 Running example Online Shop Catalog Payment Security Product Search Bank Account E-coins Credit Card High Low {1..3} Credit Card implies High Mandatory Optional Or Alternative Fig. 1. Online shop feature model [29]. Figure 1 shows the feature model of the online shop example that we will use throughout this paper. It is a configurable online shop system with different capabilities which are the system’s variability points, as proposed in [29]. One of the most common methods for modelling variability in industry consist in using feature models [6], in which variability points are mapped into features and then represented in a hierarchical diagram that depicts the relationships between features. The feature model from Figure 1 shows that all online shops must have a catalog listing all the available products, a set of payment methods, and a security level. Furthermore, an online shop can optionally have a search feature which allows users to find products more easily. Further down the hierarchy, we can see that there are three possible payment methods, at least one of which needs to be selected: bank account, e-coins, and credit card. Note that the {1..3} cardinality annotation is redundant in this case, since the or parent-child relationship requires that at least one or more sub-features are selected. Finally, 3 the security level of the online shop must be either high or low, since the alternative parent-child relationship mandates that exactly one of the sub-features has to be selected. In addition to the parental relationships between features, feature models may also have additional cross–tree constraints, which are propositional formulas that further reduce the amount of valid configurations. In our example, the “Credit Card implies High” constraint makes the feature High to be mandatory when the feature Credit Card is selected. Taking all of this into account, we can derive a total of 20 valid online shop variants. For more information about feature modeling theory, refer to [4,5]. Order Summary Start Product Search Catalog toCatalog addToCart viewProductDetails E-coins E-coins Product Details Payment Choice Bank Account Bank Account Credit Card Credit Card Payment Validation Product Selection Cart Content Checkout Feature Links Product Search viewProductDetails toCatalog searchProduct searchProduct viewCartContent toCatalog cancelOrder validPayment invalidPayment toPaymentChoice removeProductFromCart viewOrderSummary selectBankAccount selectECoins selectCreditCard validatePayment validatePayment validatePayment toCatalog Fig. 2. Online shop 150% model. One way of modeling the behavior of a software system is by employing a state machine model. This model could also be used to generate test cases if we were using Model-Based Testing (MBT) techniques. In the case of SPLs, the so-called 150% model can be built, which is a domain engineering asset that represents the behavior of the whole product line. 150% models integrate all the variability, i.e., the variability related to the whole product line into one single model [2]. When a specific product variant is selected, the variability of the 150% model is bound, forming the 100% model (i.e., the model specific to that configuration) [2]. This approach is not only considered for models but can also be used for generic code. Figure 2 shows the 150% state machine model of the online shop example, where the links to the features from the feature model have been represented with colors. Note that in many cases the 150% model 4 itself may not be a valid product variant, since it is not always valid to select all the features on a single product. Analogously to the 150% state machine model shown, other domain engineering assets can be generated, such as use case diagrams, class diagrams, and even the software source code itself. These assets should also be linked to the feature model so that they can be automatically reused for different products. In order to demonstrate how to manage an SPL project, we have implemented a simple, Java based version of our online shops example using FeatureIDE [27], a tool which can be used to develop SPLs using the feature-oriented software development paradigm. 3 Top-down vs Bottom-up approach Product-Specific Test Suites Test Execution Top-Down Test Generation Product Selection Feature model Reusable Components Domain Engineering Assets Application Engineering Assets Products Domain Engineering Application Engineering Fig. 3. SPL testing Top-Down approach. Product Line Test Suite Product-Specific Test Suites Test Execution Bottom-Up Test Generation Product Selection Feature model Reusable Components Domain Engineering Assets Application Engineering Assets Products Domain Engineering Application Engineering Fig. 4. SPL testing Bottom-Up approach. There are two main approaches at the SPL testing: (1) the top-down approach and (2) the bottom-up approach [29], which are also referred to as productcentered and product line-centered respectively in some publications [19]. On the one hand, the top-down approach consists in selecting and generating the desired product variants first, and then generating a test cases for each derived product individually. On the other hand, the bottom-up approach consists in generating a database of generic test cases for the whole SPL based on the domain engineering assets, such as the 150% model. Later, variability of these test cases is bound to test individual product variants. Figures 3 and 4 show an overview of these approaches. 5 toCatalog, searchProduct, viewProductDetails, searchProduct, toCatalog, view- ProductDetails, addToCart, toCatalog, viewCartContent, removeProductFrom- Cart, toCatalog, viewCartContent, viewOrderSummary, cancelOrder, toCatalog, viewCartContent, viewOrderSummary, toPaymentChoice, (selectBankAccount OR selectECoins OR selectCreditCard), validatePayment, validPayment Example 1. Bottom-up test case for online shops. We define a test case as a specification of inputs used for software testing. In this paper, they will consist of sequences of events that trigger transitions in our example state machine model. As an example of the bottom-up approach, if we wanted to generate a product line test suite to obtain a high transition coverage for the online shops example, we could come up with the test case shown in Example 1, where the colors represent the same feature links as in the Figure 2 model. toCatalog, searchProduct, viewProductDetails, searchProduct, toCatalog, view- ProductDetails, addToCart, toCatalog, viewCartContent, removeProductFrom- Cart, toCatalog, viewCartContent, viewOrderSummary, cancelOrder, toCatalog, viewCartContent, viewOrderSummary, toPaymentChoice, selectECoins, validatePayment, validPayment Example 2. Derived bottom-up test case for online shops. After product selection, we can derive this test case into product-specific test cases for every product, allowing us to reuse it. Note that this test case only covers one of the payment methods, even if multiple are selected. If, for instance, we selected a product with {P roductSearch, ECoins, LowSecurity}, its derived product-specific test case would be Example 2. As for the top-down approach, the product-specific test suite will be generated after selecting the product variants, so we can just leverage existing software testing techniques and tools. It has been observed by some authors that the bottom-up approach seems to scale better than the top-down approach in terms of test execution cost (considering the total test case count, number of steps and number of configurations) [19]. Furthermore, considering that the top-down approach is similar to the traditional testing system, this paper will be more focused in the bottom-up approach. 4 Proposed Process Overview This proposal is based on the strategy Design test assets for reuse [8]. This means that test plans and test cases are created as soon as possible, usually in domain engineering. Nevertheless, application engineering tests are still needed, 6 so it is also important to encourage the reuse of those product-specific test cases defined in application engineering from one to another product. Interaction Testing Product Testing Evolution Testing Test Optimization (Optional) Test Execution Feature Model Consistency Checking Asset Testing Product Line Test Generation Domain Engineering Application Engineering Product Line Test Generation Product Line Test GenerationSampling and Prioritization Fig. 5. SPL testing process timeline proposal. Figure 5 shows an overview of the activities involved in the testing process of an SPL and their mapping into domain or application engineering. It is possible to divide the SPL testing workflow in four different procedures: Asset Testing, Interaction Testing,Product Testing and Evolution Testing. These procedures are sorted in chronological order in Figure 5, but in high variability environments it is usually impossible to avoid mixing them. One example is how in [23] it is described how it is not necessary to develop and test every asset before starting to test a product by applying testing prioritization. There are SPL testing processes that are affected and modified by all of the aforementioned procedures. They are represented by white boxes in Figure 5, and the arrows represent the dependency between them. In order to achieve an optimal test execution firstly we need to ensure that every feature selected (or added to the SPL due to latter product needs) is consistent with the original feature model – Feature model consistency checking [1]. After that, prioritization of tests should be established for every single procedure. Most times, and because of the variability of the SPL, sampling techniques have to be applied in order to achieve good coverage of interaction testing between assets. This is referred to as Sampling and Prioritization. 7 Once we have a clear view of which interactions and processes should be prioritized on the SPL testing, a generic test plan is built. Named as Product Line Test Generation, this process is heavily influenced by both domain and application engineering, and depends on the already tested assets. It is essential to unify and reuse test assets (test cases, test scenarios and test results) as much as possible from one product to another. To achieve this, product-specific created test assets are stored in the generic SPL testing plan as they appear, as this has been proven to impact the effort reduction [8]. Finally, Test Optimization can optionally be performed on application engineering to achieve an optimal Test Execution for every procedure. 5 Procedures In this section, every sub-process from Figure 5 is define. Furthermore, some glimpses about different techniques every process are given. 5.1 Asset Testing Assets are defined as artifacts built for the development of different products of the same SPL [3]. In the literature they are also referred to as domain artifacts or core assets. These assets can be tested independently through unit testing making use of traditional software techniques [8]. Results from these tests are valuable for every product derived from a SPL. This process is usually performed in domain engineering, although there are some assets that cannot be tested until there is a product [23]. Depending on the perspective of the SPL development, different approaches can be made. According to [23], it is highly recommended to test commonalities first. Commonalities are those assets considered core in the SPL and that will be present in every product. This will be useful in case that a reference application [21] wants to be used at the interaction testing procedure. Once commonalities are tested, the testing of variability-affected assets can begin. In order to achieve an optimal coverage of interactions, there can be a prioritization of which variable assets need to be tested first. This will allow interaction testing to start earlier in the testing process. Example 3 shows a test case for the ProductDetails class in our online shops example, which checks the presence or absence of the ProductSearch option in the user menu. Even if ProductDetails is a core asset that is always present, this particular test case cannot be executed until a product is selected because the transition to ProductSearch may or may not exist. 5.2 Interaction Testing Asset testing is not enough for achieving a high quality SPL. In SPLs, interaction between assets causes failures, bugs and inconsistencies that can only be 8 1public void testSearchTransition() { 2// Select menu option 3input . println ("2"); // 2. BACK TO CATALOG 4// Run ProductDetails 5productDetails.run(); 6List < String > lines = output . lines () . collect () ; 7// Check output 8boolean searchProductPresent = false ; 9for ( String line : lines ) { 10 if( line . equals (" 3. SEARCH PRODUCT ")) { 11 assertFalse(searchProductPresent); 12 searchProduct = true; 13 } 14 } 15 // #if ProductSearch 16 assertTrue(searchProductPresent); 17 // #else 18 assertFalse(searchProductPresent); 19 // # endif 20 } Example 3. Asset test for the ProductDetails class. detected when certain feature combinations are present. For this reason, interaction testing is used when testing SPLs. Variability is controlled on this stage of SPL testing [7], and results will be valuable for every different products derived from the SPL. Interaction testing includes integration testing – which belongs to domain engineering [8] – and also binding testing [23], which belongs to application engineering. In high variability scenarios it is impossible to achieve a full coverage of every interaction between assets. This is caused by the number of products an SPL can have, that grows exponentially as the number of features increases. Therefore, sampling is necessary to test as many different interactions as possible while avoiding exhaustive testing [28]. The idea behind sampling is to derive a subset of all possible products that collectively cover the behavior of the SPL and reveal most of the faults by only them [28]. To sample a product subset from the entire SPL, several approaches have been proposed. Varshosaz et al., proposed a taxonomy to classify these approaches [28]. This taxonomy included (1) input data to the sampling approach (e.g., feature model), (2) type of algorithm used for sampling products (e.g., from simple greedy-based algorithm to more sophisticated population-based algorithms) and (3) type of coverage employed (e.g., feature-interaction coverage). Out of the scope of this paper, the classification also included the evaluation technique and the type of application of the approach, among which most of them were focused on testing. 9