FaMa Test Suite v1.2: ISA Technical Report ISA-10-TR-01

FaMa Test Suite v1.2 Sergio Segura, David Benavides and Antonio Ruiz-Cort´es {sergiosegura,benavides,aruiz}@us.es Applied Software Engineering Research Group University of Seville, Spain March 2010 Technical Report ISA-10-TR-01 This report was prepared by the Applied Software Engineering Research Group (ISA) Department of computer languages and systems Av/ Reina Mercedes S/N, 41012 Seville, Spain http://www.isa.us.es/ Copyright c 2010 by ISA Research Group. Permission to reproduce this document and to prepare derivative works from this document for internal use is granted, provided the copyright and ’No Warranty’ statements are included with all reproductions and derivative works. NO WARRANTY THIS ISA RESEARCH GROUP MATERIAL IS FURNISHED ON AN ’AS-IS’ BASIS. ISA RESEARCH GROUP MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTIBILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademark holder Support: This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT project SETI (TIN2009-07366) and the Andalusian Government project ISABEL (TIC-2533). List of changes Version Date Description 1.0 February 2009 First release 1.1 December 2009 Refined evaluation with mutants. New test cases added 1.2 March 2010 Test cases of the operation ValidProduct updated. 3 FaMa Test Suite v1.2 Sergio Segura, David Benavides and Antonio Ruiz-Cort´es Department of Computer Languages and Systems Av Reina Mercedes S/N, 41012 Seville, Spain {sergiosegura, benavides, aruiz}@us.es Abstract: A Feature Model (FM) is a compact representation of all the products of a software product line. Automated analysis of FMs is rapidly gaining importance: new operations of analysis have been proposed, new tools have been developed to support those operations and different logical paradigms and algorithms have been proposed to perform them. Implementing operations is a complex task that easily leads to errors in analysis solutions. In this context, the lack of specific testing mechanisms is becoming a major obstacle hindering the development of tools and affecting their quality and reliability. In this paper, we present FaMa Test Suite, a set of implementation–independent test cases to validate the functionality of FM analysis tools. This is an efficient and handy mechanism to assist in the development of tools, detecting faults and improving their quality. In order to show the effectiveness of our proposal, we evaluated the suite using mutation testing as well as real faults and tools. Our results are promising and directly applicable in the testing of analysis solutions. We intend this paper to be a first step toward the development of a widely accepted test suite to support functional testing in the community of automated analysis of feature models. Key Words: Test suite, functional testing, feature models, automated analysis, software product lines 1 Introduction Software Product Line (SPL) engineering is an approach to develop families of related systems based on the usage of reusable assets as a means to improve software quality while reducing production costs and time–to–market [1]. Products in software product lines are specified in terms of features. A feature is defined as an increment in product functionality [2]. Key to SPLs is to capture commonalities (i.e. common features) and variabilities (i.e. variant features) of the systems that belong to the product line. To this aim, feature models are commonly used. A feature model [3, 4] is a compact representation of all the products of a product line in terms of features and relationships among them (see Figure 1). The automated analysis of feature models deals with the automated extraction of information from feature models. Typical operations of analysis allow determining whether a feature model is void (i.e. it represents no products), whether it contains errors (e.g. features that cannot be part of any product) or what is the number of products of the SPL represented by the model. Catalogues with up to 30 different analysis operations on feature models have been reported in the literature [5, 6]. Analysis solutions can be categorized in those using propositional logic [7, 8, 9, 10, 11, 12, 13], constraint programming [14, 15, 16], description logic [17, 18] and ad-hoc algorithms and techniques [19, 20, 21]. There are also a number of tools supporting these analysis capabilities such as AHEAD Tool Suite [22], FaMa Framework [23], Feature Model Plug-in [24] and pure::variants [25]. The implementation of analysis operations is a hard task involving complex data structures and algorithms. This makes the development of analysis tools far from trivial and easily leads to errors increasing development time and reducing their reliability. Gaining confidence in the absence of defects in these tools is essential since the information extracted from feature models is used to support decisions all along the SPL development process [2]. However, the lack of specific testing mechanisms in this context appears as a major obstacle for engineers when trying to assess the functionality and quality of their programs. Hence, it is known that software testing accounts for about 50% of the total cost of software development [26]. Software testing intends to reveal faults in the software under test [27, 28]. This is mainly done by checking the behaviour of the program with set of input-output combinations (i.e. test cases). The main challenge when testing is to find a balance between the number of test cases and their 1 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 effectiveness [28]. Hence, trying to be exhaustive when testing software tools may easily increase the number of test cases to an unmanageable level. Similarly, using a reduced number of input-output combinations may result in a weak effectiveness. In this paper, we present a set of implementation–independent test cases to validate the functionality of feature model analysis tools. Through the implementation of our test cases, faults can be rapidly detected assisting in the development of feature model analysis tools and improving their reliability and quality. For its design and evaluation, we used popular techniques from the software testing community to assist us on the creation of a representative set of input-output combinations. These test cases can be used either in isolation or as a suitable complement for further testing methods such as white–box testing techniques [27, 29] or automated test data generators [30]. As suggested by the testing literature, each test case was designed to reveal a single type of fault. This allows users to identify clearly the source of a fault once it has been detected. Briefly, we next describe the main characteristics of our suite: – Operations tested. Current version of our suite, called FaMa Test Suite, addresses 7 out of 30 analysis operations on feature models identified in the literature [5]. These were selected for their extended use in the community of automated analysis and their heterogeneous nature. – Testing techniques. Four black-box testing techniques were used to design test cases, namely: equivalence partitioning, boundary-value analysis, pairwise testing and error guessing. A preliminary evaluation of the testing techniques to be used in our approach was presented in [31]. – Test cases. The suite is composed by 192 test cases. Each test case is designed in terms of the inputs (i.e. feature models and some other parameters) and expected outputs of the analysis operations under test. – Adequacy. We evaluated the effectiveness of our suite using mutation testing and real faults as follows. Firstly, we generated hundreds of faulty versions (so-called mutants) of three open source analysis tools integrated into the FaMa framework. Then, we executed our suite against those faulty tools and check how many faults were detected by our test cases. As a result, the suite identified 96.1% of the faults showing the feasibility of our proposal. We then refined our suite until obtaining a score of 100%. Our refined suite also showed to be effective in detecting motivating faults found in the literature and in a recent release of the FaMa framework. The remainder of the paper is structured as follows: Section 2 presents feature models, their analyses and black-box testing techniques in a nutshell. A detailed description of how we designed our test cases is presented in Section 3. Section 4 describes the adequacy evaluation and refinement of the suite. A summary of the refined suite and a brief discussion is presented in Section 5. Finally, we summarize our main conclusions and describe our future work in Section 6. 2 Preliminaries This section provides an overview of feature models, their analysis and the black–box testing techniques used in our approach. 2.1 Feature models A feature model [3, 4] represents all the products of an SPL in a single model in terms of features and relationships among them. It is organized hierarchically and is graphically depicted as a feature diagram. Figure 1 shows a simplified example of a feature model representing an e-commerce SPL. The model illustrates how features are used to specify the commonalities and variabilities of the on-line shopping systems that belong to the product line. The root feature (i.e. E-Shop) identifies 2 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 E-Shop Search Basic Advanced Catalogue Info Image Price Description Security Medium High Payment PC Bank Draft Offers Mobile GUI Credit Card Visa American Express Mandatory Optional Alternative Or Requires Excludes Banners Figure 1: A sample feature model the SPL. The relationships between a parent feature and its child features can be mainly divided into: Mandatory. A child feature has a mandatory relationship with its parent when the child is included in all products in which its parent feature appears. For instance, every on-line shopping system in our example must implement a catalogue of products. Optional. A child feature has an optional relationship with its parent when the child can be optionally included in all products in which its parent feature appears. For instance, banners is defined as an optional feature. Alternative. A set of child features have an alternative relationship with their parent when only one feature of the children can be selected when its parent feature is part of the product. In our SPL, a shopping system may implement high or medium security policy but not both in the same product. Or-Relation. A set of child features have an or-relationship with their parent when one or more of them can be included in the products in which its parent feature appears. A shopping system can implement several payment modules: bank draft,credit card or both of them. Notice that a child feature can only appear in a product if its parent feature does. The root feature is a part of all the products within the SPL. In addition to the parental relationships between features, a feature model can also contain cross-tree constraints between features. These are typically of the form: Requires. If a feature A requires a feature B, the inclusion of A in a product implies the inclusion of B in such product. On-line shopping systems accepting payments with credit card must implement a high security policy. Excludes. If a feature A excludes a feature B, both features cannot be part of the same product. Shopping systems implementing a mobile GUI cannot include support for banners. Feature models were first introduced as a part of the FODA (Feature-Oriented Domain Analysis) method back in 1990 [4]. Since then, feature modelling has been widely adopted by the software product line community and a number of extensions have been proposed in attempts to improve properties such as succinctness and naturalness. We refer the reader to [6] for a detailed survey on 3 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 the different feature modelling languages. 2.2 Automated analyses of feature models The automated analysis of feature models deals with the computer–aided extraction of information from feature models [5]. From the information obtained, marketing strategies and technical decisions can be derived [32]. The analysis of a feature model is generally performed in two steps: i) First, the model is translated into a specific logic representation such as a satisfiability problem [7, 8, 9, 10, 11, 12, 13], a constraint satisfaction problem [14, 15, 16] or a knowledge base using description logic [17, 18], ii) Then, off-the-shelf solvers are used to automatically perform a variety of operations on the logic representation of the model. Catalogues with up to 30 different analysis operations on feature models have been reported in the literature [5, 6]. Following, we summarize some of the analysis operations we will refer through the rest of the paper. Determining if a feature model is void. This operation takes a feature model as input and returns a value informing whether such feature model is void or not [6, 7, 8, 9, 10, 11, 12, 13, 14, 17, 18, 19, 20, 33]. A feature model is void if it represents no products. Finding out if a product is valid. This operation checks whether an input product (i.e. set of features) belongs to the set of products represented by a given feature model or not [6, 7, 8, 9, 10, 14, 16, 18, 33]. As an example, P={E-Shop, Catalogue, Info, Description, Security, Medium, GUI, PC}is not a valid product of the product line represented by the model in Figure 1 because it does not include the mandatory feature Payment. Obtaining all products. This operation takes a feature model as input and returns all the products represented by the model [7, 9, 10, 14, 19, 20, 33]. Calculating the number of products. This operation returns the number of products represented by a feature model [8, 10, 14, 19, 20, 21, 33]. The model in Figure 1 represents 2016 different products. Calculating variability. This operation takes a feature model as input and returns the ratio between the number of products and 2n−1 where n is the number of features in the model [14, 33]. This operation may be used to measure the flexibility of the product line. For instance, a small factor means that the number of combinations of features is very limited compared to the total number of potential products. In Figure 1, Variability = 0.00048. Calculating commonality. This operation takes a feature model and a feature as inputs and returns a value representing the proportion of valid products in which the feature appears [14, 21, 33]. This operation may be used to prioritize the order in which the features are to be developed and can also be used to detect dead features [15]. In Figure 1, Commonality(Banners) = 25%. Dead features detection. This operation takes a feature model as input and returns a set of dead features (if any) [2, 8, 11, 15, 33, 20, 13]. A dead feature is a feature that never appears in any of the products represented by the feature model [15]. These are caused by cross-tree constraints. Some common cases of dead feature are presented in Figure 4 (Section 3.2.1). Some commercial and open source tools supporting the analysis of feature models are the AHEAD Tool Suite [22], FaMa Framework [23], Feature Model Plug-in [24] and pure::variants [25]. 4 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 2.3 Black-box testing techniques Software testing is performed by means of test cases. A test case is a set of test inputs and expected outputs developed to verify compliance with a specific requirement [29, 27]. Test cases may be grouped into so-called test suites. A variety of testing techniques has been reported to assist on the design of effective test cases, i.e. those that will find more faults with less effort and time [26, 27, 29]. In this paper, we will focus on the so called black-box testing techniques. These techniques focus on checking whether a program does what it is supposed to do based on its specification [27]. No knowledge about the internal structure or implementation of the software under test is assumed. We will refer to four of these techniques along the paper. Briefly, these are: Equivalence partitioning. This technique is used to reduce the number of test cases to be developed while still maintaining a reasonable test coverage (i.e. the degree to which the test cases verifies the test requirements) [27, 29]. In this technique, the input domain of the program is divided into partitions (also called equivalence classes) in which the program is expected to process the set of data input in a similar (i.e. equivalent) way. According to this testing approach, only one or a few test cases of each partition are needed to evaluate the behaviour of the program for the corresponding partition. Thus, selecting a subset of test cases from each partition is enough to test the program effectively while keeping a manageable number of test cases. Boundary value analysis. This technique is used to guide the tester when selecting inputs from equivalence classes [27, 29]. According to this technique, programmers usually make mistakes with the inputs values located on the boundaries of the equivalence classes. This method guides the tester to select those inputs located on the “edges” of the equivalence partitions. Pairwise testing (also called 2-wise testing). This is a combinatorial software testing method focusing on testing all possible discrete combinations of two input parameters [29, 34]. According to the hypothesis behind this technique, most common errors involve one input parameter. The next most common category of errors consists of those dependent on interactions between pairs of parameters and so on. Thus, the main goal of this technique is to address the second most common cases of errors (those involving two parameters) while keeping the number of test cases in a manageable level. Error guessing. This is a software testing technique based on the ability of the tester to predict where faults are located according to its experience on the domain [29]. Using this technique, test cases are specifically designed to exercise typical error-prone points related to the type of system under test. 3 Test suite design In this section, we describe how we designed the test cases that compose our suite. These mainly are input-output combinations specifically created to reveal failures in the implementations of analysis operations on feature models. Creating test cases for every possible permutation of a program is impractical and very often impossible; there are simply too many input-output combinations [27]. Thus, as recalled by Pressman [28], the objective when testing is to “design tests that have the highest likelihood of finding most errors with a minimum amount of time and effort”. To assist us in the process, we evaluated a number of techniques reported in the literature [27, 29]. We focused on black-box techniques since we want our test cases to rely on the specification of the analysis operations rather than on specific implementations. In particular, we found the four techniques described in Section 2.3 to be effective and generic enough to be applicable to our domain. 5 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 For the design of the suite we followed four steps, namely: i) identification of the inputs and outputs of the analysis operations, ii) selection of representative instances of each type of input, iii) combination of previous instances in those operations receiving more than one input parameter, and iv) test cases report. Following, we detail how we carried out these steps. 3.1 Identification of inputs and outputs The current version of our suite addresses 7 out of 30 analysis operations on feature models identified in the literature [5] (detailed in Section 2.2). We selected these operations for its extended use in the community of automated analysis and their heterogeneous nature. In order to identify the type of the input/output parameters of the operations and avoid misunderstandings when interpreting their semantics, we used the formal definition of the operations proposed by Benavides [32]. Table 1 summarizes the operations in terms of their inputs and outputs. For the sake of simplicity, we assign an identificator to each operation to refer them along the paper. As illustrated, inputs are composed of feature models, products and features. Outputs mainly comprise collections of products and features together with numeric and boolean values. ID operation Operation Inputs Output VoidFM Void FM FM Boolean value ValidProduct Valid product FM, Product Boolean value Products Products FM Collection of products #Products Number of products FM Numeric value Variability Variability FM Numeric value Commonality Commonality FM, Feature Numeric value DeadFeatures Dead features FM Collection of features Table 1: Analysis operations addressed in the suite The feature modelling notation used in our suite corresponds to one showed in Section 2.1 (hereinafter referred as basic feature models). We selected this notation for its simplicity and extended use in current feature model analysis tools and literature. 3.2 Inputs selection In this section, we explain how we selected the inputs to be used in our test cases. For each type of input (i.e. feature models, products and features), we next describe the techniques used and how we applied them. 3.2.1 Feature models We found two testing techniques to be helpful for the selection of a suitable set of input feature models, namely: equivalence partitioning and error guessing. We applied them as follows: Equivalence partitioning. The potential number of input feature models is limitless. To select a representative set of these, we propose dividing input feature models into equivalence classes according to the different types of relationships and constraints among features, i.e. mandatory, optional, or, alternative, requires and excludes. This is a natural partition intuitively used in most proposals when defining the mapping from a feature model to a specific logic paradigm (e.g. constraint satisfaction problem) [7, 9, 10, 13, 14, 15, 18, 32]. Therefore, according to this technique, if a feature model with a single mandatory relationship is correctly managed by an operation, we could assume that those with more than one mandatory relationship would also be processed successfully. 6 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 Reasoner Mutants Equivalent Discarded Sat4jReasoner 262 27 47 JavaBDDReasoner 302 28 37 JaCoPReasoner 185 46 3 Total 749 101 87 Table 4: Mutants generation results VoidFM and ValidProduct produced the lowest scores and higher number of alive mutants. We found that mutants on these operations required input models to have a very specific pattern in order to be killed and therefore were harder to detect than mutants in the rest of operations. Average mutation scores in the three reasoners ranged between 94.4% and 100%. In total, our test suite was able to kill 1390 (96.1%) out of the 1445 mutants executed showing the effectiveness of the suite. Operation Sat4jReasoner JavaBDDReasoner JaCoPReasoner Mutants Alive Score Mutants Alive Score Mutants Alive Score VoidFM 55 20 63.6 75 12 84.0 8 0 100 ValidProduct 109 4 96.3 129 7 94.6 61 0 100 Products 86 1 98.8 130 2 98.5 37 0 100 #Products 57 1 98.2 77 2 97.4 13 0 100 Variability 82 1 98.8 104 2 98.1 36 0 100 Commonality 109 1 99.1 131 2 98.5 66 0 100 DeadFeatures - - - - - - 80 0 100 Total 498 28 94.4 646 27 95.8 301 0 100 Table 5: Mutants execution results 4.1.3 Refinement As shown in previous sections, mutation testing is an effective means to measure the effectiveness of a test suite. However, information provided by mutation testing can also be used to guide the creation of new test cases that kill the remaining alive mutants and strengthen the final test suite [41]. Following this approach, we designed a number of test cases to kill remaining undetected mutants until obtaining a score of 100% in the three FaMa reasoners. A total of 27 new test cases were created and executed. For instance, alive mutants guided us to the creation of a couple test cases to ensure that alternative relationships are not processed as or-relationships and vice–versa (e.g. test cases VM-21 and VM-22 in Appendix A). Out of the 27 test cases created, we selected those test cases that showed to be effective in killing mutants in at least two of the three subject reasoners and added them to our suite. As a result, 10 test cases were added to the initial test suite increasing the number of these until 190. 4.2 Evaluation using real faults For a further evaluation of our approach, we checked the effectiveness of our tool in detecting real faults. In particular, we first studied a motivating fault found in the literature. Then, we used our test suite to test the release 1.0 alpha of the FaMa framework, detecting one defect. These results are next reported. 13 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 4.2.1 Motivating fault found in the literature Consider the work of Batory in SPLC’05 [7], one of the seminal papers in the community of automated analysis of feature models. The paper included a bug (later fixed3) in the mapping of a feature model to a propositional formula. We implemented this wrong mapping into a mock reasoner for FaMa and checked the effectiveness of our approach in detecting the fault. Figure 7 illustrates an example of the wrong output caused by the fault. This manifests itself in alternative relationships whose parent feature is not mandatory making reasoners to consider as valid product those including multiple alternative subfeatures and excluding the parent feature (P3). As a result, the set of products returned by the tool is erroneously larger than the actual one. For instance, the number of products returned by our faulty tool when using the model in Figure 1 as input is 3,584 (instead of the actual 2,016). Note that this is a motivating fault since it can easily remain undetected even when using an input with the problematic pattern. Hence, in the previous example (either with “security” feature as mandatory or optional), the mock tool correctly identifies the model as non void (i.e. it represents at least one product), and so the fault remains latent. Security Medium High P1={Security,High} P2={Security,Medium} P3={High,Medium} Figure 7: Wrong set of products obtained with the faulty reasoner We implemented our test cases using JUnit and tested our faulty tool. The fault was detected by our test suite in the operations VoidFM,Product,#Products,Variability and Commonality remaining latent in the operations ValidProduct and DeadFeatures. These two operations required a very specific pattern to reveal the fault not included in the inputs of our test suite. This gives an idea of the complexity of testing in this domain. We found this fault sufficiently motivating to extend our suite with test cases that detect it in all the operations. Thus, we designed two new test cases to detect the fault in the operation ValidProduct and DeadFeatures and added them to our test suite resulting in a total of 192 test cases. 4.2.2 FaMa v1.0 alpha Finally, we evaluated our tool by trying to detect faults in a recent release of the FaMa Framework, FaMa v1.0 alpha. We executed the 192 test cases of our refined test suite. Tests revealed one defect. The fault affected the operations ValidProduct and Commonality in Sat4jReasoner. The source of the problem was a bug in the creation of propositional clauses in the so-called staged configurations, a new feature of the tool. 5 Test suite summary and discussion Table 6 summarizes the general aspects of the refined test suite using the common terms of the IEEE Standard for Software Testing Documentation [35]. For each operation, the number of test cases and the testing techniques used are presented. Global inputs constraints specify constraints that must be true for every input in the set of associated test cases. For the sake of simplicity, 3ftp://ftp.cs.utexas.edu/pub/predator/splc05.pdf 14 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 two main input constraints were imposed, namely: i) input feature models must be syntactically correct (e.g. checking models for conformance to a metamodel), and ii) all input parameters of the analysis operations must be provided. An operation is said to pass the test (so-called pass criteria) when all the test cases associated to that operation are successful. Trying to be exhaustive when testing feature model analyses tools can easily increase the number of test cases to an unmanageable level. To keep a reasonable balance between number of test cases and test coverage, we kept in mind a number of generic recommendations from the testing literature, namely: i) we gave priority to those decisions reducing the number of test cases, ii) we avoided redundancies by designing each test case to reveal a single type of fault, and iii) we designed simple test cases whose output could be worked out manually to avoid test themselves to become error-prone. We remark that the potential users of the suite are every tool supporting the analysis of feature model. This can be automated by simply implementing the test cases in the desired platform and executing them. A complete list of the test cases that compose the suite is reported in Appendix A. To facilitate its implementation, input models used in the test cases are also available in XML format in the FaMa Tool Suite Web site4. Test suite identifier: FaMa Test Suite (v1.2) Operations tested Test cases Techniques used Void FM 24 EP, PT Valid product 63 EP, PT, BVA Products 21 EP, PT Number of products 21 EP, PT Variability 21 EP, PT Commonality 33 EP, PT, BVA Dead features 9 EG Total 192 Global input constraints: - Input FMs must be syntactically correct - All input parameters are required Operation pass criteria: - Pass 100% of associated test cases Table 6: General overview of the FaMa Test Suite (EP: Equivalence Partitioning, PT: Pairwise Testing, BVA: Boundary-Value Analysis, EG: Error Guessing) 6 Conclusions and future work In this paper, we present a set of implementation–independent test cases to validate the functionality of tools supporting the analysis of feature models. Through the implementation of our test cases, faults can be rapidly detected assisting in the development of feature model analysis tools and improving their reliability and quality. These can be used either in isolation or as a suitable complement for further testing methods such as white–box testing techniques or automated test data generators. For its design, we used popular techniques from the software testing community to assist us on the creation of a representative set of input–output combinations. To evaluate its effectiveness, we applied mutation testing on three open source feature model analysis tools integrated into the FaMa framework. These tools use different underlying paradigms and were coded by different developers what provides the necessary heterogeneity for the evaluation. We initially obtained an average mutation score of 96.1% and refined our suite progressively until getting 100%. 4http://www.isa.us.es/fama/?FaMa Test Suite 15 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 Once refined, our suite also showed to be effective in detecting real faults found in the literature and in a recent release of FaMa. Both, the test suite documentation and the inputs models used in the test cases are ready–to–use and available at the Web Site of the FaMa Tool Suite. We intend this paper to be a first effort toward the development of a widely accepted test suite to support functional testing in the community of automated analysis of feature models. Several challenges remain for our future work in two main directions, namely: –We intend to extend our suite with new operations and testing techniques. We also plan to evaluate our suite with other tools and in development scenarios using FaMa and report our experiences to the community. –We also plan to explore the benefits obtained when combining our suite with other automated testing methods. In a previous work [30], we presented an automated test data generator for the analysis of feature models with promising results. It generates random follow–up test cases based on the relations between inputs feature models and theirs expected outputs (so-called metamorphic testing). However, it is known that metamorphic testing produce better results when combined with other test case selection strategies that generate the initial set of test cases. We intend to use our suite to guide the generation of test cases in our automated test data generator and study the gains in efficiency and efficacy. Material The tools, mutants, and test cases used in our evaluation are available at http://www.lsi.us. es/~segura/files/material/fts_1_2/ Acknowledgments We would like to thank the reviewers of the paper as well as Don Batory and Robert M. Hierons whose comments and suggestions helped us to improve the paper substantially. We also thank Jose Galindo for his technical support during the evaluation of our approach. References 1. Clements P, Northrop L. Software Product Lines: Practices and Patterns. SEI Series in Software Engineering. Addison–Wesley; 2001. 2. Batory D, Benavides D, Ruiz-Cort´es A. Automated Analysis of Feature Models: Challenges Ahead. Communications of the ACM. 2006;December:45–47. 3. Czarnecki K, Eisenecker UW. Generative Programming: Methods, Techniques, and Applications. Addison–Wesley; may 2000. ISBN 0–201–30977–7. 4. Kang K, Cohen S, Hess J, Novak W, Peterson S. Feature–Oriented Domain Analysis (FODA) Feasibility Study. SEI; 1990. CMU/SEI-90-TR-21. 5. Benavides D, Segura S, Ruiz-Cort´es A. Automated Analysis of Feature Models 20 Years Later: A Literature Review. Information Systems. 2010;In press. 6. Schobbens P, P Heymans JCT, Bontemps Y. Generic semantics of feature diagrams. Computer Networks. 2007 Feb;51(2):456–479. 7. Batory D. Feature Models, Grammars, and Propositional Formulas. In: Software Product Lines Conference, LNCS 3714; 2005. p. 7–20. 8. Czarnecki K, Kim P. Cardinality-Based Feature Modeling and Constraints: A Progress Report. In: Proceedings of the International Workshop on Software Factories At OOPSLA 2005; 2005. . 9. Gheyi R, Massoni T, Borba P. A Theory for Feature Models in Alloy. In: Proceedings of the ACM SIGSOFY First Alloy Workshop. Portland, United States; 2006. p. 71–80. Available from: http: //alloy.mit.edu/workshop/programme.html. 10. Mannion M, Camara J. Theorem Proving for Product Line Model Verification. In: Software Product- Family Engineering (PFE). vol. 3014 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg; 2003. p. 211–224. Available from: http://www.springerlink.com/content/m0k40djlmmxx8vtp/. 11. Mendon¸ca M, Wasowski A, Czarnecki K. SAT–based analysis of feature models is easy. In: Proceedings of the Sofware Product Line Conference; 2009. . 16 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 12. van der Storm T. Generic Feature-Based Software Composition. In: Software Composition. vol. 4829 of LNCS. Springer; 2007. p. 66–80. 13. Zhang W, Mei H, Zhao H. Feature-driven requirement dependency analysis and high-level software design. Requirements Engineering. 2006 June;11(3):205–220. Available from: http://www. springerlink.com/content/v1648q73m788q71x/. 14. Benavides D, Ruiz-Cort´es A, Trinidad P. Automated Reasoning on Feature Models. LNCS, Advanced Information Systems Engineering: 17th International Conference, CAiSE 2005. 2005;3520:491–503. 15. Trinidad P, Benavides D, Dur´an A, Ruiz-Cort´es A, Toro M. Automated Error Analysis for the Agilization of Feature Modeling. Journal of Systems and Software. 2008;81(6):883–896. 16. White J, Schmidt D, Trinidad DBP, Ruiz-Cort´es. Automated Diagnosis of Product-line Configuration Errors in Feature Models. In: Proceedings of the 12th Sofware Product Line Conference (SPLC’08). Limerick, Ireland; 2008. . 17. Fan S, Zhang N. Feature Model Based on Description Logics. In: Knowledge-Based Intelligent Information and Engineering Systems; 2006. Available from: http://dx.doi.org/10.1007/11893004_145. 18. Wang H, Li YF, un J, Zhang H, Pan J. Verifying Feature Models using OWL. Journal of Web Semantics. 2007 June;5:117–129. Available from: http://dx.doi.org/10.1016/j.websem.2006.11. 006. 19. van Deursen A, Klint P. Domain–Specific Language Design Requires Feature Descriptions. Journal of Computing and Information Technology. 2002;10(1):1–17. 20. van den Broek P, Galvao I. Analysis of Feature Models using Generalised Feature Trees. In: Third International Workshop on Variability Modelling of Software-intensive Systems. No. 29 in ICB- Research Report. Essen, Germany: Universit¨at Duisburg-Essen; 2009. p. 29–35. Available from: http://www.vamos-workshop.net/proceedings/VaMoS_2009_Proceedings.pdf. 21. Fernandez-Amoros D, Heradio R, Cerrada J. Inferring Information from Feature Diagrams to Product Line Economic Models. In: Proceedings of the Sofware Product Line Conference; 2009. . 22. AHEAD Tool Suite. http://www.cs.utexas.edu/users/schwartz/ATS.html;. Accessed November 2009. 23. Benavides D, Trinidad P, Segura S, Ruiz-Cort´es A. FaMa Framework. http://www.isa.us.es/fama/;. 24. Feature Modeling Plug-in. http://gp.uwaterloo.ca/fmp/;. Accessed November 2009. 25. pure::variants. http://www.pure-systems.com/;. Accessed November 2009. 26. Beizer B. Software testing techniques (2nd ed.). New York, NY, USA: Van Nostrand Reinhold Co.; 1990. 27. Myers GJ, Sandler C. The Art of Software Testing. John Wiley & Sons; 2004. 28. Pressman RS. Software Engineering: A Practitioner’s Approach. 5th ed. McGrap-Hill; 2001. 29. Copeland L. A Practitioner’s Guide to Software Test Design. Norwood, MA, USA: Artech House, Inc.; 2003. 30. Segura S, Hierons RM, Benavides D, Ruiz-Cort´es A. Automated Test Data Generation on the Analyses of Feature Models: A Metamorphic Testing Approach. In: International Conference on Software Testing, Verification and Validation. Paris, France: IEEE press; 2010. In press. 31. Segura S, Benavides D, Ruiz-Cort´es A. Functional Testing of Feature Model Analysis Tools. A First Step. In: 5th Software Product Lines Testing Workshop (SPLiT 2008). SPLC’08. Limerick, Ireland; 2008. . 32. Benavides D. On the Automated Analyisis of Software Product Lines using Feature Models. A Framework for Developing Automated Tool Support. University of Seville; 2007. 33. Trinidad P, Ruiz-Cort´es A. Abductive Reasoning and Automated Analysis of Feature Models: How are they connected? In: Third International Workshop on Variability Modelling of Software- Intensive Systems. Proceedings; 2009. p. 145–153. Available from: http://www.vamos-workshop.net/ proceedings/VaMoS_2009_Proceedings.pdf. 34. Grindal M, Offutt J, Andler SF. Combination testing strategies: a survey. Software Testing, Verification and Reliability. 2005;15(3):167–199. Available from: http://dx.doi.org/10.1002/stvr.319. 35. Draft IEEE Standard for software and system test documentation (Revision of IEEE 829-1998); 2007. Available from: http://ieeexplore.ieee.org/xpls/abs\_all.jsp?arnumber=4432350. 36. DeMillo RA, Lipton RJ, Sayward FG. Hints on Test Data Selection: Help for the Practicing Programmer. IEEE Computer. 1978;11(4):34–41. 37. Cetina C, Fons J, Pelechano V. Moskitt Feature Modeler. http://www.pros.upv.es/mfm;. Accessed November 2009. 38. Sat4j. http://www.sat4j.org/;. Accessed November 2009. 39. JavaBDD. http://javabdd.sourceforge.net/;. Accessed November 2009. 40. JaCoP. http://jacop.osolpro.com/;. Accessed November 2009. 41. Smith BH, Williams L. On guiding the augmentation of an automated test suite via mutation analysis. Empirical Software Engineering. 2009;14(3):341–369. 42. Ma YS, Offutt J. Description of Method-level Mutation Operators for Java, http://cs.gmu.edu/ ~offutt/mujava/mutopsMethod.pdf; 2005. Accessed 2/10/2009. 43. JUnit. http://www.junit.org/;. Accessed November 2009. 17 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 A Test cases In this appendix, we present the test cases included in the current version of the FAMA Test Suite. For each operation, associated test cases are detailed. A.1 Operation Void Feature Model ID Description Input Exp. Output Deps. VM-1 Check whether mandatory relationships are correctly managed by the operation. A B Non void VM-2 Check whether optional relationships are correctly managed by the operation. A B Non void VM-3 Check whether or–relationships are correctly managed by the operation. A B C Non void VM-4 Check whether alternative relationships are correctly managed by the operation. A B C Non void VM-5 Check whether ‘requires’ constraints are correctly managed by the operation. A B C Non void VM-2 VM-6 Check whether ‘excludes’ constraints are correctly managed by the operation. A B C Non void VM-2 18 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VM-7 Check whether the interaction between mandatory and optional relationships is correctly processed. A B D C E Non void VM-1 VM-2 VM-8 Check whether the interaction between mandatory and or-relationships is correctly processed. A B E F C D G Non void VM-1 VM-3 VM-9 Check whether the interaction between mandatory and alternative relationships is correctly processed. A B E F C D G Non void VM-1 VM-4 VM-10 Check whether the interaction between mandatory relationships and ‘requires’ constraints is correctly processed. A BC Non void VM-1 VM-5 VM-11 Check whether the interaction between mandatory relationships and ‘excludes’ constraints is correctly processed. A BC Void VM-1 VM-6 VM-12 Check whether the interaction between optional and or-relationships is correctly processed. A B E F C D G Non void VM-2 VM-3 VM-13 Check whether the interaction between optional and alternative relationships is correctly processed. A B E F C D G Non void VM-2 VM-4 19 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VM-14 Check whether the interaction between or- and alternative relationships is correctly processed. A D E H I B C F G Non void VM-3 VM-4 VM-15 Check whether the interaction between orrelationships and ‘requires’ constraints is correctly processed. A B C Non void VM-3 VM-5 VM-16 Check whether the interaction between orrelationships and ‘excludes’ constraints is correctly processed. A B C Non void VM-3 VM-6 VM-17 Check whether the interaction between alternative relationships and ‘requires’ constraints is correctly processed. A B C Non void VM-4 VM-5 VM-18 Check whether the interaction between alternative relationships and ‘excludes’ constraints is correctly processed. A B C Non void VM-4 VM-6 VM-19 Check whether the interaction between ‘requires’ and ‘excludes’ constraints is correctly processed. A BC Non void VM-2 VM-5 VM-6 VM-20 Check whether the interaction among three or more different types of relationships and constraints is correctly processed. A B D E C F G Non void VM-1, ..., VM-19 20 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VM-21 Check whether alternative relationships are erroneously implemented as or-relationships B C D A E Void VM-1 VM-4 VM-9 VM-10 VM-17 VM-22 Check whether or-relationships are erroneously implemented as alternative relationships B C D A E Non void VM-1 VM-3 VM-8 VM-10 VM-15 VM-23 Check whether alternative subfeatures can be part of a product without including its parent feature B C D A E Void VM-1 VM-2 VM-4 VM-5 VM-6 VM-7 VM-9 VM-10 VM-11 VM-13 VM-17 VM-18 VM-24 Check whether a product can erroneously include more than one alternative features B D G A H C F E Void VM-1 VM-4 VM-9 VM-10 VM-17 Table 7: Operation voidFM. Test cases A.2 Operation Valid Product 21 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-1 Check whether valid products are correctly identified in feature models with mandatory relationships. A B P={A,B} Valid VP-2 Check whether non-valid products are correctly identified in feature models with mandatory relationships. A B P={A} Non-valid VP-3 Check whether valid products (with a minimum set of features) are correctly identified in feature models with optional relationships. A B P={A} Valid VP-4 Check whether valid products (with a maximum set of features) are correctly identified in feature models with optional relationships. A B P={A,B} Valid VP-5 Check whether valid products (with a minimum set of features) are correctly identified in feature models with or-relationships. A B C P={A,B} Valid VP-6 Check whether valid products (with a maximum set of features) are correctly identified in feature models with or-relationships. A B C P={A,B,C} Valid 22 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-40 Check whether valid products (with a minimum set of features) are correctly identified in feature models containing or-relationships and ‘requires’ constraints. A B C P={A,C} Valid VP-5 VP-6 VP-7 VP-11 VP-12 VP-41 Check whether valid products (with a maximum set of features) are correctly identified in feature models containing or-relationships and ‘requires’ constraints. A B C P={A,B,C} Valid VP-5 VP-6 VP-7 VP-11 VP-12 VP-42 Check whether non-valid products (with a low number of features) are correctly identified in feature models containing or-relationships and ‘requires’ constraints. A B C P={A} Non-valid VP-5 VP-6 VP-7 VP-11 VP-12 VP-43 Check whether valid products are correctly identified in feature models containing orrelationships and ‘excludes’ constraints. A B C P={A,B} Valid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 VP-44 Check whether non-valid products (with a low number of features) are correctly identified in feature models containing or-relationships and ‘excludes’ constraints. A B C P={A} Non-valid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 29 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-45 Check whether non-valid products (with a high number of features) are correctly identified in feature models containing orrelationships and ‘excludes’ constraints. A B C P={A,B,C} Non-valid VP-5 VP-6 VP-7 VP-13 VP-14 VP-15 VP-46 Check whether valid products are correctly identified in feature models containing alternative relationships and ‘requires’ constraints. A B C P={A,C} Valid VP-8 VP-9 VP-10 VP-11 VP-12 VP-47 Check whether non-valid products (with a low number of features) are correctly identified in feature models containing alternative relationships and ‘requires’ constraints. A B C P={A} Non-valid VP-8 VP-9 VP-10 VP-11 VP-12 VP-48 Check whether non-valid products (with a high number of features) are correctly identified in feature models containing alternative relationships and ‘requires’ constraints. A B C P={A,B,C} Non-valid VP-8 VP-9 VP-10 VP-11 VP-12 VP-49 Check whether valid products are correctly identified in feature models containing alternative relationships and ‘excludes’ constraints. A B C P={A,B} Valid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 30 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-50 Check whether non-valid products (with a low number of features) are correctly identified in feature models containing alternative relationships and ‘excludes’ constraints. A B C P={A} Non-valid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 VP-51 Check whether non-valid products (with a high number of features) are correctly identified in feature models containing alternative relationships and ‘excludes’ constraints. A B C P={A,B,C} Non-valid VP-8 VP-9 VP-10 VP-13 VP-14 VP-15 VP-52 Check whether valid products (with a minimum set of features) are correctly identified in feature models containing ‘requires’ and ‘excludes’ constraints. A BC P={A} Valid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 VP-53 Check whether valid products (with a maximum set of features) are correctly identified in feature models containing ‘requires’ and ‘excludes’ constraints. A BC P={A,C} Valid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 VP-54 Check whether non-valid products (with a high number of features) are correctly identified in feature models containing ‘requires’ and ‘excludes’ constraints. A BC P={A,B,C} Non-valid VP-3 VP-4 VP-11 VP-12 VP-13 VP-14 VP-15 31 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-55 Check whether valid products (with a minimum set of features) are correctly identified in feature models containing three or more different types of relationships and constraints. A B D E C F G P={A,B,D} Valid VP-1, ..., VP-54 VP-56 Check whether valid products (with a maximum set of features) are correctly identified in feature models containing three or more different types of relationships and constraints. A B D E C F G P={A,B,C,E, F,G} Valid VP-1, ..., VP-54 VP-57 Check whether non-valid products (with a low number of features) are correctly identified in feature models containing three or more different types of relationships and constraints. A B D E C F G P={A,B} Non-valid VP-1, ..., VP-54 VP-58 Check whether non-valid products (with a high number of features) are correctly identified in feature models containing three or more different types of relationships and constraints. A B D E C F G P={A,B,C,D, E,F,G} Non-valid VP-1, ..., VP-54 VP-59 Check whether non-valid products excluding the root feature are correctly identified. A B P={B} Non-valid VP-3 VP-4 32 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. VP-60 Check whether non-valid products including non-existent features are correctly processed by the operation. A B P={A,H} Non-valid VP-3 VP-4 VP-61 Check whether child features in an alternative relationships can erroneously be part of a product without their parent feature. B C D A E P={A,D,E} Non Valid VP-1 ... VP-4 VP-8 ... VP-18 VP-22 ... VP-28 VP-32 ... VP-35 VP-46 ... VP-51 VP-62 Check whether a product can erroneously include more than one alternative features B D G A H C F E P={A,B,E,G,H} Non Valid VP-1 VP-2 VP-8 VP-9 VP-10 VP-22 ... VP-27 VP-46 VP-47 VP-48 VP-63 Check whether multiple alternative features can be erroneously part of a product when not including their non-mandatory parent feature A B C D P={A,C,D} Non Valid VP-3 VP-4 VP-8 VP-9 VP-10 VP-32 VP-33 VP-34 VP-35 Table 8: Operation ValidProduct. Test cases A.3 Operation All Products 33 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. P-1 Check whether mandatory relationships are correctly managed by the operation. A B {A,B} P-2 Check whether optional relationships are correctly managed by the operation. A B {A},{A,B} P-3 Check whether or–relationships are correctly managed by the operation. A B C {A,B},{A,C}, {A,B.C} P-4 Check whether alternative relationships are correctly managed by the operation. A B C {A,B},{A,C} P-5 Check whether ‘requires’ constraints are correctly managed by the operation. A B C {A},{A,C}, {A,B,C} P-2 P-6 Check whether ‘excludes’ constraints are correctly managed by the operation. A B C {A},{A,B}, {A,C} P-2 P-7 Check whether the interaction between mandatory and optional relationships is correctly processed. A B D C E {A,B},{A,B,D}, {A,B,C,E}, {A,B,C,D,E} P-1 P-2 34 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. P-8 Check whether the interaction between mandatory and or-relationships is correctly processed. A B E F C D G {A,B,C,E}, {A,B,C,F}, {A,B,C,E,F}, {A,B,D,E,G}, {A,B,C,D,E,G}, {A,B,D,F,G}, {A,B,C,D,F,G}, {A,B,D,E,F,G}, {A,B,C,D,E,F,G} P-1 P-3 P-9 Check whether the interaction between mandatory and alternative relationships is correctly processed. A B E F C D G {A,B,D,F}, {A,B,D,E}, {A,B,C,F,G}, {A,B,C,E,G} P-1 P-4 P-10 Check whether the interaction between mandatory relationships and ‘requires’ constraints is correctly processed. A BC {A,B,C} P-1 P-5 P-11 Check whether the interaction between mandatory relationships and ‘excludes’ constraints is correctly processed. A BC None P-1 P-6 35 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. P-12 Check whether the interaction between optional and or-relationships is correctly processed. A B E F C D G {A,D},{A,C}, {A,C,D}, {A,C,G}, {A,B,D,F}, {A,B,D,E}, {A,C,D,G}, {A,B,C,F}, {A,B,C,E}, {A,B,D,E,F}, {A,B,C,F,G}, {A,B,C,E,G}, {A,B,C,D,F}, {A,B,C,D,E}, {A,B,C,F,E}, {A,B,C,D,F,G}, {A,B,C,D,E,G}, {A,B,C,E,F,G}, {A,B,C,D,E,F}, {A,B,C,D,E,F,G} P-2 P-3 P-13 Check whether the interaction between optional and alternative relationships is correctly processed. A B E F C D G {A,C},{A,D}, {A,D,G}, {A,B,C,E}, {A,B,C,F}, {A,B,D,E}, {A,B,D,F}, {A,B,D,E,G}, {A,B,D,F,G} P-2 P-4 36 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. P-14 Check whether the interaction between or- and alternative relationships is correctly processed. A D E H I B C F G {A,D,C}, {A,C,E,H}, {A,C,E,I}, {A,B,D,G}, {A,B,D,F}, {A,C,D,E,H}, {A,C,D,E,I}, {A,B,D,F,G}, {A,B,E,G,H}, {A,B,E,F,H}, {A,B,E,G,I}, {A,B,E,F,I}, {A,B,D,E,G,H}, {A,B,E,F,G,H}, {A,B,D,E,F,H}, {A,B,D,E,G,I}, {A,B,E,F,G,I}, {A,B,D,E,F,I}, {A,B,D,E,F,G,H}, {A,B,D,E,F,G,I} P-3 P-4 P-15 Check whether the interaction between orrelationships and ‘requires’ constraints is correctly processed. A B C {A,C},{A,B,C} P-3 P-5 P-16 Check whether the interaction between orrelationships and ‘excludes’ constraints is correctly processed. A B C {A,B},{A,C} P-3 P-6 P-17 Check whether the interaction between alternative relationships and ‘requires’ constraints is correctly processed. A B C {A,C} P-4 P-5 P-18 Check whether the interaction between alternative relationships and ‘excludes’ constraints is correctly processed. A B C {A,B},{A,C} P-4 P-6 37 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. P-19 Check whether the interaction between ‘requires’ and ‘excludes’ constraints is correctly processed. A BC {A},{A,C} P-2 P-5 P-6 P-20 Check whether the interaction among three or more different types of relationships and constraints is correctly processed. A B D E C F G {A,B,D}, {A,B,C,D,F}, {A,B,C,E,F}, {A,B,C,E,F,G} P-1, ..., P-19 P-21 Check whether a product can erroneously include more than one alternative features. B D G A H C F E None P-1 P-4 P-9 P-10 P-17 Table 9: Operation Products. Test cases A.4 Operation Number of Products ID Description Input Exp. Output Deps. NP-1 Check whether mandatory relationships are correctly managed by the operation. A B 1 NP-2 Check whether optional relationships are correctly managed by the operation. A B 2 NP-3 Check whether or–relationships are correctly managed by the operation. A B C 3 38 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-2 Check whether optional relationships are correctly managed by the operation. A B Feature=B 50% C-3 Check whether or–relationships are correctly managed by the operation. A B C Feature=B 66% C-4 Check whether alternative relationships are correctly managed by the operation. A B C Feature=B 50% C-5 Check whether ‘requires’ constraints are correctly managed by the operation. Input feature has minimum commonality. A B C Feature=B 33% C-2 C-6 Check whether ‘requires’ constraints are correctly managed by the operation. Input feature has maximum commonality. A B C Feature=C 66% C-2 C-7 Check whether ‘excludes’ constraints are correctly managed by the operation. A B C Feature=B 33% C-2 45 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-8 Check whether the interaction between mandatory and optional relationships is correctly processed. Input feature has minimum commonality. A B D C E Feature=E 50% C-1 C-2 C-9 Check whether the interaction between mandatory and optional relationships is correctly processed. Input feature has maximum commonality. A B D C E Feature=B 100% C-1 C-2 C-10 Check whether the interaction between mandatory and or-relationships is correctly processed. Input feature has minimum commonality. A B E F C D G Feature=F 66% C-1 C-3 C-11 Check whether the interaction between mandatory and or-relationships is correctly processed. Input feature has maximum commonality. A B E F C D G Feature=B 100% C-1 C-3 C-12 Check whether the interaction between mandatory and alternative relationships is correctly processed. Input feature has minimum commonality. A B E F C D G Feature=G 50% C-1 C-4 46 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-13 Check whether the interaction between mandatory and alternative relationships is correctly processed. Input feature has maximum commonality. A B E F C D G Feature=B 100% C-1 C-4 C-14 Check whether the interaction between mandatory relationships and ‘requires’ constraints is correctly processed. A BC Feature=B 100% C-1 C-5 C-6 C-15 Check whether the interaction between mandatory relationships and ‘excludes’ constraints is correctly processed. A BC Feature=B 0% C-1 C-7 C-16 Check whether the interaction between optional and or-relationships is correctly processed. Input feature has minimum commonality. A B E F C D G Feature=G 40% C-2 C-3 C-17 Check whether the interaction between optional and or-relationships is correctly processed. Input feature has maximum commonality. A B E F C D G Feature=C 80% C-2 C-3 47 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-18 Check whether the interaction between optional and alternative relationships is correctly processed. Input feature has minimum commonality. A B E F C D G Feature=E 33% C-2 C-4 C-19 Check whether the interaction between optional and alternative relationships is correctly processed. Input feature has maximum commonality. A B E F C D G Feature=D 66% C-2 C-4 C-20 Check whether the interaction between or- and alternative relationships is correctly processed. Input feature has minimum commonality. A D E H I B C F G Feature=C 25% C-3 C-4 C-21 Check whether the interaction between or- and alternative relationships is correctly processed. Input feature has maximum commonality. A D E H I B C F G Feature=E 80% C-3 C-4 C-22 Check whether the interaction between orrelationships and ‘requires’ constraints is correctly processed. Input feature has minimum commonality. A B C Feature=B 50% C-3 C-5 C-6 C-23 Check whether the interaction between orrelationships and ‘requires’ constraints is correctly processed. Input feature has maximum commonality. A B C Feature=C 100% C-3 C-5 C-6 48 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-24 Check whether the interaction between orrelationships and ‘excludes’ constraints is correctly processed. A B C Feature=C 50% C-3 C-7 C-25 Check whether the interaction between alternative relationships and ‘requires’ constraints is correctly processed. Input feature has minimum commonality. A B C Feature=B 0% C-4 C-5 C-6 C-26 Check whether the interaction between alternative relationships and ‘requires’ constraints is correctly processed. Input feature has maximum commonality. A B C Feature=C 100% C-4 C-5 C-6 C-27 Check whether the interaction between alternative relationships and ‘excludes’ constraints is correctly processed. A B C Feature=B 50% C-4 C-7 C-28 Check whether the interaction between ‘requires’ and ‘excludes’ constraints is correctly processed. Input feature has minimum commonality. A BC Feature=B 0% C-2 C-5 C-6 C-7 49 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. C-29 Check whether the interaction between ‘requires’ and ‘excludes’ constraints is correctly processed. Input feature has maximum commonality. A BC Feature=C 50% C-2 C-5 C-6 C-7 C-30 Check whether the interaction among three or more different types of relationships and constraints is correctly processed. Input feature has minimum commonality. A B D E C F G Feature=G 25% C-1, ..., C-29 C-31 Check whether the interaction among three or more different types of relationships and constraints is correctly processed. Input feature has maximum commonality. A B D E C F G Feature=B 100% C-1, ..., C-29 C-32 Check whether non-existent features are correctly managed. A B Feature=C 0% C-2 C-33 Check whether a product can erroneously include more than one alternative feature. B D G A H C F E 0% C-1 C-4 C-12 C-13 C-14 C-25 C-26 Table 12: Operation Commonality. Test cases A.7 Operation Dead Features 50 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. DF-1 Check whether dead features caused by an ‘excludes’ constraints between a mandatory feature and an alternative child feature are correctly detected. A BC D E D DF-2 Check whether dead features caused by the a ‘requires’ constraints between a mandatory feature and an alternative child feature are correctly detected. The parent feature in the alternative relationship is mandatory. A BC D E E DF-3 Check whether dead features caused by an ‘excludes’ constraints between a mandatory feature and one of the child features of an orrelationship are correctly detected. A BC D E D DF-4 Check whether dead features caused by an ‘excludes’ constraint between a mandatory and an optional feature are correctly detected. A BC C DF-5 Check whether dead features caused by an ‘excludes’ constraint between two mandatory features are correctly detected. A BC A,B,C DF-6 Check whether dead features caused by a ‘requires’ constraint between two alternative child features are correctly detected. A B C B 51 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01 ID Description Input Exp. Output Deps. DF-7 Check whether dead features caused by an ‘excludes’ constraint between a parent and its child feature are correctly detected. A B A,B DF-8 Check whether dead features caused by an ‘excludes’ and a ‘requires’ constraints between two optional features are correctly detected. A BC B DF-9 Check whether dead features caused by the a ‘requires’ constraints between a mandatory feature and an alternative child feature are correctly detected. The parent feature in the alternative relationship is optional. A BC D E E Table 13: Operation DeadFeatures. Test cases 52 FaMa Test Suite v1.2 ISA Technical Report ISA-10-TR-01