RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs

RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs Alberto Martin-Lopez, Sergio Segura, and Antonio Ruiz-Cort´es Smart Computer Systems Research and Engineering Lab (SCORE), Research Institute of Computer Engineering (I3US) Universidad de Sevilla, Spain {alberto.martin,sergiosegura,aruiz}@us.es Abstract. Automated testing approaches for RESTful web APIs typically follow a black-box strategy, where test cases are derived from the API specification. These techniques show promising results, but they neglect constraints among input parameters (so-called inter-parameter dependencies), as these cannot be formally described in current API specification languages. As a result, black-box tools rely on brute force to generate valid test cases, i.e., those satisfying all the input constraints. This is not only extremely inefficient, but it is also unlikely to work for most real-world services, where inter-parameter dependencies are complex and pervasive. In this paper, we present RESTest, a framework for automated black-box testing of RESTful APIs. Among its key features, RESTest supports the specification and automated analysis of interparameter dependencies, enabling the use of constraint solvers for the automated generation of valid test cases. This allows to detect more faults, and faster, through a deeper evaluation of valid and invalid input parameters’ combinations and the use of novel test oracles. Evaluation results on 6 commercial APIs show that RESTest can efficiently generate up to 99% more valid test cases than random testing techniques, 60% on average. More importantly, RESTest revealed 2K failures undetected by random testing, uncovering bugs in all the services under test. Keywords: REST ·Black-box testing ·Constraint-based testing ·Web services 1 Introduction Web APIs allow systems to interact over the network, typically using web services [18]. Modern web APIs typically adhere to the REpresentational State Transfer (REST) architectural style [5], being referred to as RESTful web APIs. RESTful web APIs are comprised of one or more RESTful web services, each of which implements one or more create, read, update, or delete (CRUD) operations to access and manipulate a resource, e.g., a video in the YouTube API. RESTful APIs are commonly described using languages like the OpenAPI Specification (OAS) [15], originally created as a part of the Swagger tool suite [22]. OAS is designed to provide a structured description of a RESTful web API that allows 2 A. Martin-Lopez et al. both humans and computers to discover and understand the capabilities of a service without requiring access to the source code or additional documentation. Web APIs often impose dependency constraints that restrict the way in which two or more input parameters can be combined to form valid calls to the service, these are often called inter-parameter dependencies (or simply dependencies henceforth). For example, in the Google Maps API, when searching for places, if the location parameter is set, then the radius parameter must be set too, otherwise a 400 status code (“bad request”) is returned. In a recent study, we reviewed more than 2.5K operations from 40 industrial APIs and found that dependencies are extremely common and pervasive—they appear in 4 out of every 5 APIs across all application domains and types of operations [11]. Unfortunately, current API specification languages like OAS provide no support for the formal description of this type of dependencies, despite being a highly demanded feature by practitioners1. Instead, users are encouraged to describe dependencies among input parameters informally, using natural language, which leads to ambiguities and makes it hardly possible to interact with services without human intervention2. To address this problem, in previous work we proposed a domain-specific language for the formal specification of dependencies, called Inter-parameter Dependency Language (IDL), and a tool suite for the automated analysis of IDL using constraint programming [12] (c.f. Section 2). In this paper, we show the potential of IDL and its tool suite in the context of testing RESTful APIs. The validation of RESTful web APIs is critical as they play a key role in modern software integration. A faulty API can have a huge impact in the many applications using it. The automated detection of bugs in RESTful web APIs is an active research topic [2–4, 9, 19, 23]. Most contributions in this context follow a black-box strategy, where the specification of the API under test (described using the OAS language) is used to drive the generation of test cases [3, 4, 9, 23]. Essentially, these approaches exercise the API under test using (pseudo) random test data. Test data generation strategies include using default values [4], input data dictionaries [3], test data generators [9] and data observed in previous calls to the API [23]. Failures are detected when the observed output deviates from the specification, e.g., unexpected HTTP status codes. Problem: Current black-box testing approaches for RESTful web APIs do not support inter-parameter dependencies since, as previously mentioned, these are not formally described in the API specification used as input. As a result, existing approaches simply ignore dependencies and resort to brute force to generate valid test cases, i.e., those satisfying all input constraints. This is not only extremely inefficient, but it is also unlikely to work for most real-world services, where inter-parameter dependencies are complex and pervasive. For example, the search operation in the YouTube API has 31 input parameters, out of which 25 are involved in at least one dependency: trying to generate valid 1This is reflected in an open feature request in OAS entitled “Support interdependencies between query parameters”, with over 290 votes and 55 comments from 33 participants. https://github.com/OAI/OpenAPI-Specification/issues/256 2https://swagger.io/docs/specification/describing-parameters/ RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs 3 test cases randomly is like hitting a wall. This was confirmed in our evaluation, where 98 out of every 100 random test cases for the YouTube search operation violated one or more inter-parameter dependencies (c.f. Section 4.3). Contribution: In this paper, we present RESTest, an open-source and blackbox automated testing framework for RESTful web APIs. RESTest follows a model-based approach enabling its integration with different test case generators and testing frameworks. As its most distinctive feature, RESTest supports the specification and automated analysis of inter-parameter dependencies using the IDL tool suite. This allows to exploit constraint solving as a part of the test generation process, a testing technique generally known as constraint-based testing [8]. Constraint-based testing enables a better coverage of the program under test through the systematic generation of valid and invalid input combinations, as well as the use of novel output assertions, i.e., test oracles. For the evaluation of RESTest, we tested 9 operations from 6 commercial APIs, including Tumblr, GitHub and YouTube. Specifically, we compared random testing—state-of- the-art technique for black-box testing of RESTful APIs—and constraint-based testing. As expected, random testing struggled to generate valid test cases: 60% of the generated test cases violated inter-parameter dependencies (about 99% in the APIs of Stripe and YouTube). In contrast, constraint-based testing generated 100% valid test cases for all the services under test, keeping the test case generation time in milliseconds. More importantly, constraint-based testing detected more failures than random testing (4K vs 3K), in more services (9 vs 5), showing the potential of RESTest in practice. This work includes the following original contributions in the context of automated testing of RESTful web APIs: 1. An open-source and model-based framework for automated black-box test case generation and execution. 2. A new constraint-based approach for improving test case generation techniques, including two novel automated test oracles. 3. Experimental evidence on the limits of using random testing in real-world services with inter-parameter dependencies. 4. A comparison of random testing and constraint-based testing on 6 commercial APIs, showing the potential of both techniques, and especially constraintbased testing, to uncover real bugs. The remainder of the paper is organised as follows: Section 2 introduces the IDL tool suite, used for the automated analysis of inter-parameter dependencies in RESTful web APIs. Section 3 presents RESTest, our testing framework for RESTful APIs. Section 4 explains the evaluation performed and the results obtained. Section 5 outlines threats to validity. Section 6 describes related work. Finally, Section 7 draws conclusions and discusses future lines of research. 2 IDL Tool Suite RESTest relies on the IDL tool suite for the automated management of interparameter dependencies in RESTful APIs [12]. Inter-parameter Dependency Lan- 4 A. Martin-Lopez et al. guage (IDL) [21] is a domain-specific language for the specification of interparameter dependencies in web APIs. It is based on a thorough study of more than 2.5K operations in 40 real-world APIs [11]. Specifically, it provides support for eight different types of dependencies among input parameters consistently found in practice. Listing 1 shows an example of each type of dependency taken from commercial APIs. The syntax is self-explanatory. For example, the Requires dependency in line 1, observed in the API of YouTube, states that, when using the parameter videoDefinition, the parameter type must be set to ‘video’. IDL specifications can be integrated into OAS documents using the IDL4OAS extension [12]. This allows to enrich API specifications with an accurate, not ambiguous and machine-readable description of the dependencies among input parameters. We refer the reader to the supplementary material of the paper for examples of API specifications using the OAS language and the IDL4OAS extension [21]. 1IF videoDefinition THEN type == 'video ';// Requires 2Or(query , type ); // Or 3ZeroOrOne( radius , rankby == 'distance '); // ZeroOrOne 4AllOrNone( location , radius ) ; // AllOrNone 5OnlyOne( amount_off , perc ent_of f ); // OnlyOne 6publishedAfter >= publishedBefore; // Relational 7limit + offset <= 1000; // Arithmetic 8IF intent=='browse'THEN OnlyOne(ll AND radius , sw AND ne ); // Complex Listing 1: Examples of IDL dependencies from real-world APIs. IDLReasoner [21] is an open-source Java library for the automated analysis of IDL specifications [12]. Given an OAS specification and a set of IDL dependencies (e.g., using IDL4OAS), the tool translates them into a constraint satisfaction problem (CSP) expressed in MiniZinc [14], a constraint solving language designed for modelling optimisation problems in a high-level, solver-independent way. Then, several analysis operations can be invoked on the resulting CSP, for instance, to know whether a given API request satisfies all the inter-parameter dependencies. Section 3.2 describes the analysis operations used to support test case generation in RESTest. 3 RESTest In this section, we present RESTest, our framework for automated black-box testing of RESTful web APIs. RESTest follows a model-based approach, where test cases are automatically derived from the specification of the API under test. No access to the source code is required, which makes it possible to test APIs written in any programming language, running in local or remote servers. Figure 1 shows how RESTest works. It takes as input the OAS specification of the API under test, considered the system model. The specification can optionally describe inter-parameter dependencies using the IDL4OAS extension. Then, a so-called test model is automatically generated from the system model including test-specific configuration data. The default test model can be manually enriched with fine-grained configuration details such as test data generation RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs 5 Fig. 1: Test case generation and execution in RESTest. settings. Then, both the system and the test models are leveraged for the generation of abstract test cases following user-defined test case generation strategies such as random testing. In parallel, inter-parameter dependencies, if any, are fed into the tool IDLReasoner, providing support for their automated analysis during test case generation, for instance, to check whether an API call satisfies all the inter-parameter dependencies defined in the specification. Finally, abstract test cases are transformed into platform-specific executable test cases and they are executed. In the following sections, we detail the main steps of the process. 3.1 Default Test Model Generation RESTest takes as input the specification of the API under test, i.e., system model. Specifications described using the OAS language—arguably considered the industry-standard and used in related approaches [2–4, 9, 23]—are supported, but other API specification languages could be integrated into the framework using available converters, e.g., RAML to OAS [15]. As a distinctive feature, RESTest supports the specification of inter-parameter dependencies within the OAS document using the IDL4OAS extension [12] (c.f. Section 2). Test case generation in RESTest is driven by the system and the test models. The test model includes all test-related configuration settings for the API under test. A default test model, formatted in YAML (i.e., same language used in OAS), is automatically generated from the input API specification (system model). Such test model might be enough to generate effective test cases in some APIs. However, in practice, some manual tuning is often necessary, for example, to generate input values hardly inferred from the specification such as identifiers or codes. In particular, RESTest supports the following configuration settings: –Operations under test. It is possible to specify the subset of the API operations to be tested. Specific test configuration settings can be defined for each operation under test. 6 A. Martin-Lopez et al. –Authentication data. This includes the API keys or tokens required to call secured APIs. –Test data generators. This allows to customise the data values used for each input parameter. Test data generators in RESTest include random value generators, regular expression generators, boundary-value generators, and data dictionaries, among others. Default generators are configured according to the type of the input parameters, e.g., English words for string parameters. –Weights. Testers might be interested in testing some parameters more thoroughly than others, for example, those more used in practice. Weights allow to do so. A weight is a real number in the range [0,1]. The higher the weight of a parameter, the more frequently it will be used in test cases. By default, all parameters have a weight of 0.5. 3.2 Automated Analysis of Inter-Parameter Dependencies API specifications including inter-parameter dependencies in IDL are provided as input to IDLReasoner [12]. This tool transforms the specification into a CSP and automatically checks for inconsistencies in the specification, informing the user about any errors, e.g., parameters that cannot be selected. Once the specification is validated, IDLReasoner provides test case generators with a catalogue of helpful analysis operations. Among these, three analysis operations stand out as particularly helpful during test case generation, namely: –isValidRequest. This operation takes as input an API specification (including inter-parameter dependencies) and a service request (i.e., a list of parameters and their values), and returns a Boolean indicating whether the request is valid or not. A service request is valid if it satisfies all the inter-parameter dependencies defined in the specification. –getRandomValidRequest. This operation receives the API specification of an API operation, and returns a random valid request for the operation, that is, a random assignment of values to input parameters satisfying all the dependencies of the specification. –getRandomInvalidRequest. Contrary to the previous operation, this operation returns a random request violating one or more dependencies. The use of the IDL tool suite allows to decouple the automated management of dependencies from the specific test case generation approach used. This makes RESTest highly generic and easy to maintain. Furthermore, it eases the use of different CSP solvers and the development of new analysis operations. 3.3 Abstract Test Case Generation Test cases can be derived from the system and test models using one or more test case generation techniques. These test cases are abstract or platform-independent, meaning that they can be later transformed into executable test cases for specific testing frameworks and programming languages. RESTest currently supports random and constraint-based test case generation, but other techniques RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs 7 (e.g., search-based generation) could be easily integrated extending the right interfaces. Abstract test cases comprise test inputs, expected outputs (test oracles), and the required information to build the API request, e.g., the endpoint. RESTest currently supports testing at the operation-level, that is, each test case performs a single API request. RESTest generates both nominal and faulty test cases. Nominal test cases aim to test the API with valid inputs, i.e., those conforming to the API specification. In practice, it is not always possible to guarantee that a nominal test case represents a valid call to the API since it could violate some inter-parameter dependency, for example. Therefore, nominal test cases can be regarded as potentially valid test cases aimed at obtaining successful responses from the API (i.e., 2XX status codes). Faulty test cases check the ability of the API to handle invalid inputs, and therefore they expect a client error as a response (i.e., 4XX status codes). Faulty test cases are generated by creating faulty variants (i.e., mutants) of nominal test cases. For example, RESTest supports the automated generation of faulty test cases by excluding mandatory parameters, using out- of-range values (e.g., assigning a string to an integer parameter), and violating the JSON schema of the request body, among others. Additionally, as a novel feature of RESTest, the framework supports the automated generation of invalid requests violating inter-parameter dependencies using IDLReasoner. Test case generation techniques mostly focus on generating test inputs, however, half of the challenge in testing lies on test oracles, that is, how to distinguish correct outputs from incorrect ones. RESTest supports the five test oracles described below, where the last two are novel as they rely on the automated analysis of inter-parameter dependencies. –5XX. The status code must be lower than 500 (server error). –OAS. The response must conform to the OAS schema. –2XXP. If the request violates the specification of individual parameters (e.g., a mandatory parameter is missing), the status code must not be 2XX (successful response). –2XXD. If the request violates one or more inter-parameter dependencies, the status code must not be 2XX. –4XX. If the request is valid according to the API specification, the status code must not be 4XX (client error response). Oracles 2XXDand 4XX are novel contributions of our work. Both of them reveal failures undetectable by current state-of-the-art test oracles. It is worth noting that oracle 4XX is particularly helpful as it allows to detect critical bugs: those making the API return a client error response (4XX status code) with a valid API call. Detecting this kind of failures is only possible thanks to the automated analysis of inter-parameter dependencies, which allows to automatically determine whether a request is valid before calling the actual API (assuming that the specification is correct and that the right test data generators are used). 8 A. Martin-Lopez et al. 3.4 Test Case Generation and Execution The last step is concerned with test execution. Abstract test cases are instantiated into executable test cases using specific testing frameworks and libraries. RESTest currently supports the generation of executable test cases using REST Assured [17], a Java library for testing RESTful services, developed as a JU- nit extension. However, other frameworks and programming languages could be easily supported by implementing specific test writers. Test execution can be done either offline or online. In offline testing, test case generation and execution are independent tasks. This has certain benefits. For example, test cases can be generated once, and then be executed many times as a part of regression testing. Also, test generation and test execution can be performed on different machines and at different times. In online testing, test case generation and execution are interleaved. This enables, for example, fully autonomous testing of RESTful web APIs, e.g., generating and executing test cases 24/7 as a part of a Continuous Integration (CI) setup. RESTest supports both offline and online testing. However, more sophisticated techniques for online testing remain to be implemented. For example, the test generation algorithms can react to the actual outputs of the API under test, e.g., to guide search-based test case generation algorithms based on the coverage achieved so far [13]. 4 Evaluation In this section, we assess the ability of RESTest to generate valid test cases (i.e., those satisfying all the input constraints) and to reveal failures in real-world APIs with inter-parameter dependencies. To this end, we compare random testing (RT)—state-of-the-art technique for black-box testing of RESTful APIs—and constraint-based testing (CBT). We address the following research questions: – RQ1:What is the effectiveness of CBT in generating valid test cases for real-world APIs with inter-parameter dependencies? – RQ2:What is the fault-finding capability of CBT in real-world APIs with inter-parameter dependencies? 4.1 Services Under Test We tested 9 services from 6 commercial RESTful APIs with millions of users. We selected both read and write operations from those services. In order to assess the potential of CBT, we selected operations containing the eight types of dependencies identified in our study [11], with more than 50% of their parameters involved in at least one dependency. Table 1 provides a summary of the services under test (SUTs). For each SUT, the table shows an identifier (used to refer to it within the rest of the paper), API name, description of the operation tested, number of input parameters (P), number of IDL dependencies (D), and number (and percentage) of different parameters involved in at least one dependency RESTest: Black-Box Constraint-Based Testing of RESTful Web APIs 9 ID API Operation P D PD (%) Foursquare Foursquare Search venues 17 8 10 (59%) GitHub GitHub Get user repositories 5 2 3 (60%) Stripe-CC Stripe Create coupon 9 3 5 (56%) Stripe-CP Stripe Create product 18 6 11 (61%) Tumblr Tumblr Get blog likes 5 1 3 (60%) Yelp Yelp Search businesses 14 3 7 (50%) YouTube-GCT YouTube Get comment threads 11 5 8 (73%) YouTube-GV YouTube Get videos 12 5 7 (58%) YouTube-S YouTube Search 31 16 25 (81%) Mean 13.6 5.4 8.8 (62%) Table 1: RESTful API operations used in the evaluation. P = Number of parameters, D = Number of IDL dependencies, PD = Number and percentage of parameters involved in at least one dependency. (PD). On average, the operations have 14 parameters, 5 dependencies and 9 parameters involved in dependencies. The OAS specification of each API under test, used as input in RESTest, was taken from the API website or from the APIs.guru repository [1]. When the specification was not available (Foursquare, Tumblr and Yelp), we created it manually based on the online API documentation. Then, we looked for interparameter dependencies described in the documentation and included them as a part of the specification using the IDL4OAS extension. The links to the APIs under test and their OAS specifications are available as part of the supplementary material of the paper [21]. 4.2 Test Case Generation Techniques Next, we describe the test case generation techniques used in the evaluation. Random testing (RT). This is the state-of-the-art approach used as baseline in our work [4, 9, 23]. Nominal test cases are generated by randomly selecting a subset of the operation parameters and assigning random values to them within their domain. All parameters are selected with the same probability (i.e., weight = 0.5) except mandatory ones, which are always included in the API request. Notice that this approach neglects inter-parameter dependencies and so the generated test cases may not be valid, i.e., they may generate responses with 4XX status codes (client error). Faulty test cases are generated by mutating nominal test cases as described in Section 3.3, e.g., excluding a mandatory parameter from the API call. Constraint-based testing (CBT). Nominal test cases are generated in two steps. First, the domain of each input parameter is discretised and reduced to a fixed number of random values, within their domain, using RESTest test data generators. Then, the analysis operation getRandomValidRequest is invoked on IDLReasoner to generate a request that satisfies all inter-parameter dependencies. Analogously to RT, faulty test cases can be generated by mutating nominal 16 A. Martin-Lopez et al. References 1. APIs.guru, https://apis.guru, accessed April 2020 2. Arcuri, A.: RESTful API Automated Test Case Generation with EvoMaster. ACM TOSEM 28(1), 1–37 (2019) 3. Atlidakis, V., Godefroid, P., Polishchuk, M.: Checking Security Properties of Cloud Services REST APIs. In: ICST (2020) 4. Ed-douibi, H., Izquierdo, J.L.C., Cabot, J.: Automatic Generation of Test Cases for REST APIs: A Specification-Based Approach. In: EDOC. pp. 181–190 (2018) 5. Fielding, R.T.: Architectural Styles and the Design of Network-based Software Architectures. Ph.D. thesis (2000) 6. Gamez-Diaz, A., Fernandez, P., Ruiz-Cortes, A.: Automating SLA-Driven API Development with SLA4OAI. In: ICSOC. pp. 20–35 (2019) 7. Gamez-Diaz, A., Fernandez, P., Ruiz-Cort´es, A.: Governify for APIs: SLA-Driven Ecosystem for API Governance. In: ESEC/FSE. pp. 1120–1123 (2019) 8. Gotlieb, A.: Constraint-Based Testing: An Emerging Trend in Software Testing. In: Advances in Computers, vol. 99, pp. 67–101. Elsevier (2015) 9. Karlsson, S., Causevic, A., Sundmark, D.: QuickREST: Property-based Test Generation of OpenAPI Described RESTful APIs. In: ICST (2020) 10. Li, Y., Sun, Z.a., Fang, J.Y.: Generating an Automated Test Suite by Variable Strength Combinatorial Testing for Web Services. CIT 24(3), 271–282 (2016) 11. Martin-Lopez, A., Segura, S., Ruiz-Cort´es, A.: A Catalogue of Inter-Parameter Dependencies in RESTful Web APIs. In: ICSOC. pp. 399–414 (2019) 12. Martin-Lopez, A., Segura, S., M¨uller, C., Ruiz-Cort´es, A.: Specification and Automated Analysis of Inter-Parameter Dependencies in Web APIs. Submitted to IEEE Transactions on Services Computing (2020), https://bit.ly/2ECr9rc 13. Martin-Lopez, A., Segura, S., Ruiz-Cort´es, A.: Test Coverage Criteria for RESTful Web APIs. In: A-TEST. pp. 15–21 (2019) 14. MiniZinc: Constraint Modeling Language, https://www.minizinc.org, accessed April 2020 15. OpenAPI Specification, https://www.openapis.org, accessed April 2020 16. Semantic Markup for Web Services (OWL-S), https://www.w3.org/Submission/ OWL-S, accessed May 2020 17. REST Assured, http://rest-assured.io, accessed April 2020 18. Richardson, L., Amundsen, M., Ruby, S.: RESTful Web APIs. O’Reilly Media, Inc. (2013) 19. Segura, S., Parejo, J.A., Troya, J., Ruiz-Cort´es, A.: Metamorphic Testing of REST- ful Web APIs. IEEE TSE 44(11), 1083–1099 (2018) 20. Sun, C.a., Li, M., Jia, J., Han, J.: Constraint-Based Model-Driven Testing of Web Services for Behavior Conformance. In: ICSOC. pp. 543–559 (2018) 21. Supplementary material of the paper, https://github.com/isa-group/ icsoc-2020-supplementary-material 22. Swagger, http://swagger.io, accessed April 2020 23. Viglianisi, E., Dallago, M., Ceccato, M.: RestTestGen: Automated Black-Box Testing of RESTful APIs. In: ICST (2020) 24. Web Services Description Language (WSDL) Version 2.0, https://www.w3.org/ TR/wsdl20, accessed May 2020 25. Xu, L., Yuan, Q., Wu, J., Liu, C.: Ontology-based Web Service Robustness Test Generation. In: WSE. pp. 59–68 (2009)