ARTE: Automated Generation of Realistic Test Inputs for Web APIs

1 ARTE: Automated Generation of Realistic Test Inputs for Web APIs Juan C. Alonso , Alberto Martin-Lopez , Sergio Segura , Member, IEEE, José María García and Antonio Ruiz-Cortés , Member, IEEE Abstract—Automated test case generation for web APIs is a thriving research topic, where test cases are frequently derived from the API specification. However, this process is only partially automated since testers are usually obliged to manually set meaningful valid test inputs for each input parameter. In this article, we present ARTE, an approach for the automated extraction of realistic test data for web APIs from knowledge bases like DBpedia. Specifically, ARTE leverages the specification of the API parameters to automatically search for realistic test inputs using natural language processing, search-based, and knowledge extraction techniques. ARTE has been integrated into RESTest, an open-source testing framework for RESTful APIs, fully automating the test case generation process. Evaluation results on 140 operations from 48 real-world web APIs show that ARTE can efficiently generate realistic test inputs for 64.9% of the target parameters, outperforming the state-of-the-art approach SAIGEN (31.8%). More importantly, ARTE supported the generation of over twice as many valid API calls (57.3%) as random generation (20%) and SAIGEN (26%), leading to a higher failure detection capability and uncovering several real-world bugs. These results show the potential of ARTE for enhancing existing web API testing tools, achieving an unprecedented level of automation. Index Terms—Test data generation, automated testing, web APIs, Web of Data F 1 INTRODUCTION 1WEB Application Programming Interfaces (APIs) allow2 heterogeneous software systems to talk to each other3 over the network [1], [2]. Modern web APIs typically adhere4 to the REpresentational State Transfer (REST) architectural5 style, being referred to as RESTful web APIs [3]. RESTful6 web APIs typically allow applications to interact by ex-7 changing JSON messages sent over HTTP. In practice, this8 allows, for example, checking the result of a football match9 (BeSoccer API [4]), posting a tweet (Twitter API [5]), booking10 a hotel room (Amadeus API [6]), translating a text (DeepL11 API [7]), or finding a route between two locations (Open-12 route API [8]). RESTful APIs are commonly described using13 languages such as the OpenAPI Specification (OAS) [9].14 An OAS document provides a structured specification of a15 RESTful web API that allows both humans and computers16 to discover and understand the capabilities of a service17 without requiring access to the source code or additional18 documentation. In what follows, we will use the terms19 RESTful web API, web API, or just API interchangeably.20 Testing web APIs adequately requires using realistic21 test inputs such as country names, codes, coordinates, or22 addresses. As an example, the hotel search operation in23 the Amadeus API [6] requires users to provide valid ho-24 tel names (e.g., “Hotel California”), hotel chains (e.g.,25 “Hilton”), IATA airport codes (e.g., “BUE” for Buenos Aires),26 ISO currency codes (e.g., “EUR” for Euro), and ISO language27 codes (e.g., “FR” for French), among others. Generating28 meaningful values for these types of parameters randomly29 •J.C. Alonso, A. Martin-Lopez, S. Segura, J.M. García, and A. Ruiz-Cortés are with the Smart Computer systems Research and Engineering Lab (SCORE) and the Research Institute of Informatics Engineering (I3US), Universidad de Sevilla, Spain. E-mail: {javalenzuela, alberto.martin, sergiosegura, josemgarcia, aruiz}@us.es is rarely feasible. Even if a test data generator could bypass 30 the syntactic validation generating values with the right 31 format, the chances of constructing API requests that return 32 some results—and therefore exercise the core functionality 33 of the API—would be remote. To address this issue, most 34 test case generation approaches resort to data dictionaries:35 sets of input values collected by the testers, either manu- 36 ally [10] or, when possible, automatically [11]. This means 37 a major obstacle for automation since data dictionaries 38 must be created and maintained for each non-trivial input 39 parameter, on each API under test. Other authors propose 40 using the default or sample values included in the API 41 specification as test inputs, if any, but those are solely 42 intended to explain the behavior of the API, and they are 43 insufficient to test it thoroughly [12]. 44 Several authors have addressed the problem of generat- 45 ing realistic test inputs for desktop, web, and mobile apps 46 using semantic knowledge discovery techniques [13], [14]. 47 Specifically, they propose to query knowledge bases like 48 DBpedia [15] to discover realistic test input values for the 49 graphical user interface (GUI) elements of the application 50 under test. For example, if the GUI includes a text field with 51 the label “DOI”, their approaches would search for “DOI” 52 identifiers in DBpedia. To the best of our knowledge, this 53 strategy—based on querying knowledge bases for realistic 54 test inputs—has not been applied to the context of web APIs 55 yet, despite its potential. 56 Automatically generating realistic test data for web APIs 57 requires facing some unique challenges since, unlike GUIs, 58 APIs are intended for developers, rather than for users. 59 To start with, unlike GUI labels, API parameters may fol- 60 low many different naming conventions. Hence, for exam- 61 ple, different APIs could refer to the concept country code 62 using very different parameters’ names such as country 63 2 (Asos API [16]), countryCode (DHL API [17]), country_code64 (Numverify API [18]), cc (Foursquare API [19]), c, or65 even a less explanatory term, such as location (Domainr66 API [20]) or excludedCountryIds (GeoDBCities API [21]).67 This may make it necessary to resort to the description of68 each parameter in the search for further information that69 helps to identify the key concept. However, descriptions70 are given in natural language and, as names, they can be71 very heterogeneous both in style and length. Finally, API72 specifications occasionally include further information such73 as expected patterns (i.e., regular expressions) and sample74 values. Exploiting all these aspects for the generation of75 realistic test data is the key challenge addressed in our work.76 In this article, we present an approach for the Automated77 generation of Realistic TEst inputs (ARTE) for web APIs.78 Specifically, our approach focuses on RESTful APIs as the79 de facto standard, but it could be applied to any web API80 provided that there is an API specification. ARTE leverages81 semantic knowledge discovery for the generation of realistic82 test inputs. In particular, ARTE exploits the specification83 of the various elements of the API under test, such as the84 name and description of its parameters, by querying knowl-85 edge bases to automatically generate realistic test inputs.86 In contrast to related approaches, ARTE includes a novel87 step for the automated inference of regular expressions88 from previously generated inputs, increasing the accuracy89 of the semantic queries and the overall performance of90 the approach. Furthermore, ARTE has been integrated into91 RESTest [22], a state-of-the-art tool for black-box test case92 generation for RESTful APIs, making our approach fully93 automated and publicly available.94 We evaluated the effectiveness of ARTE by comparing95 its performance with existing random techniques and the96 related tool SAIGEN—recently proposed in the context of97 automated testing of mobile apps [14]—in two scenarios:98 (1) on the generation of realistic test inputs for 48 web APIs;99 and (2) on the generation of valid API calls, API coverage,100 and the detection of failures in 6 industrial web APIs. Ex-101 perimental results show that ARTE can generate meaningful102 valid inputs for 64.9% of the target API parameters (137 out103 of 211), outperforming SAIGEN (31.8%). More importantly,104 ARTE supported the generation of over twice as many valid105 API calls (57.3%) as random generation (20%) and SAIGEN106 (26%). As a result, ARTE achieved higher coverage and107 revealed more failures in more APIs, detecting confirmed108 bugs in the web APIs of Amadeus [6] and DHL [17] not109 detected by related techniques. These results show the po-110 tential of ARTE to enhance current specification-driven web111 API testing tools.112 To summarize, after introducing the background on113 RESTful APIs and the Web of Data (Section 2), this paper114 provides the following original research and engineering115 contributions:116 •ARTE, a novel approach for the automated extraction of117 realistic test inputs for web APIs from knowledge bases118 like DBPedia (Section 3).119 •Integration of ARTE into the open-source testing frame-120 work RESTest, making our approach readily applicable121 in practice (Section 4).122 •An empirical comparison of ARTE with random data123 generation techniques and the state-of-the-art approach124 SAIGEN [14] on the generation of realistic test inputs 125 and its impact on testing real-world APIs (Section 5). 126 •A publicly available replication package [23] containing 127 the source code and datasets discussed in the article. We 128 trust that this will also serve as a benchmark for future 129 contributions in the topic. 130 We discuss the threats to validity in Section 6, related 131 work in Section 7, and conclude the paper in Section 8. 132 2 BACKGROUND 133 This section introduces key concepts to contextualize our 134 proposal, namely, RESTful APIs and the Web of Data. 135 2.1 RESTful APIs 136 Modern web APIs typically follow the REpresentational 137 State Transfer (REST) [3] architectural style, being referred 138 to as RESTful web APIs [1]. RESTful web APIs are usually 139 decomposed into multiple RESTful web services [2], each of 140 which implements one or more create, read, update, and 141 delete (CRUD) operations on a resource (e.g., a playlist 142 in the Spotify API [24]). These operations can be invoked 143 by sending specific HTTP requests to specific API end- 144 points. For example, a POST HTTP request to the URI 145 https://api.spotify.com/v1/users/42/playlists would 146 create a playlist for the user with ID 42 in the Spotify API. 147 RESTful APIs can be described in the OAS language [9], 148 arguably the current industry standard. Listing 1 depicts 149 an excerpt of the OAS specification of the DHL API [17]. 150 As illustrated, an OAS document describes the API mainly 151 in terms of the operations supported, as well as their in- 152 put parameters and the possible responses. The operation 153 shown in Listing 1 allows to search for DHL service point 154 locations (lines 1-5) in JSON format (lines 6-7). The opera- 155 tion receives nine input parameters (lines 8-43). Successful 156 responses should include a 200 status code and the set of 157 results matching the input filters (lines 44-48). Note that it 158 is possible to specify constraints such as regular expressions 159 for strings and min/max values for numbers (line 27). 160 Motivated by their critical role in software integration, 161 many researchers have addressed the challenge of auto- 162 matically generating test cases for RESTful web APIs [11], 163 [12], [22], [25], [26], [27], [28]. Most approaches in this 164 domain follow a black-box strategy, where test cases are 165 automatically derived from the API specification, typically 166 in OAS format [11], [12], [22], [25], [27]. Roughly speaking, 167 these approaches generate (pseudo-)random API calls by 168 assigning values to the API input parameters, and then 169 checking whether the API responses conform to the API 170 specification. For the generation of test inputs, some authors 171 resort to random values (fuzzing) [27] or default values [12], 172 but these are rarely enough to test the APIs thoroughly. 173 Hence, most approaches resort to data dictionaries: sets of 174 predefined input values [10]. For example, we could create 175 a list of valid postal codes from which to select test inputs 176 for the parameter postalCode in DHL (Listing 1). However, 177 creating and maintaining data dictionaries for each non- 178 trivial input parameter is a costly manual endeavour—this 179 is the problem that motivates our work. 180 3 1paths: 2’/find-by-address’: 3get: 4operationId: findByAddress 5description: Find DHL locations based on an address. 6produces: 7- application/json 8parameters: 9-name: countryCode 10 in: query 11 description:’A two-letter ISO 3166-1 alpha-2 code.’ 12 required:true 13 type: string 14 x-example:’DE’ 15 -name: postalCode 16 in: query 17 description:’Postal code for an address.’ 18 required:false 19 type: string 20 x-example:’53113’ 21 -name: limit 22 in: query 23 description:’Maximum of results to return.’ 24 required:false 25 type: number 26 default:15 27 maximum:50 28 x-example:20 29 -name: locationType 30 in: query 31 description:’Type of the DHL Service Point location.’ 32 required:false 33 type: string 34 enum: 35 - servicepoint 36 - locker 37 - postoffice 38 - postbank 39 -name: streetAddress ... 40 -name: serviceType ... 41 -name: radius ... 42 -name: addressLocality ... 43 -name: providerType ... 44 responses: 45 ’200’: 46 description:’List of DHL Service Point locations.’ 47 schema: 48 $ref:’#/definitions/supermodelIoLogisticsPUDOLocations’ Listing 1. OAS excerpt of the DHL API. 2.2 Web of Data181 The Web of Data is a global data space in continu-182 ous growth that contains billions of interlinked queryable183 data published following the Linked Data principles [29].184 According to these principles, resources are identified185 using Uniform Resource Identifiers (URIs) [30] and re-186 source relationships are specified using the Resource De-187 scription Framework (RDF) [31]. RDF is a standard that188 specifies how to identify relationships between resources189 in the form of triples composed by a subject, a pred-190 icate, and an object, denoted as <subject, predicate, ob-191 ject>. The predicate specifies the relationship (or link) that192 holds between the subject and object entities. For exam-193 ple, the triple <http://dbpedia.org/resource/George_R._R.194 _Martin,http://dbpedia.org/ontology/author,http://dbpedia.195 org/resource/A_Game_of_Thrones> indicates that George196 R.R. Martin (subject) is the author (predicate) of “A Game197 of Thrones” (object). Subjects and predicates are URIs repre-198 senting the entities and link types, respectively. Objects can199 be either resource URIs or literals, i.e., data values.200 SPARQL [32] is a query language aimed at performing201 queries to datasets represented as RDF triples. Knowledge202 bases [33] like DBpedia [15] and Wikidata [34] consist of203 RDF graphs where triples generated from various sources204 are interlinked and can be explored using SPARQL queries. 205 Listing 2 shows a sample SPARQL query to search for 206 book titles with their corresponding ISBN and number of 207 pages. The clause FILTER(condition) is used to restrict 208 the results to those satisfying the given Boolean condition. 209 This clause can be used, for example, to obtain values that 210 match a regular expression or arithmetic conditions (such as 211 minimum or maximum values). In the example, only data 212 belonging to entities that contain the target predicates (title, 213 ISBN, and number pages) matching the regular expression 214 (codes consisting of 5 groups of an undefined number of 215 digits separated by ‘-’) and with 100 or more pages will be 216 returned. 217 218 1SELECT DISTINCT ?title ?pages ?isbn WHERE {219 2?subject <http://dbpedia.org/property/title> ?title ;220 3<http://dbpedia.org/ontology/numberOfPages> ?pages ;221 4<http://dbpedia.org/ontology/isbn> ?isbn .222 5FILTER (?pages >= 100) 223 6FILTER regex(str(?isbn), ’^([0-9]*[-| ]){4}[0-9]*$’) 224 7}225 226 Listing 2. SPARQL query to search for book titles with their ISBN and number of pages. 227 228 3 ARTE 229 In this section, we present ARTE, an approach for the 230 Automated generation of Realistic TEst inputs for web APIs. 231 Specifically, ARTE leverages the information in the API 232 specification to search for syntactically and semantically 233 valid test data in knowledge bases like DBpedia. We define 234 syntactically and semantically valid inputs as follows: 235 Definition 1 (Syntactically valid input).An input value 236 is syntactically valid if it satisfies the syntactic constraints 237 defined in the API specification and it is accepted by the 238 API under test without returning an error. For example, 239 “Germany” is not a syntactically valid value for the param- 240 eter countryCode in the API of DHL (Listing 1) because, 241 although it conforms to the specification (string value), the 242 API only accepts two-letter ISO 3166-1 alpha-2 codes, as 243 explained in the description of the parameter. Conversely, 244 the value “DE” (ISO 3166-1 alpha-2 code for Germany) is 245 syntactically valid. 246 Definition 2 (Semantically valid input).An input value 247 is semantically valid if it is coherent with the API domain. 248 For example, “Berlin” is a semantically valid test input 249 value for the parameter addressLocality in the API of DHL 250 (Listing 1), whereas “dog” is not. 251 A value can be syntactically valid, but semantically 252 invalid, and vice versa. For example, “dog” is a syntactically 253 valid value for the parameter addressLocality in the API of 254 DHL—it conforms to the specification (string value) and is 255 processed by the API without errors—but it is not coherent 256 with the semantics of the parameter. Conversely, “DEU” 257 (ISO 3166-1 alpha-3 code for Germany) is a semantically 258 valid value for the parameter countryCode, but it is not 259 syntactically valid, since the API expects two-letter codes, 260 i.e., “DE” for Germany. 261 In what follows, we describe the main steps of ARTE, 262 highlighted in Figure 1. A more formal description of our 263 approach using pseudocode is provided as supplemental 264 material. 265 4 Predicates Generate regex? Search for test inputs 3 Search for additional values? Yes No Curated parameter names Processing test parameters 1 Search for predicates 2 Output Test input values Regex Automated generation of regular expressions 4 Yes No Input OpenAPI Specification Target parameters Fig. 1. Workflow of ARTE. 3.1 Processing test parameters266 In this first step, the information of the test parameters is col-267 lected from the input API specification including their name,268 description and, if available, extra syntactic constraints like269 regular expressions and min/max values. As an additional270 input, the user must specify the list of parameters for which271 ARTE should generate meaningful data. This is helpful to272 exclude trivial parameters like dates, or domain-specific273 identifiers (e.g., a YouTube video ID), unlikely to be found274 in general-purpose knowledge bases like DBpedia. For such275 parameters, a different test data generation strategy may be276 used (e.g., data dictionaries).277 Some popular APIs include parameters with a single278 character in their names. This is the case, for example, of279 the Open Movie Database (OMDb) API [35] and the dblp280 computer science bibliography API [36]. When this happens,281 it is very common that this single character is the first282 letter of the full name of the parameter, e.g., ‘t’ for the283 movie title in the OMDb API. Based on this idea, ARTE284 tries to infer the full parameter name from its description285 using natural language processing (NLP) techniques [37],286 [38]. After using part-of-speech tagging [39], stop words287 removal [40], and lemmatization [41], we are left with only288 the nouns included in the description. We observed that289 the full parameter name often matches the shortest noun290 from the list that begins with the same letter as the one-291 character original name of the parameter e.g., ‘q’ for search292 query in the DBLP search API [36] given the description “The293 query string to search for”. Thus, we used this as the default294 heuristic for selecting the full parameter name from the list295 of nouns. When multiple candidate words have the same296 length, ARTE selects the first one in alphabetical order, and297 if no candidate word is found, the letter is used as keyword.298 Implementing other heuristics would be straightforward.299 3.2 Search for predicates 300 In this step, a user-defined knowledge database (e.g., DBpe- 301 dia) is queried to obtain predicates that are representative 302 of the target input parameters. For each target parameter, 303 the search for predicates is performed in two iterative steps. 304 First, a representative keyword of the target parameter is 305 generated from the name and the description of the pa- 306 rameter, by applying the matching rules presented later 307 on in this section. Second, for each candidate keyword, a 308 SPARQL query is constructed as shown in Listing 3, where 309 the string keyword is replaced by the selected keyword. This 310 query conducts a search for predicates (line 2) that contain 311 the provided keyword (filter by regular expression, line 3, 312 where the flag ’i’ means “case-insensitive”) and ordered 313 by length in ascending order (line 4). 314 315 1SELECT DISTINCT ?predicate WHERE {316 2?predicate ardf:Property 317 3FILTER regex(str(?predicate), ’keyword’, ’i’) 318 4} ORDER BY strlen(str(?predicate)) 319 320 Listing 3. SPARQL query used to search for predicates. After executing the query, if successful, a list of predi- 321 cates containing the keyword is returned. As an example, 322 the following are a subset of the predicates found in DBpe- 323 dia when using the keyword “currency”: 324 •http://dbpedia.org/property/currency 325 •http://dbpedia.org/ontology/currency 326 •http://dbpedia.org/property/currencyIso 327 •http://dbpedia.org/ontology/currencyCode 328 Predicates are ordered by length in ascending order, with 329 all of them containing the exact keyword. ARTE selects the 330 first npredicates returned by the query (5 in our experi- 331 ments), computing the support of each of them. Similarly 332 to Mariani et al. [13], we define the support of a predicate 333 as the number of unique RDF triples that contain it. The 334 support of a predicate is obtained by running a SPARQL 335 query as the one shown in Listing 2 including the COUNT 336 set function (see supplemental material for an example). 337 This query will include filters with the syntactic constraints 338 of the API specification (regex and min/max values), if any, 339 to exclude invalid values. A predicate is accepted only if its 340 support is greater than a user-defined threshold (20 in our 341 evaluation). 342 To identify relevant keywords, several matching rules are 343 applied to the name and description of the parameters. 344 These rules are ordered by priority, meaning that, as soon 345 as a keyword is found with a rule, it is used to search 346 for predicates. If a predicate is accepted (i.e., its support 347 is greater than the configured threshold), the search for 348 predicates for that parameter terminates, and the process 349 starts over again for the next parameter, otherwise ARTE 350 tries to identify new keywords with the remaining rules. In 351 our preliminary experiments, we found that the description 352 of the parameter, and not only its name, often includes key 353 information for the generation of meaningful test inputs. 354 Therefore, the matching rules implemented in ARTE exploit 355 the description of the parameter first (rules 1-3 below) and 356 then its name (rules 4-6). 357 In what follows, we describe the matching rules cur- 358 rently used in ARTE, ordered from highest to lowest pri- 359 ority: 360 5 1) If the name of the parameter appears in its description361 followed by the words “code” or “id”, both words are362 used concatenated. For example, if the parameter name363 is country and its description is “A valid country code”,364 this rule will be matched and the keyword countryCode365 will be used for the search for predicates.366 2) If a word Kwith the same initial characters as the name367 of the parameter appears in the description followed by368 the words “code” or “id”, both the name of the param-369 eter and Kare concatenated with “code” or “id” and370 used for the search of predicates. For example, if the371 parameter name is lang and its description is “A lan-372 guage code”, the keywords langCode and languageCode373 will be used in the search for predicates.374 3) If a noun or an unknown word (e.g., an acronym or375 a non-English word) is found in the description of376 the parameter, followed by the words “code” or “id”,377 both words are concatenated and used for the search of378 predicates. If multiple matches are found, the keywords379 are considered in order of appearance. For example,380 if the parameter name is origin and its description is381 “A valid country code or airport code”, the keywords382 countryCode and airportCode will be used for the383 search for predicates.384 4) This rule has no condition. Whenever it is reached, the385 unmodified parameter name is used for the search for386 predicates, e.g., streetAddress (Listing 1).387 5) If the name of the parameter is in snake case format388 (i.e., word1_word2) or in kebab case format (i.e., word1-389 word2), the parameter name is converted to camel case390 format (i.e., word1Word2) and used in the search for391 predicates. For example, the operation for matching392 businesses in the Yelp API [42] contains a parameter393 called zip_code; using this as a keyword (rule 4) re-394 turns 0 candidate predicates in DBpedia, whereas using395 zipCode returns a list of 5 predicates.396 6) If the parameter name is in snake case, kebab case or397 camel case format, it is split into multiple words, and398 each one is used as a keyword to search for predicates.399 For example, several operations of the Great Circle400 Mapper API [43] contain a parameter called icao_iata,401 which accepts both IATA and ICAO codes. Using the402 unmodified parameter name (rule 4) produces 0 results,403 and so does converting it to camel case format (rule 5).404 However, when searching for predicates with the key-405 words icao and iata, multiple predicates are obtained.406 Matching rules 4 to 6 are based on the common naming407 convention for web API parameters, whereas rules 1 to 3408 are based on collocations we found during our preliminary409 work with 10 randomly selected APIs from different do-410 mains [23] (not included in our evaluation dataset to avoid411 overfitting). We remark that the list of matching rules is not412 exhaustive and new patterns could be readily targeted in413 the future by adding new rules. If no predicates are found414 for any of the matching rules, the parameter is ignored and415 no input values will be generated for it.416 3.3 Search for test inputs417 At this stage, the predicates obtained during the previous418 phase are used to drive the search for meaningful test419 subject title director dbp:title Results: 23 langCode dbp:langCode dbp:director dbp:langCode dbp:title Results: 65 Results: 392 Results: 170,271 discard dbp:langCode discard dbp:director discard dbp:title dbp:title dbp:langCode dbp:director dbp:director Fig. 2. Graphical representation of the search for test inputs. inputs. When searching test inputs for a specific parame- 420 ter, it is important to consider it in the context of all the 421 parameters supported by the API operation. For example, 422 a search for title may return hundreds or even thou- 423 sands of title-type inputs belonging to very diverse entities 424 (e.g., movies, people, videogames, books, etc.). However, if 425 we accompany such predicate with another semantically 426 related one such as director, the search is significantly 427 narrowed, increasing the chances of generating semantically 428 valid test inputs, e.g., movie titles. For this reason, instead 429 of evaluating the predicates in isolation, our approach starts 430 by looking for entities that contain all the predicates simul- 431 taneously and, if not enough inputs are obtained, then the 432 predicates are progressively discarded. 433 As an example, Listing 4 shows the SPARQL query 434 constructed for the search of test inputs (i.e., objects in RDF 435 triples) for the parameters title,director and langCode.436 This query returned 23 matches in DBpedia. Each match 437 contains three values, one for each parameter (i.e., title,438 director and langCode). However, some values may be 439 repeated, therefore less than 23 unique values could be 440 obtained for each parameter. A threshold is established to 441 define the minimum number of unique values that should 442 be obtained (100 in our experiments). Suppose that, for a 443 subset of parameters, the threshold is not reached (e.g., 444 there are enough values for langCode, but not for director 445 and title). Then, ARTE would repeat the query depicted 446 in Listing 4 considering only the predicates related to the 447 parameters for which the threshold has not been achieved 448 (i.e., director and title), thus widening the search. 449 450 1SELECT DISTINCT ?title, ?director, ?langCode WHERE {451 2?subject <http://dbpedia.org/property/title> ?title ;452 3<http://dbpedia.org/property/director> ?director ;453 4<http://dbpedia.org/property/langcode> ?langCode .454 5}455 456 Listing 4. SPARQL query for searching test inputs. Figure 2 (left-hand side) depicts an example for the query 457 in Listing 4. A search is performed for entities including 458 the predicates director,title, and langCode, obtaining 23 459 results. Suppose that such search does not return the mini- 460 mum threshold for any of the three parameters. This being 461 the case, ARTE would execute a number of queries equal 462 to the number of parameters, each containing all predicates 463 but one (right-hand side of Figure 2). As illustrated, if title 464 or director are discarded, 65 and 392 results are returned, 465 respectively, but if langCode is discarded, the resulting 466 query would return 170,271 results. Values for the title 467 6 TABLE 1 Example of automatically generated regular expression. Valid values Invalid values Regex CN ITA ^\w[^\d]$ ES DOM IT BGR JP 92 AD HUN and director parameters would be extracted from this last468 query (since it is the one returning the highest number of469 results), while values for the langCode parameter would be470 obtained as a result of using its predicate in isolation.471 This query-decomposition process continues until all pa-472 rameters acquire the established threshold of input values,473 or until all predicates are executed in isolation.474 3.4 Automated generation of regular expressions475 Finally, ARTE can optionally generate regular expressions476 for each target parameter based on the input values obtained477 in the previous phases, by integrating the tool RegexGener-478 ator++ [44], [45], [46] (details in Section 4). This allows to479 refine the search for syntactically valid inputs, improving480 the effectiveness of the approach. Table 1 depicts an example481 of this process for the parameter market in the Spotify API482 [24]. Specifically, the regular expression in Table 1 matches483 strings that are two-characters long, where the second char-484 acter is not a digit; this pattern conforms to the format of485 ISO alpha-2 country codes, and allows to filter out values486 of different length (e.g., “ITA”) and numerical values (e.g.,487 “92”). The basic steps of the process are as follows:488 1) A set of input values is generated as described in the489 previous sections, and classified as syntactically valid490 or invalid (first and second columns in Table 1). Inputs491 can be classified as valid or invalid either manually492 or, as in our work, automatically based on the API re-493 sponses (see Section 4 for details). The number of valid494 and invalid values must reach a minimum threshold495 defined by the user (5 in our experiments).496 2) Based on the generated valid and invalid inputs, a497 regular expression is generated (column “Regex” in498 Table 1). This regular expression should match and not499 match as many valid and invalid inputs as possible,500 respectively.501 3) If the percentage of valid values matching the regular502 expression (i.e., recall) exceeds a minimum threshold503 (90% in our evaluation), the generation of the expres-504 sion is considered successful. In that case, the list of505 input values generated until that moment is filtered506 such that only those values matching the regular ex-507 pression are kept. Invalid values matching the regular508 expression (i.e., false positives) may still be helpful509 to identify potential bugs. For instance, the regular510 expression generated for the parameter countryCode in511 the DHL API matched both “UK” and “FR”, however,512 the former was rejected by the API (i.e., classified as513 invalid). This may reveal unintended behavior, e.g.,514 codes that should be supported by the API but are not.515 If a regular expression is successfully generated, ARTE516 extracts more input values by using the predicates that have517 TABLE 2 ARTE configuration parameters and default values. # Parameter Value 1 Number of predicates selected when searching for predicates 5 2 Min predicate support 20 3 Min number of unique parameter values 100 4 Min recall for accepting a regex 90% 5 Min number of valid and invalid values for generating a regex 5 6 Max number of extra predicates to leverage using regex 3 7 Max number of consecutive attempts to generate a regex 2 not been leveraged yet (obtained as described in Section 3.2) 518 adding the generated regular expression as a filter clause 519 in the SPARQL query (Listing 4). These inputs will match 520 the regular expression and therefore they should be more 521 likely to be syntactically valid. On the contrary, if a regular 522 expression cannot be inferred, the process restarts in step 523 1, i.e., ARTE will try to generate a regular expression with 524 more input values classified as valid or invalid. 525 The process ends once the maximum number of consec- 526 utive attempts to generate regular expressions is reached (2 527 in our evaluation), or after ARTE has extracted input values 528 from a maximum number of extra predicates leveraging the 529 generated regular expressions (set to 3 in our experiments). 530 4 TOOLING 531 We implemented ARTE in Java, leveraging existing libraries 532 for specific tasks, namely: (1) Jena [47], for the creation 533 of SPARQL queries; (2) Stanford CoreNLP [48], for NLP 534 related tasks; and (3) RegexGenerator++ [44], [45], [46], for 535 the generation of regular expressions. RegexGenerator++ 536 uses search-based techniques for automatically generating 537 context-aware regular expressions based on strings tagged 538 as valid or invalid (i.e., matching or not matching the regular 539 expression, respectively) within a corpus, namely, a text 540 that provides some context. We slightly modified Regex- 541 Generator++ to make it not context-aware, by adding the 542 anchors ‘ˆ’ and ‘$’ at the start and end of the generated regu- 543 lar expressions, respectively. This modification significantly 544 improves the performance of ARTE for the generation of 545 regular expressions. 546 Table 2 summarizes the configuration parameters of 547 ARTE, described in the previous section, and the default 548 values used in our evaluation. These values were selected 549 based on our preliminary work with the tool. They provide 550 a balance between performance and effectiveness. Hence, 551 for example, setting low values for parameters 1-3 would 552 result in faster execution, but lower probabilities of finding 553 good values. Analogously, setting low values for parameters 554 4 and 5 would make it easier for ARTE to generate a regular 555 expression, with the risk of it not being sufficiently accurate. 556 On the contrary, setting a high value for parameters 6 and 7 557 would result in a slower execution with the risk of obtaining 558 overfitted regular expressions. We refer the reader to the 559 documentation of the project in GitHub [49] for more details 560 about the parameters and their impact on the performance 561 of ARTE. 562 We integrated ARTE into RESTest [22], a state-of-the- 563 art black-box test case generation framework for RESTful 564 APIs. More precisely, ARTE automatically generates data 565 7 dictionaries (i.e., sets of valid and invalid values) for the566 selected API parameters, releasing testers from that burden.567 This makes ARTE easily applicable to any of the test case568 generation algorithms implemented in RESTest.569 For the generation of regular expressions, input values570 must be labeled as valid or invalid. By integrating ARTE571 into RESTest, we automate this step as follows:572 •An input parameter value is classified as valid if it was573 present in an API call for which a successful response574 (2XX status code) was obtained.575 •An input parameter value is classified as invalid if it576 was present in an API call for which a “client error”577 response (4XX status code) was obtained and either: (1)578 it was the only parameter used in the API call; or (2)579 it was accompanied by other parameters whose values580 were already classified as valid.581 •Input values not meeting any of the previous con-582 straints are labeled as unclassified and ignored when583 inferring regular expressions.584 RESTest generates and executes a fixed number of test585 cases by iterations. After each iteration, ARTE automatically586 classifies the generated values as described above and tries587 to infer a regular expression (Section 3.4).588 5 EVALUATION589 We aim to answer the following research questions:590 •RQ1: How effective is ARTE in generating realistic test591 inputs for web APIs? We aim to measure the effectiveness592 of ARTE in generating syntactically and semantically593 valid test inputs for real-world web APIs.594 •RQ2: What is the impact of ARTE on the automated gen-595 eration of test cases for web APIs? The final goal of our596 approach is to improve current state-of-the-art methods597 for test case generation of web APIs. Therefore, we wish598 to compare the effectiveness of existing methods and599 ARTE in terms of valid API requests generated and test600 coverage achieved.601 •RQ3: Does ARTE improve the failure-detection capability602 of existing test case generation techniques? We aim to603 investigate whether input values generated by ARTE604 reveal more failures than those generated by related605 techniques.606 In the next sections, we explain the two experiments607 performed for answering the research questions. In both608 cases, we used the default configuration of ARTE, and we609 relied on DBpedia as the selected knowledge base, more610 specifically, the 2016-10 core dataset.1 611 The experiments were performed in a desktop machine612 equipped with Intel i7-6700 [email protected], 16GB RAM, and613 125GB SSD running Windows 10 Pro 64 bit and Java 8.614 5.1 Baselines615 We compared ARTE against four baselines, three (pseudo)-616 random test data generation strategies (experiment 2) and617 SAIGEN [14] (experiments 1 and 2), described below.618 1. http://downloads.dbpedia.org/2016-10/ 5.1.1 Random test data generation techniques 619 We implemented three related, but different random test 620 data generation approaches recently used in the context of 621 automated test case generation for web APIs. All the gen- 622 erators were implemented using RESTest. Next, we present 623 them, from the most naive to the most sophisticated one. 624 5.1.1.1 Fuzzing: This technique aims at finding im- 625 plementation bugs (especially security-related) by using 626 random, malformed or unexpected input data [50]. In our 627 experiments, we use the fuzzing test case generator imple- 628 mented in RESTest [22], which generates random values 629 including out-of-range numerical values, long strings with 630 unusual characters, empty strings, and null values. 631 5.1.1.2 Data dictionaries: This approach proposes to 632 use a small set of predefined values for each parameter type. 633 Specifically, we used the small data dictionaries proposed by 634 Atlidakis et al. [27], namely: “sampleString” and “” (empty 635 string) for string parameters, “0” and “1” for integers, and 636 “1.23” for doubles. 637 5.1.1.3 Data generators: This approach uses specific 638 test data generators for each parameter type [22], [25]. 639 We used the default test data generators integrated into 640 RESTest, which also leverage the regular expressions and 641 min/max constraints included in the API specification, if 642 any. Basically, we generated random English words for 643 string parameters, numbers between 1 and 100 for integers, 644 and floating numbers between 1 and 100 for doubles. 645 5.1.2 SAIGEN 646 To the best of our knowledge, our work is the first to lever- 647 age the Web of Data for driving test data generation for web 648 APIs. However, semantic information retrieval techniques 649 have already been successfully applied in the context of GUI 650 testing of desktop, web, and mobile applications [13], [14]. 651 In this article, we compare ARTE against the most recent of 652 these contributions, SAIGEN (Semantic Aware Input GEN- 653 erator) [14], a related tool for the automated generation of 654 realistic test inputs for mobile apps. 655 Both approaches, ARTE and SAIGEN, exploit knowl- 656 edge bases for the generation of realistic test inputs, but 657 with significant differences. First, SAIGEN searches for test 658 inputs based on the GUI labels and potential synonyms, 659 whereas ARTE exploits the API specification, applying NLP 660 techniques to the names and descriptions of the parameters. 661 This means that ARTE can leverage further information 662 than SAIGEN, but in practice this also imposes new chal- 663 lenges since, as explained in Section 3, parameters can 664 use different naming conventions (e.g., parameters with a 665 single letter) and very heterogeneous descriptions in nat- 666 ural language. Second, SAIGEN searches for predicates by 667 adding the selected keyword to the namespace prefix of the 668 knowledge base, whereas ARTE constructs specific SPARQL 669 queries. In practice, this means that searches in SAIGEN 670 are restricted to predicates containing exactly the specified 671 keyword (e.g., currency), whereas ARTE widens the search 672 space by looking for predicates containing the keyword 673 (e.g., currencyIso). Lastly, ARTE integrates a novel step for 674 the refinement of the test inputs extracted with automati- 675 cally generated regular expressions (Section 3.4). 676 Since SAIGEN targets GUI elements only, in our experi- 677 ments we ran the tool using the name of the API parameters 678 8 as if they were GUI labels. We used the default settings of679 SAIGEN, configuring it to work with the selected version of680 the knowledge base.681 5.2 Experiment 1: Generation of realistic test inputs682 In this experiment, we aim to answer RQ1 by evaluating the683 effectiveness of ARTE in generating realistic test inputs for684 real-world web APIs. In what follows, we describe the setup685 and the results of the experiment.686 5.2.1 Experimental setup687 For this experiment, we resorted to RapidAPI [51], a pop-688 ular online repository containing more than 35K web APIs,689 classified in 46 different categories. Specifically, we created a690 dataset of 40 APIs as follows. We selected the first 10 APIs of691 each category, i.e., 46 ×10 =460 APIs. APIs and categories692 were sorted in the same order as they were displayed on693 the platform at the time of performing the search on May,694 2021. Then, we selected the first 40 APIs of the list, filtering695 out those meeting any of the following exclusion criteria:696 (1) APIs containing exclusively domain-specific parameters697 (e.g., database IDs) or trivial parameters (such as limit,698 offset or enumerated values), for which random generation699 would suffice, (2) APIs for which there was no validation for700 any of the parameters (i.e., any value would be considered701 valid), (3) APIs containing only confidential parameters702 (e.g., username, password, email, or credit card number), (4)703 APIs not returning any response, (5) APIs without param-704 eters, (6) paid APIs (so-called premium in RapidAPI), and705 (7) APIs whose documentation was not written in English.706 The resulting dataset contains 40 APIs from 11 different707 categories. Table 3 shows, for each API, its name, category,708 number of operations, and number of parameters used in709 our evaluation. Parameters used in more than one operation710 within the same API are considered only once, e.g., iso_a2711 in the Referential API [52]. In total, the dataset includes712 173 different parameters from 122 API operations. These713 parameters span multiple concepts such as website URLs,714 city names, ingredient names, and currency codes, among715 many others.716 RapidAPI does not provide a publicly available OAS717 specification for the APIs in the repository. To address this718 issue, we generated the OAS specification of each API using719 the web scraping libraries Selenium [53] and Beautiful-720 Soup [54] on the web user interface of RapidAPI, which721 displays, among others, the name and type of each input722 parameter for every API operation.723 To further evaluate our approach, we used a second724 dataset of 8 industrial popular RESTful APIs from differ-725 ent domains, depicted in Table 4. The dataset includes 38726 parameters from 18 API operations. The OAS specification727 of Spotify [24] was downloaded from the APIs.guru repos-728 itory [55]. The specifications of Yelp Fusion [42] and REST-729 Countries [56] were manually created from the documenta-730 tion available on the official website. The OAS specification731 of the remaining APIs were downloaded from their official732 websites. The specification of the Amadeus Hotel API [6]733 was the only one including regular expressions for some of734 its parameters.735 We compared ARTE against SAIGEN, since it is the 736 only baseline specifically tailored for the automated gen- 737 eration of realistic test inputs. Additionally, we measured 738 the performance of ARTE before applying the refinement 739 with automatically generated regular expressions, denoted 740 as ARTE NR (ARTE No Regex). 741 ARTE and SAIGEN may generate hundreds or even 742 thousands of values for each parameter, making it infea- 743 sible to manually test all of them. To evaluate whether 744 syntactically and semantically valid values were generated, 745 we tested the APIs with 10 randomly selected values per 746 parameter, manually checking whether at least one of them 747 was valid. A similar approach was followed by the authors 748 of SAIGEN [14]. We refer the reader to Section 3 for the 749 definition of syntactically and semantically valid test inputs. 750 5.2.2 Experimental results 751 The results of ARTE and SAIGEN in the RapidAPI dataset 752 are shown in the last nine columns of Table 3. On aver- 753 age, ARTE found syntactically valid inputs for 78% of the 754 parameters (135 out of 173), whereas SAIGEN generated 755 syntactically valid inputs for 42.8% of them (74 out of 173). 756 Analogously, ARTE generated semantically valid inputs for 757 64.2% of the parameters (111 out of 173), while SAIGEN 758 generated semantically valid inputs for 31.8% (55 out of 759 173). Regarding realistic values (both syntactically and se- 760 mantically valid), ARTE generated realistic values for 63% 761 of the parameters (109 out of 173), whereas SAIGEN only 762 generated realistic values for 30.6% of them (53 out of 173). 763 ARTE generated realistic inputs for 100% of the parameters 764 in 9 out of 40 APIs (4 with SAIGEN), and 50% or more 765 in 23 of them (13 with SAIGEN). It is noteworthy that the 766 generation of regular expressions allowed to increase the 767 percentage of realistic test inputs in 11 out of the 25 APIs for 768 which ARTE did not initially obtain 100% of syntactically 769 valid parameters. Among others, ARTE automatically gen- 770 erated regular expressions for parameters such as season 771 (API-FOOTBALL and API-BASKETBALL APIs), currency 772 (Skyscanner and Alpha Vantage APIs) and state (RedLine 773 and Realty Mole APIs). 774 Table 4 shows the results for the set of industrial APIs, 775 where the results of ARTE are even better than those ob- 776 tained with the RapidAPI dataset. As illustrated, ARTE 777 generated syntactically and semantically valid inputs for 778 73.7% of the parameters (28 out of 38), in contrast with 779 SAIGEN, which generated realistic inputs for only 36.8% of 780 the parameters (14 out of 38). Furthermore, ARTE generated 781 realistic inputs for 100% of the target parameters in 3 out 782 of 8 APIs (DHL, OMDb and RESTcountries), and over 50% 783 in 6 of them, whereas SAIGEN did not manage to generate 784 realistic inputs for all the parameters in any of the APIs, 785 generating 50% or more in 3 of them (DHL, RESTCountries 786 and Yelp). The automated generation of regular expres- 787 sions yielded an improvement for the parameter currency 788 of the RESTCountries API. Additionally, it improved the 789 automated generation of valid test cases in the APIs of 790 RESTCountries, Spotify and DHL, as explained in Section 791 5.3 792 In total, considering both the RapidAPI dataset and 793 the industrial APIs, ARTE and SAIGEN generated realistic 794 inputs for 64.9% (137 out of 211) and 31.8% (67 out of 211) 795 9 TABLE 3 Per API breakdown of input values generation for the RapidAPI dataset. O = Operations, P = Parameters. API Category O PSyntactically valid (%) Semantically valid (%) Syntactically and semantically valid (%) SAIGEN ARTE NR ARTE SAIGEN ARTE NR ARTE SAIGEN ARTE NR ARTE AeroDataBox Transportation 6 8 37.5 62.5 75 37.5 62.5 75 37.5 62.5 75 Airport info Transportation 1 2 50 100 100 50 100 100 50 100 100 AirportIX Transportation 6 6 33.3 16.7 50 16.7 0 33.3 16.7 0 33.3 Alpha Vantage Finance 3 5 0 0 40 0 0 40 0 0 40 API-BASKETBALL Sports 5 6 83.3 83.3 100 33.3 16.7 33.3 33.3 16.7 33.3 API-FOOTBALL Sports 5 4 50 75 100 50 50 75 50 50 75 Astronomy Science 1 2 100 100 100 100 100 100 100 100 100 Aviation Reference Data Transportation 5 6 66.7 66.7 83.3 66.7 66.7 83.3 66.7 66.7 83.3 CarbonFootprint Science 1 3 33.3 33.3 33.3 33.3 33.3 33.3 33.3 33.3 33.3 Countries Cities Location 4 4 75 100 100 75 100 100 75 100 100 Currency Converter Finance 2 4 50 75 75 0 50 50 0 50 50 Domainr Business 2 3 66.7 66.7 66.7 0 33.3 33.3 0 33.3 33.3 Face Detection Visual Recognition 1 1 0 0 0 0 0 0 0 0 0 Fixer Currency Finance 3 4 0 100 100 0 100 100 0 100 100 Football Prediction Sports 1 1 0 0 0 0 0 0 0 0 0 GeoDB Cities Data 4 8 0 62.5 75 0 62.5 75 0 62.5 75 Google Maps Geocoding Location 2 3 66.7 66.7 66.7 33.3 33.3 33.3 33.3 33.3 33.3 Great Circle Mapper Travel 3 3 0 33.3 33.3 0 33.3 33.3 0 33.3 33.3 Hotels Travel 3 3 66.7 100 100 0 33.3 33.3 0 33.3 33.3 Hotels Com Provider Travel 1 2 100 100 100 100 100 100 100 100 100 Movie Database (Imdb alt.) Entertainment 2 2 50 100 100 0 50 50 0 50 50 NAVITIME Route (car) Transportation 2 3 0 0 0 0 0 0 0 0 0 NAVITIME Route (totalnavi) Transportation 1 2 0 0 0 0 0 0 0 0 0 Periodic Table of Elements Science 1 1 0 0 0 0 0 0 0 0 0 Priceline com Provider Travel 2 4 50 100 100 50 100 100 50 100 100 Realty in US Business 4 8 62.5 87.5 100 50 75 87.5 50 75 87.5 Realty Mole Property Business 3 7 57.1 71.4 85.7 85.7 85.7 100 57.1 71.4 85.7 Recipe - Food - Nutrition Food 8 7 14.3 71.4 71.4 14.3 85.7 85.7 14.3 71.4 71.4 RedLine Zipcode Location 6 8 25 75 87.5 25 75 87.5 25 75 87.5 Referential Data 8 18 27.8 72.2 72.2 27.8 44.4 44.4 27.8 44.4 44.4 Rent Estimate Data 1 3 100 100 100 100 100 100 100 100 100 Restb.ai Watermark Detection Visual Recognition 1 1 0 100 100 0 100 100 0 100 100 Skyscanner Flight Search Transportation 8 5 40 40 80 0 0 40 0 0 40 Spott Location 1 4 100 100 100 50 50 50 50 50 50 Subtitles for YouTube Data 1 1 100 100 100 100 100 100 100 100 100 TrailAPI Travel 2 5 40 100 100 40 80 80 40 80 80 Travel Advisor Transportation 3 8 50 100 100 25 87.5 87.5 25 87.5 87.5 UPHERE.SPACE Science 2 2 50 50 50 0 0 0 0 0 0 US Restaurant Menus Food 6 5 80 100 100 60 80 80 60 80 80 Yahoo Finance Finance 1 1 0 0 0 0 0 0 0 0 0 Total 122 173 42.8 69.9 78 31.8 56.1 64.2 30.6 54.9 63 TABLE 4 Per API breakdown of input values generation for the industrial APIs. O = Operations, P = Parameters. API O PSyntactically valid (%) Semantically valid (%) Syntactically and semantically valid (%) SAIGEN ARTE NR ARTE SAIGEN ARTE NR ARTE SAIGEN ARTE NR ARTE Amadeus Hotel 2 7 42.9 85.7 85.7 42.9 85.7 85.7 42.9 85.7 85.7 Deutschebahn StaDa 1 4 0 25 25 0 25 25 0 25 25 DHL Location Finder 2 6 50 100 100 50 100 100 50 100 100 Marvel 1 5 60 100 100 40 40 40 40 40 40 OMDb 1 3 33.3 100 100 0 100 100 0 100 100 RESTCountries 4 4 75 75 100 100 100 100 75 75 100 Spotify 5 4 25 75 75 25 75 75 0 50 50 Yelp Fusion 2 5 60 80 80 80 80 80 60 80 80 Total 18 38 44.7 81.6 84.2 44.7 76.3 76.3 36.8 71.1 73.7 of the parameters, respectively. It is worth recalling that796 SAIGEN was specifically designed for mobile apps, and797 therefore its poor performance on web APIs was expected.798 This supports the ability of ARTE to address the specific799 characteristics of web APIs.800 Regarding performance, ARTE took on average 21.3801 seconds (standard deviation σ= 30.6 seconds) to generate802 the set of input values for all the parameters of each API,803 whereas SAIGEN required 11.8 seconds on average per API804 (σ= 11.2 seconds).805 In view of these results, we can answer RQ1 as follows:806 RQ1: ARTE is effective in generating realistic test inputs for real-world web APIs. With a sample of 10 values per parameter, ARTE generated syntactically and semantically valid inputs for 64.9% of the parameters (137 out of 211), approximately twice as many as the baseline, SAIGEN, 31.8% (67 out of 211). 807 5.3 Experiment 2: Automated testing 808 In this experiment, we aim to answer RQ2 and RQ3 by 809 evaluating how ARTE can contribute to the automated 810 generation of valid API calls, API coverage, and detection 811 of failures. Next, we describe the setup and the results of 812 the experiment. 813