Black-Box and White-Box Test Case Generation for RESTful APIs: Enemies or Allies?

Black-Box and Whi e-Box Tes Case Gene a ion o REST ul APIs: Enemies o Allies? Albe o Ma in-Lopez SCORE Lab, I3US Ins i u e Uni e sidad de Se illa Se ille, Spain [email p o ec ed] And ea A cu i K is iania Uni e si y College Oslo Me opoli an Uni e si y Oslo, No way [email p o ec ed] Se gio Segu a SCORE Lab, I3US Ins i u e Uni e sidad de Se illa Se ille, Spain se [email p o ec ed] An onio Ruiz-Co ´ es SCORE Lab, I3US Ins i u e Uni e sidad de Se illa Se ille, Spain [email p o ec ed] Abs ac —Au oma ed es case gene a ion o REST ul APIs is a h i ing esea ch opic due o hei c i ical ole in so wa e in eg a ion. Tes ing app oaches can be di ided in o black-box and whi e-box. Black-box app oaches exploi he API speci ica ion o he gene a ion o es cases, while whi e-box app oaches can also le e age he sou ce code. Bo h s a egies ha e shown g ea p omise, bu hey ha e no been ully compa ed ye , hinde ing he selec ion o he igh ool o he job. In his pape , we epo on ou expe ience compa ing black-box and whi e-box es case gene a ion o REST ul APIs using he s a e- o - he-a ools RESTes (black-box) and E oMas e (whi e-box). Also, we p opose in eg a ing bo h app oaches by using black- box es cases as he seed o whi e-box sea ch-based es case gene a ion. E alua ion esul s on ou REST ul APIs in ol ing o e 40 million API calls show ha he e is no one-size- i s-all s a egy. Mo e impo an ly, he combina ion o black-box and whi e-box yielded he bes esul s in mos case s udies in e ms o code co e age and aul inding, pa ing he way o be e ools in eg a ing he bes o bo h pe spec i es. As a esul o ou wo k, we p o ide lessons lea ned and open challenges o guiding he use and u he de elopmen o cu en ool suppo . Index Te ms—REST, API, web se ice, SBST, seeding I. INTRODUCTION Web Applica ion P og amming In e aces (APIs) allow he - e ogeneous sys ems o in e ac o e he ne wo k [1], [2]. Web APIs a e apidly p oli e a ing as a key elemen o os e so wa e eusabili y, in eg a ion, and inno a ion, enabling new consump ion models such as wea ables and sma home apps. Companies such as Facebook, Twi e , Google, eBay o Ne lix ecei e billions o API calls e e yday om housands o di e en hi d-pa y applica ions and de ices, which cons i- u es mo e han hal o hei o al a ic [1]. Mode n web APIs ypically adhe e o he REp esen a ional S a e T ans e (REST) a chi ec u al s yle [3], being e e ed o as REST ul APIs. REST ul APIs p o ide a s anda d mechanism o imple- men c ea e, e ie e, upda e, and dele e (CRUD) ope a ions o e esou ces (e.g., a YouTube ideo) in a dis ibu ed way. The widesp ead use o REST ul APIs is e lec ed in he size o popula API di ec o ies such as P og ammableWeb [4], cu en ly indexing o e 24K REST ul APIs om domains such as shopping, inances, social, o elephony. Tes ing REST ul APIs is c i ical due o hei key ole in so wa e in eg a ion. A bug in an o ganiza ion’s API could ha e a huge impac bo h in e nally (se ices elying on ha API) and ex e nally ( hi d-pa y applica ions and end use s). To add ess his challenge, nume ous app oaches o es case gene a ion o REST ul APIs ha e eme ged in ecen yea s [5]–[12]. These app oaches can be di ided in o black- box and whi e-box. Black-box app oaches [5], [6], [8]–[12] exploi he speci ica ion o he API, ypically in he OpenAPI Speci ica ion (OAS) [13] o ma , o d i e he gene a ion o es cases in he o m o (pseudo-) andom API calls. They o en use cus om da a gene a o s, manually implemen ed by he use s, based on domain knowledge o he applica ion. Whi e-box app oaches [7] le e age he API speci ica ion and i s sou ce code o gene a e es cases ha co e as much code as possible, ying o minimize he amoun o manual wo k om he use s. Bo h app oaches y o iden i y se e e o s (i.e., 5XX HTTP s a us codes) and misma ches wi h he API speci ica ion. Bo h es case gene a ion s a egies o REST ul APIs, black-box and whi e-box, ha e been la gely e alua ed in isola ion, showing hei own me i s and limi a ions. Howe e , hey ha e no been ully compa ed ye . Thus, many ques ions emain unanswe ed ega ding he scope and pe o mance o each app oach. Besides his, bo h app oaches ha e ollowed comple ely di e en pa hs so a , and hus he po en ial bene i s o hei combina ion emain unexplo ed. In his pape , we epo on ou expe ience compa ing black-box and whi e-box es case gene a ion echniques o REST ul APIs. Speci ically, we use he s a e-o - he-a ools RESTes [14] (black-box) and E oMas e [7] (whi e- box). Addi ionally, we p opose he in eg a ion o bo h ap- p oaches by using black-box es cases—de i ed om he API speci ica ion—as he seed o sea ch-based es case gene a ion—le e aging he sou ce code o he API. We epo on he esul s o an ex ensi e empi ical e alua ion compa ing he e ec i eness and pe o mance o hese h ee app oaches: black-box, whi e-box and hyb id (black-box + whi e-box) on ou open-sou ce REST ul APIs o di e en sizes and complexi y. The esul s show ha ou no el hyb id app oach pe o med bes in mos scena ios. In e ms o aul inding, he hyb id app oach was he only one capable o unco e ing bugs in all sys ems unde es (SUTs). We p o ide an in- dep h discussion and analysis o he esul s ob ained by each echnique, and possible causes leading o such esul s. Ou wo k con ibu es o a be e unde s anding o he s eng hs and 1 limi a ions o black-box and whi e-box es case gene a ion o REST ul APIs, and shows he po en ial o combining bo h s a egies. Howe e , mo e wo k and expe imen a ion will be equi ed in he u u e o complemen ou esul s including he use o di e en es ing echniques, ools, and case s udies. A e explaining he backg ound (Sec ion II) and discussing ela ed wo k (Sec ion III), his pape p o ides he ollowing o iginal esea ch and enginee ing con ibu ions: •A no el hyb id es ing app oach based on he use o black-box es cases as he seed o whi e-box es gene a ion, and i s implemen a ion in E oMas e [7] (Sec ion IV). •A compa ison o black-box, whi e-box, and hyb id ech- niques o es case gene a ion o REST ul APIs in e ms o code co e age and aul inding using wo di e en s a e-o - he-a ools and ou open-sou ce APIs (Sec ion V). •Lessons lea ned om he compa ison o he h ee ech- niques, including p ac ical guidelines o he selec ion o he igh app oach depending on se e al ac o s such as he size o he SUT and he a ailable esou ces (Sec ion VI). •A lis o challenges o he h ee es case gene a ion app oaches unde compa ison de i ed om he esul s o he s udy (Sec ion VII). We add ess h ea s o alidi y in Sec ion VIII and conclude he pape in Sec ion IX. II. BACKGROUND A. REST ul Web APIs REST [3] is a so wa e a chi ec u al s yle o building dis ibu ed sys ems. Mos cu en web se ices ollow he p inciples de ined by REST and a e e e ed o as REST ul (o simply REST) web se ices. REST ul web se ices p o ide a s anda d app oach o in e ac wi h esou ces o e he ne wo k. A esou ce is any piece o da a ha can be exposed o he Web, o example, a documen (Google D i e API [15]), a pic u e (Flick API [16]), o e en a wee (Twi e API [17]). Resou ces can be accessed and manipula ed by means o CRUD ope a ions. These ope a ions can be in oked by sending HTTP eques s o speci ic API endpoin s, each o which is iden i ied by an HTTP me hod (e.g., GET) and a pa h (e.g., /documen s). A REST ul API may be composed o one o mo e REST ul se ices. REST ul web APIs a e usually desc ibed wi h in e - ace desc ip ion languages like he OpenAPI Speci ica ion (OAS) [13]. OAS is hea ily used nowadays o au oma ing ce ain asks in he API li ecycle such as code gene a ion [18], moni o ing [19] and es ing [14]. An OAS documen (also called schema) desc ibes a REST ul API in e ms o he allowed inpu s (HTTP eques s) and he expec ed ou pu s (HTTP esponses). Figu e 1 depic s an ex ac o an OAS schema, aken om LanguageTool, a p oo eading API [20]. As illus a ed, he POST /check endpoin allows o de ec mis akes in a ex (line 4). The speci ica ion de ails he Fig. 1. OAS exce p om he LanguageTool API. in o ma ion abou he ope a ion pa ame e s (lines 7-59), such as hei da a ype (line 57), whe he hey a e equi ed o op ional (line 58) and de aul alues (line 59). Fo e e y ope a ion, he se o expec ed esponses is also desc ibed (lines 65-69), including hei HTTP s a us code (line 66) and hei o ma (line 69). Op ionally, when he ope a ion con ains in e - pa ame e dependencies (e.g., mu ually exclusi e pa ame e s), hese can be speci ied wi h he IDL4OAS ex ension [21] (lines 60-64). Fo example, he inpu ex agmen can be p o ided as a plain s ing using pa ame e ex (line 8) o as a JSON documen using pa ame e da a (line 13), bu only one o hem mus be se (line 61). Acco ding o a ecen s udy by Ma in-Lopez e al. [22], 4 ou o e e y 5 indus ial APIs con ain hese in e -pa ame e dependencies. B. REST ul API Tes Case Gene a ion Tes ing a REST ul API a he sys em le el in ol es gene - a ing HTTP eques s and asse ing hei esponses. A es case comp ises one o mo e eques s. 2 Black-box es ing app oaches le e age he speci ica ion o he API unde es (e.g., an OAS documen ) o au oma ically gene a e es cases. Howe e , he speci ica ion may no su ice o gene a e ealis ic es inpu s, o i may simply be w ong. Fo example, he language pa ame e in Figu e 1 is de ined as a s ing, bu only p ope ly o med language codes will be accep ed. Manual wo k is ypically needed o add his missing in o ma ion. Ano he d awback o black-box es ing is ha , due o he lack o con ol o e he SUT, es cases a e mo e cos ly: s a e ul in e ac ions (e.g., upda ing and dele ing a esou ce) can only be achie ed i he SUT is in he p ope s a e (e.g., a esou ce exis s). This means ha e e y es case mus no only exe cise he SUT in a ce ain way, bu also se i up wi h p e ious HTTP calls. As i s key s eng h, black-box es ing does no equi e access o he sou ce code, and can he e o e be po en ially applied o any API ega dless o how i is implemen ed (e.g., he p og amming language) o whe e i is deployed, locally o emo ely. Whi e-box es ing app oaches o REST ul APIs exploi he sou ce code o he SUT o gene a e es cases. Ad anced heu is ics such as ain checking and es abili y ans o ma- ions [23] can be applied o maximize objec i es such as code co e age and aul inding. In con as o black-box es ing, whi e-box es ing can only be applied when he code o he API is a ailable, and i is he e o e implemen a ion-dependen , e.g., he heu is ics o measu e he b anch dis ance mus be adap ed o e e y p og amming language [23]. Howe e , his limi a ion o whi e-box es ing is also i s bes asse , since he sou ce code o he sys em may con ain aluable in o ma ion o he gene a ion o es cases. Mo eo e , since he e is ull con ol o he SUT, es cases a e less cos ly. Fo example, he SUT can be se o a speci ic s a e wi hou he need o using ex a HTTP eques s (as equi ed in black-box es ing) [24]. C. RESTes RESTes [14] is a black-box es ing amewo k o REST ul web APIs. I ollows a model-based es ing app oach [25]: based on he OAS speci ica ion o he API unde es (so- called sys em model), i au oma ically gene a es a es model, which can be manually augmen ed o ailo he es ing p ocess. Bo h models a e subsequen ly used o d i e he au oma ed gene a ion o es cases. RESTes elies on es da a gene a o s o gene a e inpu da a. These a e au oma ically con igu ed o e e y API pa am- e e (e.g., a gene a o o English wo ds o s ing pa am- e e s), ne e heless, i is possible o manually con igu e mo e ealis ic gene a o s acco ding o he pa ame e ’s domain (e.g., a gene a o o alid language codes o he language pa- ame e in Figu e 1). Rega ding es case gene a ion, RESTes le e ages cons ain p og amming sol e s o au oma ically gene a e eques s sa is ying he in e -pa ame e dependencies o he API. Dependencies mus be de ined as a pa o he OAS speci ica ion using he IDL4OAS ex ension [21]. We chose RESTes , and in pa icula i s cons ain -based es case gene a o , as a good ep esen a i e o black-box es ing ools due o i s suppo o da a gene a o s and in e -pa ame e dependencies. Bo h ea u es ha e shown o be e ec i e in inding eal-wo ld bugs in indus ial APIs such as YouTube and Yelp [8]. D. E oMas e E oMas e [7] is a whi e-box es ing ool o REST ul web APIs. I in eg a es a sea ch-based echnique o he au oma ed gene a ion o sys em-le el es cases. The de aul sea ch algo i hm used in E oMas e is he Many Independen Objec i e (MIO) algo i hm [26]. In MIO, es cases a e e ol ed and e alua ed independen ly, and only a he end o he sea ch, a es sui e is cons uc ed by choosing he combina ion o es cases ha co e mo e a ge s (e.g., sou ce code s a emen s and HTTP s a us codes). Fo he ep esen a ion o he p oblem, E oMas e conside s a solu ion as a sys em-le el es sui e o he REST ul API, which is composed o one o mo e es cases, i.e., indi iduals. Each es case is a sequence o one o mo e HTTP calls. Th oughou he sea ch, E oMas e mu a es he es cases ei he by modi ying hei s uc u e (i.e., adding o emo ing HTTP calls) o he da a o one speci ic HTTP call. E oMas e comp ises wo main componen s: he co e, which handles he gene a ion and e olu ion o es cases; and a con olle lib a y, used o he manual se up o he SUT, e.g., i is esponsible o ins umen ing he SUT so ha me ics such as he b anch dis ance can be collec ed. E oMas e also p o ides some basic suppo o black-box es ing, essen ially andom gene a ion wi h no suppo o da a gene a o s o in e -pa ame e dependencies. Howe e , he black-box con igu a ion in E oMas e pe o ms signi ican ly wo se han he whi e-box s a egy implemen ed in he own ool [5]. To he bes o ou knowledge, E oMas e is he only ool suppo ing whi e-box sys em-le el es ing o REST ul web APIs. III. RELATED WORK REST ul API es ing is an ac i e ield o esea ch nowadays. Tes ing app oaches can be di ided in o black-box and whi e- box, he o me being mo e common han he la e . Black-box app oaches mainly di e in h ee aspec s: (1) how hey gen- e a e s a e ul in e ac ions in he SUT; (2) how hey gene a e he inpu da a o eed in o he API; and (3) wha es o acles hey use. Rega ding he gene a ion o s a e ul in e ac ions, some app oaches do no explici ly suppo hem since hey es he API ope a ions in isola ion [12]. O he ools, like RES es , euse da a om p e ious API ope a ions’ esponses (e.g., a esou ce iden i ie ) o subsequen eques s o di e en ope a ions, hus e en ually achie ing s a e ul in e ac ions [14]. Las ly, a numbe o app oaches gene a e s uc u ed se- quences o eques s, ei he by using p ede ined empla es (e.g., POST-PUT-GET) [11], [27], o by dynamically deciding he nex eques o execu e based on he esul o he las one (e.g., i a POST was success ul, execu e a GET, o he wise execu e aPOST again) [9], [10]. 3 Rega ding he gene a ion o inpu da a, se e al s a egies can be ollowed. Ed-douibi e al. [12] p oposed ex ac ing de aul and example alues om he OAS speci ica ion o he API unde es . A lidakis e al. [11] in oduced RESTle , a uzzing ool o REST ul APIs. RESTle uses uzzing dic iona ies o each da a ype (e.g., 0and 1 o in ege pa ame e s). O he ools like QuickREST [9] o RESTes [14] suppo he use o cus omizable es da a gene a o s (e.g., a gene a o o eal coo dina es o a mapping API). Las ly, Viglianisi e al. [10] p oposed ex ac ing alues om API esponses and using hem as inpu s in subsequen API eques s. This las app oach is only applicable o APIs con aining p oduce -consume ela ionships among hei ope a ions (i.e., an ope a ion e u ns some da a ha ano he ope a ion needs as inpu ), which is no he case o some o he APIs used as case s udies in his pape , o ins ance. Las ly, ega ding he es o acles used, mos es ing ap- p oaches ely on he p esence o 5XX s a us codes (i.e., se e e o s) and he con o mance wi h he OAS schema [9]–[12]. O he au ho s ha e p oposed mo e ho ough o acles such as checking he s a us code [8], me amo phic ela ions [6] and secu i y p ope ies [28]. Whi e-box es ing app oaches o REST ul APIs a e less common in he li e a u e. A cu i [7] ad oca es o a sea ch- based app oach, implemen ed in he open-sou ce ool E o- Mas e (discussed in he p e ious sec ion). E oMas e in e- g a es di e se ad anced heu is ics such as es abili y ans- o ma ions [23] and he handling o SQL da abases [24]. Such heu is ics d i e he gene a ion o inpu da a ha is con inuously e ol ed, aimed a gene a ing es cases ha co e as much code and ind as many aul s (5XX s a us codes) as possible. Besides sys em-le el es ing, o he ools such as E oSui e [29] and Randoop [30] allow o au oma ically gen- e a e uni es s o Ja a p og ams, bu hey a e no speci ically ailo ed o REST ul web APIs. Rega ding he compa ison o black-box and whi e-box es ing echniques (one o he main con ibu ions o ou wo k), only one pape has p e iously add essed his ma e : A cu i [5] compa ed black-box and whi e-box es ing in eigh case s udies wi h E oMas e , showing ha he la e always ou pe o med he o me bo h in e ms o code co e age and aul inding. Howe e , he black-box app oach unde compa ison was a he nai e, i.e., basic andom gene a ion. Compa ed o p e ious wo k, we e alua e wo di e en s a e-o - he-a ools o black-box and whi e-box es case gene a ion o REST ul APIs, RESTes and E oMas e , which p o ides a ai e compa ison. On he one hand, RESTes is speci ically designed o exploi he in o ma ion in he API speci ica ion using cons ain -p og amming echniques. On he o he hand, E oMas e le e ages he sou ce code o he gene a ion o e - ec i e es cases using sea ch-based algo i hms. Addi ionally, we p opose a hyb id app oach ha combines bo h s a egies, and show i s po en ial o ou pe o m black-box and whi e-box es ing in isola ion. The hyb id app oach p esen ed in his pape is based on exis ing open-sou ce es ing ools o REST ul APIs, namely, RESTes [14] and E oMas e [7]. We ex ended bo h ools o enable hei in e -ope abili y, and we in oduce sys em-le el es case seeding as a way o enhance sea ch-based es ing o REST ul web APIs. P e ious au ho s ha e p oposed mul iple seeding s a egies o enhancing uni es gene a ion [31], such as eusing p e ious solu ions o seeding alues obse ed a un ime. These s a egies could be complemen a y o ou app oach, and hey could be used o u he imp o e sys em- le el es case gene a ion. IV. BLACK-BOX + WHITE-BOX TEST CASE GENERATION We p esen a no el sys em-le el es case seeding app oach, whe e he es cases gene a ed by he black-box app oach a e seeded in o he sea ch algo i hm used by he whi e-box app oach. The e o e, ins ead o s a ing “ om sc a ch”, he sea ch s a s wi h a (po en ially) ho ough es sui e, which is subsequen ly e ol ed aiming o maximize code co e age and aul s ound. A. Mo i a ion Le us conside he ope a ion o p oo eading a ex in he LanguageTool API, depic ed in Figu e 1. This ope a ion accep s 10 inpu pa ame e s and imposes nume ous cons ain s on hem, including bo h in e -pa ame e dependencies (e.g., i pa ame e p e e edVa ian s is used, hen language mus be se o ‘au o’) and cons ain s on single pa ame e s (e.g., pa ame e mo he Tongue mus be a p ope ly o med language code). In o de o an API call o be alid (i.e., o ob ain a 2XX HTTP s a us code), i mus sa is y all hese cons ain s. These cons ain s a e e lec ed in he sou ce code o he sys em as mul iple b anch condi ions. I any o hese condi ions is no sa is ied, p ocessing e mina es and a 400 s a us code is immedia ely e u ned. Au oma ically gene a ing a alid API call is no i ial. A alid call mus include he speci ic combina ion o API pa ame e s (e.g., mo he Tongue) se o he speci ic alues (e.g., ‘en-US’) such ha all he inpu cons ain s p esen in he API a e sa is ied. This in ol es gene a ing es inpu s ha sa is y he nume ous b anch condi ions ound in he sou ce code be o e exe cising he ac ual unc ionali y o he API, i.e., p oo eading he ex . E en when using a sea ch algo i hm o le e age he sou ce code, a la ge numbe o i e a ions may be equi ed. Lis ing 1 depic s a es case1gene a ed by E oMas e o he ope a ion in Figu e 1 a he beginning o he sea ch. This es case was ejec ed by he API, because i did no use a alid alue o pa ame e language. Lis ing 2, on he o he hand, shows a es case gene a ed by RESTes . This es case did ob ain a success ul esponse, since i passed all he inpu alida ion logic implemen ed in he sys em. RESTes can gene a e his ype o eques s au oma ically because: (1) i uses ealis ic da a gene a o s o each pa ame e (e.g., a language code gene a o o pa ame e language); and (2) i analyzes he dependencies exp essed wi h IDL4OAS (lines 1Due o space cons ain s, we do no show he asse ions on he e u ned esponses in his and he subsequen lis ings. 4 1@Tes 2public oid es _17() { 3Res Assu ed.gi en() 4.con en Type("applica ion/x-www- o m-u lencoded") 5. o mPa am("enabledOnly"," alse") 6. o mPa am(" ex ","gh53wgh2") 7. o mPa am("enabledRules","bny 34yha ") 8. o mPa am("language","b9p1s ") 9.when() 10 .pos ("/check"); 11 } Lis ing 1. Random es case gene a ed by E oMas e a he s a o he sea ch. Fig. 2. Tes - o- es ansla ion app oach. 60-64 om Figu e 1) and uses cons ain sol e s o gene a e alid combina ions o pa ame e s and alues [8]. B. Implemen a ion We implemen ed ou app oach as an ex ension o he open- sou ce ools RESTes [14] and E oMas e [7]. RESTes gene - a es he ini ial popula ion o es cases (seed), and E oMas e subsequen ly e ol es hem. This en ails a echnical challenge: how o ans o m RESTes ’s es cases in o a o ma ha E o- Mas e can handle? Figu e 2 depic s he p oposed app oach o ensu ing in e -ope abili y be ween di e en es ing app oaches and es case o ma s. Ra he han a di ec ansla ion o he sou ce es cases in o he a ge o ma , we ad oca e o ansla ing he seed in o an in e media y o ma which can be subsequen ly pa sed. This has se e al bene i s: (1) al eady exis ing es sui es (e.g., hose manually c ea ed by he SUT de elope s) can be used as a seed by implemen ing speci ic es ansla o s; (2) seeds can be gene a ed by exis ing sou ce es case gene a o s (e.g., RESTes [14]) by implemen ing es w i e s o he in e media y o ma ; (3) only one es pa se is needed o eeding he seed in o he sea ch algo i hm, he eby allowing he a ge es case gene a o (e.g., E oMas e [7]) o le e age he seed. The in e media y o ma selec ed o ou app oach is Pos man. Pos man [32] is an indus y-s anda d pla o m o web API de elopmen . I p o ides suppo o se e al asks h oughou he API li ecycle, such as he c ea ion o API clien s and documen a ion, o e en basic au oma ed es ing. Pos man allows o c ea e es sui es o REST ul APIs and expo hem o so-called Pos man collec ions, in JSON o ma . This makes ou app oach eadily applicable in p ac ice in hose cases whe e a p e ious Pos man es sui e al eady exis s. When he es sui e is in a di e en o ma (e.g., JUni [33] using he REST Assu ed lib a y [34]), only a es ansla o 1@Tes 2public oid es _ 4idj2asd3s6_check() { 3Res Assu ed.gi en() 4.con en Type("applica ion/x-www- o m-u lencoded") 5. o mPa am("p e e edVa ian s","en-GB,ja-JP,sk-SK") 6. o mPa am("mo he Tongue","sk-SK") 7. o mPa am("language","en") 8. o mPa am(" ex ","s ick a ound nose") 9.when() 10 .pos ("/check"); 11 } Lis ing 2. Realis ic es case gene a ed by RESTes . TABLE I CASE STUDIES USED IN THE EVALUATION. SUT Classes LOCs Endpoin s RESTcoun ies 20 1,450 22 YouTubeMock 30 3,371 1 Ohsome 75 9,813 122 LanguageTool 1,021 162,341 1 To al 1,146 176,975 147 needs o be c ea ed (g ayed-ou a ea o Figu e 2). Fo he implemen a ion o ou app oach, we ex ended bo h RESTes and E oMas e wi h a Pos man es w i e and a Pos man es pa se , espec i ely. V. EMPIRICAL STUDY The aim o his p ac ical expe ience epo is o p o ide empi ical e idence on he applicabili y and he limi a ions o black-box, whi e-box and hyb id es ing echniques o REST ul APIs. To his end, we pose he ollowing esea ch ques ions: •RQ1: How do black-box cons ain -based es ing and whi e-box sea ch-based es ing compa e in e ms o code co e age and aul inding? •RQ2: How does ou no el hyb id app oach compa e wi h he black-box and whi e-box echniques in e ms o code co e age and aul inding? A. Case S udies Table I summa izes he main ea u es o he case s udies selec ed, including hei name and size in e ms o Ja a classes, lines o code (LOCs) and HTTP endpoin s. RESTcoun ies2 is an API o que ying in o ma ion abou coun ies based on se e al il e s such as he code, he cu ency o he con i- nen . YouTubeMock3is an open-sou ce implemen a ion o he sea ch ope a ion o he YouTube API [35]. Ohsome4allows o consume OpenS ee Map da a [36] o he whole wo ld and o ge agg ega ed s a is ics abou i . Las ly, LanguageTool5is a p oo eading API wi h suppo o mo e han 20 languages. We selec ed REST ul APIs belonging o a ied applica ion domains and wi h e y di e en sizes. These APIs a e non- i ial, since hey deal wi h inpu cons ain s wi h a ied 2h ps://gi hub.com/apilaye / es coun ies 3h ps://gi hub.com/opensou cingapis/YouTubeMock 4h ps://gi hub.com/GIScience/ohsome-api 5h ps://gi hub.com/language ool-o g/language ool 5 TABLE II BUDGETS AND SEEDS USED FOR EACH SUT. SUT Budge s Seeds Low High Min Med Max RESTcoun ies 5K 100K 22 220 2.2K YouTubeMock 10K 200K 10 100 1K Ohsome 30K 600K 122 1.22K 12.2K LanguageTool 100K 1M 100 1K 10K deg ees o complexi y, such as he in e -pa ame e dependen- cies p esen in Figu e 1 (lines 60-64). In o de o make a ai compa ison be ween black-box and whi e-box es ing, we selec ed only s a eless APIs, since he API s a e canno be ese be ween es cases in black-box es ing (see Sec ion II-B o de ails). Fo each API unde es , we s udied i s docu- men a ion and ex ended i s OAS schema o explici ly de ine in e -pa ame e dependencies using IDL4OAS [21]. B. Expe imen al Se up Fo each case s udy, we e alua ed h ee echniques, namely: •BB: The black-box app oach, ha is, he cons ain -based es ing echnique implemen ed in RESTes . •WB: The whi e-box app oach, namely, he sea ch-based echnique implemen ed in E oMas e . •HYBRID: Ou no el hyb id app oach, implemen ed as an ex ension o bo h ools, i.e., RESTes ’s es cases a e seeded in o E oMas e . Table II shows he budge s wi h which we e alua ed each es ing echnique. Fo HYBRID, he size o he seeds used is also shown. Bo h me ics a e measu ed in e ms o HTTP calls, and hey we e adjus ed acco ding o he SUT size (in e ms o LOCs and numbe o endpoin s) since i s ongly de e mines he ime equi ed o whi e-box heu is ics o e ol e owa d good solu ions. Fo ins ance, YouTubeMock was assigned wice he budge o RESTcoun ies because i has wice as many LOCs, and Ohsome was assigned a minimum seed o 122 HTTP calls because i con ains 122 endpoin s. Fo BB, we conside ed only he minimum o he wo budge s used o WB and HYBRID, o make he expe imen s a o dable.6This is in line wi h how black-box es ing is pe o med in p ac ice, whe e esou ces a e o en cons ained due o es ic i e quo a limi a ions imposed by comme cial APIs [8], [37]. Fo each SUT, we pe o med a o al o 9 expe imen s: one o BB ( he minimum budge ), wo o WB (bo h budge s), and six o HYBRID ( wo budge s × h ee seed sizes). In o de o accoun o he andomness o he algo i hms, we epea ed each expe imen 10 imes, ollowing he guidelines in [38]. As an excep ion, some con igu a ions we e un only once, ei he because hey ook oo much ime o comple e (e.g., 24 hou s) o due o echnical limi a ions o he ools used (discussed in Sec ions VI and VII).7Gi en ou SUTs, his led o 4 ×9× 6RESTes does no sh ink es sui es, and so i is ime-consuming o measu e he co e age o millions o es cases, which mus be done manually and dozens o imes, as explained u he on. 7Con igu a ions un only once: o Ohsome, BB; o LanguageTool, WB wi h he high budge , HYBRID wi h he high budge and all seed sizes, and HYBRID wi h he low budge and la ges seed. TABLE III RQ1: COMPARISON OF BB AND WB. SUT Co e age (%) 5XX BB WB ˆ A12 p- alue BB WB ˆ A12 p- alue RESTcoun ies 74.9 73.6 0.68 0.136 0.0 0.5 0.25 0.014 YouTubeMock 81.9 26.0 1.00 <0.001 0.0 0.0 0.50 NaN Ohsome 74.0 20.4 1.00 0.111 122.0 0.0 1.00 0.004 LanguageTool 46.6 35.3 1.00 <0.001 1.0 1.0 0.50 NaN 10 – (9 ×6) = 306 expe imen uns, o a o al o mo e han 40 million HTTP calls. In o de o make a ai compa ison be ween he h ee ech- niques, we manually execu ed he (hund eds o ) gene a ed es sui es and measu ed he code co e age wi h In elliJ IDEA,8 a Ja a IDE. Fo he aul s ound, we elied on he numbe o dis inc API endpoin s e u ning a leas one 5XX s a us code, as done in [5], [7]. Tooling Se up: We pe o med he expe imen s wi h wo open-sou ce ools, RESTes and E oMas e . Bo h ools equi e some manual se up. In he case o RESTes , we modi ied he es models o each SUT o include ealis ic es da a gene a- o s o domain-speci ic pa ame e s (e.g., YouTube ideo IDs o he YouTubeMock API). In he case o E oMas e , we w o e he equi ed Ja a classes used by he co e componen o s a , s op and ese he SUT. In bo h cases, we used he de aul con igu a ion o he ools. C. Resul s To e alua e he e ec i eness o each es ing echnique, we compu ed he pe cen age o code co e age achie ed (in e ms o s a emen co e age o he whole applica ion) and aul s ound (in e ms o he numbe o endpoin s e u ning 5XX s a us codes), ou o 10 uns. In o de o answe he esea ch ques ions, we pe o med pai compa isons be ween he echniques e alua ed, using he Va gha-Delaney e ec size ˆ A12 and he Mann-Whi ney-Wilcoxon U- es (a α= 0.05 signi icance le el) [38]. 1) RQ1: Black-Box s Whi e-Box: Table III shows he esul s achie ed by black-box and whi e-box es ing o he lowe budge o HTTP calls (i.e., he only budge used o BB). “NaN” alues ob ained o he p- alues mean ha bo h echniques pe o med exac ly in he same way in all cases (e.g., bo h BB and WB ound one aul y endpoin in LanguageTool in he 10 expe imen uns). In e ms o code co e age, BB ou pe o med WB wi h s ong s a is ical e idence in 2 ou o 4 SUTs. Howe e , no e ha he high p- alue o Ohsome is due o unning he BB echnique only once. The la ges di e ences we e obse ed in YouTube- Mock and Ohsome (>50% code co e age), because hese SUTs p esen e y complex inpu cons ain s on and among hei pa ame e s. These cons ain s need o be sa is ied in o de o pass he inpu alida ion logic implemen ed in he sys em and exe cise hei ac ual unc ionali y (e.g., compu ing he a ea o a ci y in Ohsome). In ac , none o he eques s gene a ed by E oMas e o hese SUTs ob ained a success ul esponse. 8h ps://www.je b ains.com/idea/ 6 TABLE IV CONFIGURATIONS OF HYBRID. SUT Budge Co e age (%) 5XX Min Med Max Min Med Max RESTcoun ies 5K 79.5 79.7 79.4 0.9 0.9 0.8 100K 80.2 80.2 80.1 1.0 1.0 1.0 YouTubeMock 10K 68.9 81.4 86.3 0 0 0 200K 74.2 84.5 87.7 0 0.2 0.5 Ohsome 30K 68.7 74.6 75.1 45.3 116.1 122.0 600K 70.3 75.8 75.8 46.0 116.0 122.0 LanguageTool 100K 41.3 42.7 45.0 1.0 1.0 1.0 1M 43.0 43.0 45.0 1.0 1.0 1.0 In he case o RESTcoun ies and LanguageTool, hese SUTs do no expose as complex cons ain s, and he e o e a sea ch- based app oach can e en ually in e hose om he sou ce code and gene a e high-co e age es cases. In e ms o aul inding, BB and WB pe o med mo e simi- la ly. Bo h o hem ound one aul 9in LanguageTool and none in YouTubeMock, and each echnique ound unique bugs in one SUT (WB unco e ed a aul y endpoin in RESTcoun ies and BB ound aul s in all 122 endpoin s o Ohsome). An endpoin may e u n a 500 s a us code due o mul iple aul s in he SUT, o ice e sa, mul iple endpoin s migh ail due o he same bug. Wi hou a manual in-dep h analysis, ou o he scope o his pape , i is no possible o ell he exac numbe o aul s ound by each o he (hund eds o ) es sui es gene a ed. Howe e , an explo a o y analysis in he Ohsome API seems o indica e ha mos o i s ailu es a e due o he same aul s. All ope a ions ha e nea ly he same pa ame e s, he e o e all endpoin s implemen mos ly he same alida ion logic, whe e he aul s a e loca ed. RQ1: Wi h a low budge o HTTP calls, black-box cons ain -based es ing achie ed be ween 1.3% and 55.9% highe code co e age han whi e-box es ing. In e ms o he aul s ound, bo h echniques pe o med simila ly. 2) RQ2: Hyb id s Black-Box & Whi e-Box App oaches: As p e iously explained, we e alua ed HYBRID wi h h ee di e en seed sizes. Table IV shows he esul s ob ained o each SUT, budge and he h ee seed sizes (min, med, and 9Recall ha , by “ aul ” o “bug”, we ac ually e e o “an endpoin e u ning a leas one 5XX s a us code”, as done by p e ious au ho s [5], [7]. max) desc ibed in Table II. The highes alues on each ow a e highligh ed in bold ace. The bes con igu a ion was ob ained wi h he la ges seed, excep o RESTcoun ies, whe e he medium seed yielded he bes esul s. This may be explained by he ac ha he sea ch algo i hm may be o e i ed when seeding many es cases, and ins ead i is necessa y o explo e he sea ch space wi h mo e a ied es cases. We compa ed he bes pe o ming con igu a ion o HYBRID agains BB and WB, in pai s. Table V illus a es such com- pa ison. Compa ed bo h o BB and WB, he imp o emen s achie ed in he code co e age by ou no el hyb id app oach a e s a is ically signi ican wi h s ong e ec sizes, in all possible con igu a ions (all SUTs and budge s) excep o LanguageTool, whe e BB pe o med bes . This di e ence may be caused by he peculia i ies o his SUT. LanguageTool is composed o hund eds o Ja a classes consis ing o “language ules”, whose code is only execu ed when he ule is igge ed. A ule is igge ed when a ex p o ided as inpu con ains a mis ake ma ching he ule pa e n. Since BB uses a andom ex gene a o , i may be mo e e ec i e han using a sea ch- based app oach o e ol e s ings (which is no i ial [23], [39]) in ways ha exe cise new code. O e all, and in aw e ms, HYBRID co e ed 2.2% mo e code han BB and 28.8% mo e code han WB. Figu e 3 shows he co e age achie ed by BB (3a), WB (3b, 3d) and HYBRID (3c, 3e) a 5% in e als o he o e all budge used. Fo WB and HYBRID, wo cha s a e shown, one o he low budge (3b, 3c) and ano he o he high budge (3d, 3e). This igu e e eals a signi ican di e ence in how he h ee echniques pe o m as he budge is p og essi ely consumed. While BB needs a small po ion o he budge o each a co e age close o he maximum achie ed, he co e age o WB and HYBRID e ol es mo e s eadily, because he sea ch algo i hm keeps gene a ing es cases ha co e new a ge s (e.g., sou ce code s a emen s and b anches). Rega ding he aul s ound, we can say wi h high con idence (i.e., p- alue <0.05) ha HYBRID pe o med be e han BB in one scena io—RESTcoun ies, low budge —and be e han WB in h ee scena ios, namely, YouTubeMock wi h he high budge and Ohsome wi h bo h budge s. This esul in he Ohsome SUT was expec ed, since he seeded es sui e al eady con ained aul - e ealing es cases, which WB was no able o gene a e. I is no ewo hy, howe e , ha HYBRID unco e ed a bug in YouTubeMock no unco e ed by ei he TABLE V RQ2: COMPARISON OF HYBRID WITH BB AND WB. SUT Budge Co e age (%) 5XX HYBRID (h) BB (b) WB (w) ˆ Ahb p- alue ˆ Ahw p- alue HYBRID (h) BB (b) WB (w) ˆ Ahb p- alue ˆ Ahw p- alue RESTcoun ies 5K 79.7 74.9 73.6 1.00 <0.001 0.98 <0.001 0.9 0.0 0.5 0.95 <0.001 0.70 0.064 100K 80.2 - 79.0 - - 0.88 0.002 1.0 - 1.0 - - 0.50 NaN YouTubeMock 10K 86.3 81.9 26.0 1.00 <0.001 1.00 <0.001 0.0 0.0 0.0 0.50 NaN 0.50 NaN 200K 87.7 - 47.3 - - 1.00 <0.001 0.5 - 0.0 - - 0.75 0.014 Ohsome 30K 75.1 74.0 20.4 1.00 0.035 1.00 <0.001 122.0 122.0 0.0 0.50 NaN 1.00 <0.001 600K 75.8 - 21.2 - - 1.00 <0.001 122.0 - 0.2 - - 1.00 <0.001 LanguageTool 100K 45.0 46.6 35.3 0.00 0.111 1.00 0.195 1.0 1.0 1.0 0.50 NaN 0.50 NaN 1M 45.0 - 36.0 - - 1.00 1.000 1.0 - 1.0 - - 0.50 NaN 7 (a) BB, low budge . (b) WB, low budge . (c) HYBRID, low budge . (d) WB, high budge . (e) HYBRID, high budge . Fig. 3. T end o co e age e olu ion o he BB,WB and HYBRID app oaches. Fo BB, only he median es sui e (i.e., he i h bes pe o ming in e ms o code co e age) is conside ed, since his had o be done manually. Fo WB and HYBRID, he a e age co e age o all es sui es is compu ed, since E oMas e can ou pu hese s a s au oma ically. BB no WB. This highligh s he po en ial o ou p oposal: seeding an e olu iona y algo i hm wi h a ho ough es sui e can p o ide a boo s ap o he sea ch, and e en ually make i ind new aul s. E en in he cases whe e no new bugs we e unco e ed, mo e code was co e ed in less ime. Fo example, in RESTcoun ies, 5K HTTP calls su iced o co e 79.7% code wi h HYBRID, while WB needed 100K HTTP calls o each 79.0% co e age. RQ2: Ou no el hyb id app oach co e ed be ween 1.2% and 60.3% mo e code han whi e-box es ing in all SUTs, and be ween 1.1% and 4.8% mo e han black-box es ing in 3 ou o 4 SUTs. In e ms o aul inding, he hyb id app oach was he only one unco e ing aul s in all SUTs. VI. LESSONS LEARNED In his sec ion, we p o ide lessons lea ned, de i ed om he empi ical s udy. We may ema k ha hese lessons a e es ic ed o he scope o ou p ac ical expe ience and u u e esea ch will be needed o gene alizing hem u he . Lesson 1: Black-box es ing is signi ican ly as e han whi e- box es ing. Figu e 3 shows ha , o BB,∼5% o he budge su ices o each ∼95% o he code co e age ob ained wi h he whole budge . In con as , WB and HYBRID need mo e budge , bu be e esul s could be ob ained. Mo eo e , since WB and HYBRID apply se e al heu is ics o collec ing me ics such as he s a emen co e age o he b anch dis ance [40], his also ansla es in o a ime o e head. This was ex eme in he case o LanguageTool, whe e a single HYBRID-1M-budge expe imen un ook o e 10 hou s o comple e in a s anda d PC.10 BB ook less han 3 hou s o gene a e a simila es sui e. The e migh be some scena ios whe e ime can be limi ed. Fo example, om a de elope ’s iewpoin , when de eloping a new ea u e o an en e p ise applica ion, his mus be es ed ho oughly be o e going in o p oduc ion. Taking in o accoun he p e ious s a emen , ha is, ∼5% o he budge is enough o each ∼95% o he code co e age in black-box es ing, i is wo h conside ing he ollowing ques ion: wha would de elope s p e e ? Ob aining a easonably ho ough es sui e in 10 minu es, o a mo e exhaus i e es sui e in 10 hou s? Which o he wo es sui es would be enough o e i y ha a new ea u e did no b eak any hing? Bo h app oaches could be complemen a y, e.g., using BB as a quick check and in eg a ing HYBRID as a pa o a con inuous in eg a ion (CI) se up. Lesson 2: Hyb id es ing is mo e e ec i e han black-box and whi e-box es ing in isola ion. This is one o he main conclusions de i ed om he e alua ion. In e ms o code co e age, HYBRID achie ed he bes esul s in 3 ou o 4 SUTs, inc easing co e age by up o 4.8% and 60.3% compa ed o BB and WB, espec i ely. In e ms o aul inding, HYBRID was he only echnique unco e ing bugs in all SUTs. No e ha HYBRID could also be applied wi h al eady exis ing es sui es, e.g., one c ea ed by he SUT de elope s. E en small es sui es can mean a signi ican boo s ap o he sea ch. Fo ins ance, es sui es comp ising jus one es case pe API endpoin in Ohsome (122 in o al) and 10 es cases in YouTubeMock achie ed +50% code co e age han when no seed was p o ided (Table V). C a ing such kind o es sui es is a i ial ask, especially hanks o cu en ool suppo , which 10Technical speci ica ions: In el i5 CPU, 16GB RAM, 256GB SSD, Win- dows 10, Ja a 8. 8 allows o do i manually (e.g., wi h he Swagge ool sui e [41] o Pos man [32]) o au oma ically (e.g., wi h RESTes [14]). This makes ou no el hyb id app oach eadily applicable in p ac ice when a es sui e al eady exis s, and easily applicable when i does no . Lesson 3: Black-box cons ain -based es ing is gene ally p e e ed o la ge and complex sys ems. Based on ou empi ical s udy, WB is mo e e ec i e han BB o simple o smalle SUTs when gi en high budge s (e.g., RESTcoun ies). Howe e , when he SUT con ains in ica e inpu cons ain s, he black-box app oach gene ally ou pe o ms he whi e-box, because he gi en cus om domain knowledge in he da a gene a o s and he handling o in e -pa ame e dependen- cies [22] signi ican ly helps in dealing wi h hose cons ain s. A sea ch algo i hm may s uggle o lea n how o gene a e high-co e age inpu s in sho ime, due o complex b anch condi ions implemen ed in he SUT code. This same limi a ion applies o HYBRID as well (since i also le e ages he sou ce code), and i became clea in LanguageTool. LanguageTool is he bigges SUT, wi h 10 imes mo e LOCs han he o he SUTs oge he . I is a CPU-bound applica ion (e.g., no CRUD ope a ions on a da abase), doing complex compu a ions on s ing ex inpu s. In his case, some expe imen uns o WB and HYBRID e en ailed o inish. The E oMas e Ja a p ocess an ou o memo y due o he p esence o e y long egula exp essions in he code. E oMas e could no handle hose (e.g., wi h mo e han 200K cha ac e s) and c ashed. In o he cases, E oMas e simply imed ou due o e y long execu ion imes (e.g., mo e han 24 hou s). All hese esul s emphasize he ollowing: (1) whi e-box es ing is subjec o echnical limi a ions, depending on he SUT; and (2) black-box es ing is applicable ega dless o how he SUT is implemen ed, i s size o i s complexi y. Lesson 4: A manual se up is equi ed (and ecommended) in all he s udied app oaches. In REST ul API es ing, as in many o he disciplines, he e is a ade-o be ween au oma ion and es ho oughness. App oaches achie ing he highes au oma ion deg ee apply uzz o andom es ing [5], [11] o le e age echniques ha a e simply no applicable in all scena ios [10], [12]. The h ee echniques e alua ed in his wo k equi e some manual se up. Wi hou a p ope empi ical in es iga ion, i is ha d o ell which one equi es less wo k, bu based on ou expe ience, E oMas e is quicke o se up han RESTes . In ac , E oMas e ’s con igu a ion equi es w i ing one Ja a class, while RESTes equi es modi ying a YAML ile whose size depends on he numbe o API endpoin s and pa ame e s. Howe e , as sugges ed by Table V, he manual se up done o RESTes may play a key ole in he inal esul s ob ained, especially o complex APIs dealing wi h e y domain-speci ic pa ame e s. A e de elope s willing o do some manual wo k o imp o e he es ing p ocess? O do hey p e e 100% au oma ion? This is some hing o be empi ically in es iga ed in u u e wo k. Lesson 5: Whi e-box es sui es a e de e minis ic, black- box es sui es a e usually no . One o he main di e ences be ween whi e-box and black-box es ing is ha , in he o me , he s a e o he SUT can be ese be ween es cases (e.g., emp ying a da abase). The e o e, i is possible o c ea e a comple ely de e minis ic es sui e, whe e he ou pu o e e y es case does no depend a all on he es cases p e iously execu ed. This is no possible in black-box es ing o s a e ul APIs, whe e, o ins ance, he esponse o he 100 h API eques depends en i ely on he p e ious 99 eques s execu ed. This complica es debugging, since a aul - e ealing es case may no e eal he same aul again, i he s a e o he SUT has changed. Whi e-box and hyb id es ing can be used o gene a e eg ession es sui es, whe eas black-box es ing is mo e ypically applied o uzzing web se ices o gi en pe iods o ime [11]. VII. OPEN CHALLENGES In he nex pa ag aphs, we discuss open challenges o be ackled by u u e esea ch in REST ul API es ing. These a e aligned wi h wo p ima y goals: inc easing es au oma ion and ho oughness. Challenge 1: Highe deg ee o au oma ion. As p e iously men ioned, he echniques e alua ed in his pape equi e some manual se up. This is some hing ha may p e en p ac i ione s om adop ing hem. Fo example, E oMas e equi es he es e o w i e a d i e class de ining how o s a , s op and ese he SUT, among o he s. Au oma ic code gene a ion echniques could be applied o ease his ask. On he o he hand, black-box es ing echniques in gene al may equi e he addi ion o missing in o ma ion in he API speci ica ion, such as in e -pa ame e dependencies. Ad anced echniques ha can au oma ically ex ac his in o ma ion a e equi ed, e.g., by analyzing eques - esponse pa e ns and in e ing in a ian s p esen in he API, as hin ed in he wo k by Mi abella e al. [42]. Challenge 2: In e ence o ealis ic es inpu s. Ve y ela ed o he p e ious challenge, es ing REST ul APIs ho oughly depends o a g ea ex en on he inpu s (HTTP eques s) used. C ea ing and main aining da a dic iona ies o each pa ame e is e y cos ly [9], [14]. Based on he API speci ica ion, which usually desc ibes he a ailable endpoin s and pa ame e s, he e is need o echniques ha exploi i o in e ealis ic alues o each API pa ame e , e.g., wi h na u al language p ocessing (NLP) capabili ies. The wo k o p e ious au ho s in GUI es ing [43], [44] may inspi e u u e esea ch o add ess his challenge. Challenge 3: Ad anced whi e-box heu is ics. The e ec i e- ness o whi e-box es ing, o e en i s applicabili y, highly depends on he heu is ics used o le e age he sou ce code. I hese a e no co ec ly implemen ed, hey may make whi e- box es ing unusable (e.g., like he c ashes ha we ob ained in E oMas e wi h LanguageTool). I he heu is ics a e oo simple, whi e-box es ing may no be e ec i e. In E oMas e , ain checking and es abili y ans o ma ions, among o he s, 9