Black-Box and Whi e-Box Tes Case Gene a ion o
REST ul APIs: Enemies o Allies?
Albe o Ma in-Lopez
SCORE Lab, I3US Ins i u e
Uni e sidad de Se illa
Se ille, Spain
[email p o ec ed]
And ea A cu i
K is iania Uni e si y College
Oslo Me opoli an Uni e si y
Oslo, No way
[email p o ec ed]
Se gio Segu a
SCORE Lab, I3US Ins i u e
Uni e sidad de Se illa
Se ille, Spain
se [email p o ec ed]
An onio Ruiz-Co ´
es
SCORE Lab, I3US Ins i u e
Uni e sidad de Se illa
Se ille, Spain
[email p o ec ed]
Abs ac —Au oma ed es case gene a ion o REST ul APIs
is a h i ing esea ch opic due o hei c i ical ole in so wa e
in eg a ion. Tes ing app oaches can be di ided in o black-box and
whi e-box. Black-box app oaches exploi he API speci ica ion
o he gene a ion o es cases, while whi e-box app oaches
can also le e age he sou ce code. Bo h s a egies ha e shown
g ea p omise, bu hey ha e no been ully compa ed ye ,
hinde ing he selec ion o he igh ool o he job. In his
pape , we epo on ou expe ience compa ing black-box and
whi e-box es case gene a ion o REST ul APIs using he s a e-
o - he-a ools RESTes (black-box) and E oMas e (whi e-box).
Also, we p opose in eg a ing bo h app oaches by using black-
box es cases as he seed o whi e-box sea ch-based es case
gene a ion. E alua ion esul s on ou REST ul APIs in ol ing
o e 40 million API calls show ha he e is no one-size- i s-all
s a egy. Mo e impo an ly, he combina ion o black-box and
whi e-box yielded he bes esul s in mos case s udies in e ms
o code co e age and aul inding, pa ing he way o be e
ools in eg a ing he bes o bo h pe spec i es. As a esul o ou
wo k, we p o ide lessons lea ned and open challenges o guiding
he use and u he de elopmen o cu en ool suppo .
Index Te ms—REST, API, web se ice, SBST, seeding
I. INTRODUCTION
Web Applica ion P og amming In e aces (APIs) allow he -
e ogeneous sys ems o in e ac o e he ne wo k [1], [2].
Web APIs a e apidly p oli e a ing as a key elemen o os e
so wa e eusabili y, in eg a ion, and inno a ion, enabling new
consump ion models such as wea ables and sma home apps.
Companies such as Facebook, Twi e , Google, eBay o Ne lix
ecei e billions o API calls e e yday om housands o
di e en hi d-pa y applica ions and de ices, which cons i-
u es mo e han hal o hei o al a ic [1]. Mode n web
APIs ypically adhe e o he REp esen a ional S a e T ans e
(REST) a chi ec u al s yle [3], being e e ed o as REST ul
APIs. REST ul APIs p o ide a s anda d mechanism o imple-
men c ea e, e ie e, upda e, and dele e (CRUD) ope a ions
o e esou ces (e.g., a YouTube ideo) in a dis ibu ed way.
The widesp ead use o REST ul APIs is e lec ed in he size
o popula API di ec o ies such as P og ammableWeb [4],
cu en ly indexing o e 24K REST ul APIs om domains such
as shopping, inances, social, o elephony.
Tes ing REST ul APIs is c i ical due o hei key ole in
so wa e in eg a ion. A bug in an o ganiza ion’s API could
ha e a huge impac bo h in e nally (se ices elying on
ha API) and ex e nally ( hi d-pa y applica ions and end
use s). To add ess his challenge, nume ous app oaches o
es case gene a ion o REST ul APIs ha e eme ged in ecen
yea s [5]–[12]. These app oaches can be di ided in o black-
box and whi e-box. Black-box app oaches [5], [6], [8]–[12]
exploi he speci ica ion o he API, ypically in he OpenAPI
Speci ica ion (OAS) [13] o ma , o d i e he gene a ion o es
cases in he o m o (pseudo-) andom API calls. They o en use
cus om da a gene a o s, manually implemen ed by he use s,
based on domain knowledge o he applica ion. Whi e-box
app oaches [7] le e age he API speci ica ion and i s sou ce
code o gene a e es cases ha co e as much code as possible,
ying o minimize he amoun o manual wo k om he use s.
Bo h app oaches y o iden i y se e e o s (i.e., 5XX HTTP
s a us codes) and misma ches wi h he API speci ica ion.
Bo h es case gene a ion s a egies o REST ul APIs,
black-box and whi e-box, ha e been la gely e alua ed in
isola ion, showing hei own me i s and limi a ions. Howe e ,
hey ha e no been ully compa ed ye . Thus, many ques ions
emain unanswe ed ega ding he scope and pe o mance o
each app oach. Besides his, bo h app oaches ha e ollowed
comple ely di e en pa hs so a , and hus he po en ial bene i s
o hei combina ion emain unexplo ed.
In his pape , we epo on ou expe ience compa ing
black-box and whi e-box es case gene a ion echniques
o REST ul APIs. Speci ically, we use he s a e-o - he-a
ools RESTes [14] (black-box) and E oMas e [7] (whi e-
box). Addi ionally, we p opose he in eg a ion o bo h ap-
p oaches by using black-box es cases—de i ed om he
API speci ica ion—as he seed o sea ch-based es case
gene a ion—le e aging he sou ce code o he API. We epo
on he esul s o an ex ensi e empi ical e alua ion compa ing
he e ec i eness and pe o mance o hese h ee app oaches:
black-box, whi e-box and hyb id (black-box + whi e-box)
on ou open-sou ce REST ul APIs o di e en sizes and
complexi y. The esul s show ha ou no el hyb id app oach
pe o med bes in mos scena ios. In e ms o aul inding,
he hyb id app oach was he only one capable o unco e ing
bugs in all sys ems unde es (SUTs). We p o ide an in-
dep h discussion and analysis o he esul s ob ained by each
echnique, and possible causes leading o such esul s. Ou
wo k con ibu es o a be e unde s anding o he s eng hs and
1
limi a ions o black-box and whi e-box es case gene a ion
o REST ul APIs, and shows he po en ial o combining bo h
s a egies. Howe e , mo e wo k and expe imen a ion will be
equi ed in he u u e o complemen ou esul s including he
use o di e en es ing echniques, ools, and case s udies.
A e explaining he backg ound (Sec ion II) and discussing
ela ed wo k (Sec ion III), his pape p o ides he ollowing
o iginal esea ch and enginee ing con ibu ions:
•A no el hyb id es ing app oach based on he use o
black-box es cases as he seed o whi e-box es
gene a ion, and i s implemen a ion in E oMas e [7]
(Sec ion IV).
•A compa ison o black-box, whi e-box, and hyb id ech-
niques o es case gene a ion o REST ul APIs in
e ms o code co e age and aul inding using wo
di e en s a e-o - he-a ools and ou open-sou ce APIs
(Sec ion V).
•Lessons lea ned om he compa ison o he h ee ech-
niques, including p ac ical guidelines o he selec ion
o he igh app oach depending on se e al ac o s such
as he size o he SUT and he a ailable esou ces
(Sec ion VI).
•A lis o challenges o he h ee es case gene a ion
app oaches unde compa ison de i ed om he esul s o
he s udy (Sec ion VII).
We add ess h ea s o alidi y in Sec ion VIII and conclude
he pape in Sec ion IX.
II. BACKGROUND
A. REST ul Web APIs
REST [3] is a so wa e a chi ec u al s yle o building
dis ibu ed sys ems. Mos cu en web se ices ollow he
p inciples de ined by REST and a e e e ed o as REST ul (o
simply REST) web se ices. REST ul web se ices p o ide a
s anda d app oach o in e ac wi h esou ces o e he ne wo k.
A esou ce is any piece o da a ha can be exposed o he
Web, o example, a documen (Google D i e API [15]), a
pic u e (Flick API [16]), o e en a wee (Twi e API [17]).
Resou ces can be accessed and manipula ed by means o
CRUD ope a ions. These ope a ions can be in oked by sending
HTTP eques s o speci ic API endpoin s, each o which is
iden i ied by an HTTP me hod (e.g., GET) and a pa h (e.g.,
/documen s). A REST ul API may be composed o one o
mo e REST ul se ices.
REST ul web APIs a e usually desc ibed wi h in e -
ace desc ip ion languages like he OpenAPI Speci ica ion
(OAS) [13]. OAS is hea ily used nowadays o au oma ing
ce ain asks in he API li ecycle such as code gene a ion [18],
moni o ing [19] and es ing [14]. An OAS documen (also
called schema) desc ibes a REST ul API in e ms o he
allowed inpu s (HTTP eques s) and he expec ed ou pu s
(HTTP esponses). Figu e 1 depic s an ex ac o an OAS
schema, aken om LanguageTool, a p oo eading API [20].
As illus a ed, he POST /check endpoin allows o de ec
mis akes in a ex (line 4). The speci ica ion de ails he
Fig. 1. OAS exce p om he LanguageTool API.
in o ma ion abou he ope a ion pa ame e s (lines 7-59), such
as hei da a ype (line 57), whe he hey a e equi ed o
op ional (line 58) and de aul alues (line 59). Fo e e y
ope a ion, he se o expec ed esponses is also desc ibed (lines
65-69), including hei HTTP s a us code (line 66) and hei
o ma (line 69). Op ionally, when he ope a ion con ains in e -
pa ame e dependencies (e.g., mu ually exclusi e pa ame e s),
hese can be speci ied wi h he IDL4OAS ex ension [21] (lines
60-64). Fo example, he inpu ex agmen can be p o ided
as a plain s ing using pa ame e ex (line 8) o as a JSON
documen using pa ame e da a (line 13), bu only one o
hem mus be se (line 61). Acco ding o a ecen s udy by
Ma in-Lopez e al. [22], 4 ou o e e y 5 indus ial APIs
con ain hese in e -pa ame e dependencies.
B. REST ul API Tes Case Gene a ion
Tes ing a REST ul API a he sys em le el in ol es gene -
a ing HTTP eques s and asse ing hei esponses. A es case
comp ises one o mo e eques s.
2
Black-box es ing app oaches le e age he speci ica ion o
he API unde es (e.g., an OAS documen ) o au oma ically
gene a e es cases. Howe e , he speci ica ion may no su ice
o gene a e ealis ic es inpu s, o i may simply be w ong.
Fo example, he language pa ame e in Figu e 1 is de ined
as a s ing, bu only p ope ly o med language codes will
be accep ed. Manual wo k is ypically needed o add his
missing in o ma ion. Ano he d awback o black-box es ing
is ha , due o he lack o con ol o e he SUT, es cases a e
mo e cos ly: s a e ul in e ac ions (e.g., upda ing and dele ing
a esou ce) can only be achie ed i he SUT is in he p ope
s a e (e.g., a esou ce exis s). This means ha e e y es case
mus no only exe cise he SUT in a ce ain way, bu also se
i up wi h p e ious HTTP calls. As i s key s eng h, black-box
es ing does no equi e access o he sou ce code, and can
he e o e be po en ially applied o any API ega dless o how
i is implemen ed (e.g., he p og amming language) o whe e
i is deployed, locally o emo ely.
Whi e-box es ing app oaches o REST ul APIs exploi he
sou ce code o he SUT o gene a e es cases. Ad anced
heu is ics such as ain checking and es abili y ans o ma-
ions [23] can be applied o maximize objec i es such as code
co e age and aul inding. In con as o black-box es ing,
whi e-box es ing can only be applied when he code o he
API is a ailable, and i is he e o e implemen a ion-dependen ,
e.g., he heu is ics o measu e he b anch dis ance mus be
adap ed o e e y p og amming language [23]. Howe e , his
limi a ion o whi e-box es ing is also i s bes asse , since he
sou ce code o he sys em may con ain aluable in o ma ion
o he gene a ion o es cases. Mo eo e , since he e is ull
con ol o he SUT, es cases a e less cos ly. Fo example, he
SUT can be se o a speci ic s a e wi hou he need o using
ex a HTTP eques s (as equi ed in black-box es ing) [24].
C. RESTes
RESTes [14] is a black-box es ing amewo k o REST ul
web APIs. I ollows a model-based es ing app oach [25]:
based on he OAS speci ica ion o he API unde es (so-
called sys em model), i au oma ically gene a es a es model,
which can be manually augmen ed o ailo he es ing p ocess.
Bo h models a e subsequen ly used o d i e he au oma ed
gene a ion o es cases.
RESTes elies on es da a gene a o s o gene a e inpu
da a. These a e au oma ically con igu ed o e e y API pa am-
e e (e.g., a gene a o o English wo ds o s ing pa am-
e e s), ne e heless, i is possible o manually con igu e mo e
ealis ic gene a o s acco ding o he pa ame e ’s domain (e.g.,
a gene a o o alid language codes o he language pa-
ame e in Figu e 1). Rega ding es case gene a ion, RESTes
le e ages cons ain p og amming sol e s o au oma ically
gene a e eques s sa is ying he in e -pa ame e dependencies
o he API. Dependencies mus be de ined as a pa o he
OAS speci ica ion using he IDL4OAS ex ension [21].
We chose RESTes , and in pa icula i s cons ain -based es
case gene a o , as a good ep esen a i e o black-box es ing
ools due o i s suppo o da a gene a o s and in e -pa ame e
dependencies. Bo h ea u es ha e shown o be e ec i e in
inding eal-wo ld bugs in indus ial APIs such as YouTube
and Yelp [8].
D. E oMas e
E oMas e [7] is a whi e-box es ing ool o REST ul
web APIs. I in eg a es a sea ch-based echnique o he
au oma ed gene a ion o sys em-le el es cases. The de aul
sea ch algo i hm used in E oMas e is he Many Independen
Objec i e (MIO) algo i hm [26]. In MIO, es cases a e e ol ed
and e alua ed independen ly, and only a he end o he sea ch,
a es sui e is cons uc ed by choosing he combina ion o es
cases ha co e mo e a ge s (e.g., sou ce code s a emen s and
HTTP s a us codes). Fo he ep esen a ion o he p oblem,
E oMas e conside s a solu ion as a sys em-le el es sui e
o he REST ul API, which is composed o one o mo e es
cases, i.e., indi iduals. Each es case is a sequence o one o
mo e HTTP calls. Th oughou he sea ch, E oMas e mu a es
he es cases ei he by modi ying hei s uc u e (i.e., adding
o emo ing HTTP calls) o he da a o one speci ic HTTP
call.
E oMas e comp ises wo main componen s: he co e,
which handles he gene a ion and e olu ion o es cases; and a
con olle lib a y, used o he manual se up o he SUT, e.g.,
i is esponsible o ins umen ing he SUT so ha me ics
such as he b anch dis ance can be collec ed.
E oMas e also p o ides some basic suppo o black-box
es ing, essen ially andom gene a ion wi h no suppo o
da a gene a o s o in e -pa ame e dependencies. Howe e , he
black-box con igu a ion in E oMas e pe o ms signi ican ly
wo se han he whi e-box s a egy implemen ed in he own
ool [5]. To he bes o ou knowledge, E oMas e is he only
ool suppo ing whi e-box sys em-le el es ing o REST ul
web APIs.
III. RELATED WORK
REST ul API es ing is an ac i e ield o esea ch nowadays.
Tes ing app oaches can be di ided in o black-box and whi e-
box, he o me being mo e common han he la e . Black-box
app oaches mainly di e in h ee aspec s: (1) how hey gen-
e a e s a e ul in e ac ions in he SUT; (2) how hey gene a e
he inpu da a o eed in o he API; and (3) wha es o acles
hey use.
Rega ding he gene a ion o s a e ul in e ac ions, some
app oaches do no explici ly suppo hem since hey es he
API ope a ions in isola ion [12]. O he ools, like RES es ,
euse da a om p e ious API ope a ions’ esponses (e.g.,
a esou ce iden i ie ) o subsequen eques s o di e en
ope a ions, hus e en ually achie ing s a e ul in e ac ions [14].
Las ly, a numbe o app oaches gene a e s uc u ed se-
quences o eques s, ei he by using p ede ined empla es (e.g.,
POST-PUT-GET) [11], [27], o by dynamically deciding he
nex eques o execu e based on he esul o he las one (e.g.,
i a POST was success ul, execu e a GET, o he wise execu e
aPOST again) [9], [10].
3
Rega ding he gene a ion o inpu da a, se e al s a egies can
be ollowed. Ed-douibi e al. [12] p oposed ex ac ing de aul
and example alues om he OAS speci ica ion o he API
unde es . A lidakis e al. [11] in oduced RESTle , a uzzing
ool o REST ul APIs. RESTle uses uzzing dic iona ies o
each da a ype (e.g., 0and 1 o in ege pa ame e s). O he
ools like QuickREST [9] o RESTes [14] suppo he use
o cus omizable es da a gene a o s (e.g., a gene a o o eal
coo dina es o a mapping API). Las ly, Viglianisi e al. [10]
p oposed ex ac ing alues om API esponses and using
hem as inpu s in subsequen API eques s. This las app oach
is only applicable o APIs con aining p oduce -consume
ela ionships among hei ope a ions (i.e., an ope a ion e u ns
some da a ha ano he ope a ion needs as inpu ), which is no
he case o some o he APIs used as case s udies in his
pape , o ins ance.
Las ly, ega ding he es o acles used, mos es ing ap-
p oaches ely on he p esence o 5XX s a us codes (i.e., se e
e o s) and he con o mance wi h he OAS schema [9]–[12].
O he au ho s ha e p oposed mo e ho ough o acles such as
checking he s a us code [8], me amo phic ela ions [6] and
secu i y p ope ies [28].
Whi e-box es ing app oaches o REST ul APIs a e less
common in he li e a u e. A cu i [7] ad oca es o a sea ch-
based app oach, implemen ed in he open-sou ce ool E o-
Mas e (discussed in he p e ious sec ion). E oMas e in e-
g a es di e se ad anced heu is ics such as es abili y ans-
o ma ions [23] and he handling o SQL da abases [24].
Such heu is ics d i e he gene a ion o inpu da a ha is
con inuously e ol ed, aimed a gene a ing es cases ha co e
as much code and ind as many aul s (5XX s a us codes)
as possible. Besides sys em-le el es ing, o he ools such as
E oSui e [29] and Randoop [30] allow o au oma ically gen-
e a e uni es s o Ja a p og ams, bu hey a e no speci ically
ailo ed o REST ul web APIs.
Rega ding he compa ison o black-box and whi e-box
es ing echniques (one o he main con ibu ions o ou
wo k), only one pape has p e iously add essed his ma e :
A cu i [5] compa ed black-box and whi e-box es ing in eigh
case s udies wi h E oMas e , showing ha he la e always
ou pe o med he o me bo h in e ms o code co e age
and aul inding. Howe e , he black-box app oach unde
compa ison was a he nai e, i.e., basic andom gene a ion.
Compa ed o p e ious wo k, we e alua e wo di e en s a e-o -
he-a ools o black-box and whi e-box es case gene a ion
o REST ul APIs, RESTes and E oMas e , which p o ides a
ai e compa ison. On he one hand, RESTes is speci ically
designed o exploi he in o ma ion in he API speci ica ion
using cons ain -p og amming echniques. On he o he hand,
E oMas e le e ages he sou ce code o he gene a ion o e -
ec i e es cases using sea ch-based algo i hms. Addi ionally,
we p opose a hyb id app oach ha combines bo h s a egies,
and show i s po en ial o ou pe o m black-box and whi e-box
es ing in isola ion.
The hyb id app oach p esen ed in his pape is based on
exis ing open-sou ce es ing ools o REST ul APIs, namely,
RESTes [14] and E oMas e [7]. We ex ended bo h ools o
enable hei in e -ope abili y, and we in oduce sys em-le el
es case seeding as a way o enhance sea ch-based es ing o
REST ul web APIs. P e ious au ho s ha e p oposed mul iple
seeding s a egies o enhancing uni es gene a ion [31],
such as eusing p e ious solu ions o seeding alues obse ed
a un ime. These s a egies could be complemen a y o ou
app oach, and hey could be used o u he imp o e sys em-
le el es case gene a ion.
IV. BLACK-BOX + WHITE-BOX TEST CASE GENERATION
We p esen a no el sys em-le el es case seeding app oach,
whe e he es cases gene a ed by he black-box app oach
a e seeded in o he sea ch algo i hm used by he whi e-box
app oach. The e o e, ins ead o s a ing “ om sc a ch”, he
sea ch s a s wi h a (po en ially) ho ough es sui e, which is
subsequen ly e ol ed aiming o maximize code co e age and
aul s ound.
A. Mo i a ion
Le us conside he ope a ion o p oo eading a ex in
he LanguageTool API, depic ed in Figu e 1. This ope a ion
accep s 10 inpu pa ame e s and imposes nume ous cons ain s
on hem, including bo h in e -pa ame e dependencies (e.g., i
pa ame e p e e edVa ian s is used, hen language
mus be se o ‘au o’) and cons ain s on single pa ame e s
(e.g., pa ame e mo he Tongue mus be a p ope ly o med
language code). In o de o an API call o be alid (i.e., o
ob ain a 2XX HTTP s a us code), i mus sa is y all hese
cons ain s. These cons ain s a e e lec ed in he sou ce code
o he sys em as mul iple b anch condi ions. I any o hese
condi ions is no sa is ied, p ocessing e mina es and a 400
s a us code is immedia ely e u ned.
Au oma ically gene a ing a alid API call is no i ial.
A alid call mus include he speci ic combina ion o API
pa ame e s (e.g., mo he Tongue) se o he speci ic alues
(e.g., ‘en-US’) such ha all he inpu cons ain s p esen in
he API a e sa is ied. This in ol es gene a ing es inpu s ha
sa is y he nume ous b anch condi ions ound in he sou ce
code be o e exe cising he ac ual unc ionali y o he API, i.e.,
p oo eading he ex . E en when using a sea ch algo i hm o
le e age he sou ce code, a la ge numbe o i e a ions may be
equi ed.
Lis ing 1 depic s a es case1gene a ed by E oMas e o
he ope a ion in Figu e 1 a he beginning o he sea ch. This
es case was ejec ed by he API, because i did no use
a alid alue o pa ame e language. Lis ing 2, on he
o he hand, shows a es case gene a ed by RESTes . This es
case did ob ain a success ul esponse, since i passed all he
inpu alida ion logic implemen ed in he sys em. RESTes
can gene a e his ype o eques s au oma ically because: (1)
i uses ealis ic da a gene a o s o each pa ame e (e.g., a
language code gene a o o pa ame e language); and (2)
i analyzes he dependencies exp essed wi h IDL4OAS (lines
1Due o space cons ain s, we do no show he asse ions on he e u ned
esponses in his and he subsequen lis ings.
4
1@Tes
2public oid es _17() {
3Res Assu ed.gi en()
4.con en Type("applica ion/x-www- o m-u lencoded")
5. o mPa am("enabledOnly"," alse")
6. o mPa am(" ex ","gh53wgh2")
7. o mPa am("enabledRules","bny 34yha ")
8. o mPa am("language","b9p1s ")
9.when()
10 .pos ("/check");
11 }
Lis ing 1. Random es case gene a ed by E oMas e a he s a o he sea ch.
Fig. 2. Tes - o- es ansla ion app oach.
60-64 om Figu e 1) and uses cons ain sol e s o gene a e
alid combina ions o pa ame e s and alues [8].
B. Implemen a ion
We implemen ed ou app oach as an ex ension o he open-
sou ce ools RESTes [14] and E oMas e [7]. RESTes gene -
a es he ini ial popula ion o es cases (seed), and E oMas e
subsequen ly e ol es hem. This en ails a echnical challenge:
how o ans o m RESTes ’s es cases in o a o ma ha E o-
Mas e can handle? Figu e 2 depic s he p oposed app oach o
ensu ing in e -ope abili y be ween di e en es ing app oaches
and es case o ma s. Ra he han a di ec ansla ion o
he sou ce es cases in o he a ge o ma , we ad oca e o
ansla ing he seed in o an in e media y o ma which can
be subsequen ly pa sed. This has se e al bene i s: (1) al eady
exis ing es sui es (e.g., hose manually c ea ed by he SUT
de elope s) can be used as a seed by implemen ing speci ic
es ansla o s; (2) seeds can be gene a ed by exis ing sou ce
es case gene a o s (e.g., RESTes [14]) by implemen ing es
w i e s o he in e media y o ma ; (3) only one es pa se is
needed o eeding he seed in o he sea ch algo i hm, he eby
allowing he a ge es case gene a o (e.g., E oMas e [7])
o le e age he seed.
The in e media y o ma selec ed o ou app oach is
Pos man. Pos man [32] is an indus y-s anda d pla o m o
web API de elopmen . I p o ides suppo o se e al asks
h oughou he API li ecycle, such as he c ea ion o API
clien s and documen a ion, o e en basic au oma ed es ing.
Pos man allows o c ea e es sui es o REST ul APIs and
expo hem o so-called Pos man collec ions, in JSON o ma .
This makes ou app oach eadily applicable in p ac ice in
hose cases whe e a p e ious Pos man es sui e al eady exis s.
When he es sui e is in a di e en o ma (e.g., JUni [33]
using he REST Assu ed lib a y [34]), only a es ansla o
1@Tes
2public oid es _ 4idj2asd3s6_check() {
3Res Assu ed.gi en()
4.con en Type("applica ion/x-www- o m-u lencoded")
5. o mPa am("p e e edVa ian s","en-GB,ja-JP,sk-SK")
6. o mPa am("mo he Tongue","sk-SK")
7. o mPa am("language","en")
8. o mPa am(" ex ","s ick a ound nose")
9.when()
10 .pos ("/check");
11 }
Lis ing 2. Realis ic es case gene a ed by RESTes .
TABLE I
CASE STUDIES USED IN THE EVALUATION.
SUT Classes LOCs Endpoin s
RESTcoun ies 20 1,450 22
YouTubeMock 30 3,371 1
Ohsome 75 9,813 122
LanguageTool 1,021 162,341 1
To al 1,146 176,975 147
needs o be c ea ed (g ayed-ou a ea o Figu e 2). Fo he
implemen a ion o ou app oach, we ex ended bo h RESTes
and E oMas e wi h a Pos man es w i e and a Pos man es
pa se , espec i ely.
V. EMPIRICAL STUDY
The aim o his p ac ical expe ience epo is o p o ide
empi ical e idence on he applicabili y and he limi a ions
o black-box, whi e-box and hyb id es ing echniques o
REST ul APIs. To his end, we pose he ollowing esea ch
ques ions:
•RQ1: How do black-box cons ain -based es ing and
whi e-box sea ch-based es ing compa e in e ms o code
co e age and aul inding?
•RQ2: How does ou no el hyb id app oach compa e wi h
he black-box and whi e-box echniques in e ms o code
co e age and aul inding?
A. Case S udies
Table I summa izes he main ea u es o he case s udies
selec ed, including hei name and size in e ms o Ja a classes,
lines o code (LOCs) and HTTP endpoin s. RESTcoun ies2
is an API o que ying in o ma ion abou coun ies based on
se e al il e s such as he code, he cu ency o he con i-
nen . YouTubeMock3is an open-sou ce implemen a ion o he
sea ch ope a ion o he YouTube API [35]. Ohsome4allows
o consume OpenS ee Map da a [36] o he whole wo ld and
o ge agg ega ed s a is ics abou i . Las ly, LanguageTool5is
a p oo eading API wi h suppo o mo e han 20 languages.
We selec ed REST ul APIs belonging o a ied applica ion
domains and wi h e y di e en sizes. These APIs a e non-
i ial, since hey deal wi h inpu cons ain s wi h a ied
2h ps://gi hub.com/apilaye / es coun ies
3h ps://gi hub.com/opensou cingapis/YouTubeMock
4h ps://gi hub.com/GIScience/ohsome-api
5h ps://gi hub.com/language ool-o g/language ool
5

TABLE II
BUDGETS AND SEEDS USED FOR EACH SUT.
SUT Budge s Seeds
Low High Min Med Max
RESTcoun ies 5K 100K 22 220 2.2K
YouTubeMock 10K 200K 10 100 1K
Ohsome 30K 600K 122 1.22K 12.2K
LanguageTool 100K 1M 100 1K 10K
deg ees o complexi y, such as he in e -pa ame e dependen-
cies p esen in Figu e 1 (lines 60-64). In o de o make a
ai compa ison be ween black-box and whi e-box es ing, we
selec ed only s a eless APIs, since he API s a e canno be
ese be ween es cases in black-box es ing (see Sec ion II-B
o de ails). Fo each API unde es , we s udied i s docu-
men a ion and ex ended i s OAS schema o explici ly de ine
in e -pa ame e dependencies using IDL4OAS [21].
B. Expe imen al Se up
Fo each case s udy, we e alua ed h ee echniques, namely:
•BB: The black-box app oach, ha is, he cons ain -based
es ing echnique implemen ed in RESTes .
•WB: The whi e-box app oach, namely, he sea ch-based
echnique implemen ed in E oMas e .
•HYBRID: Ou no el hyb id app oach, implemen ed as
an ex ension o bo h ools, i.e., RESTes ’s es cases a e
seeded in o E oMas e .
Table II shows he budge s wi h which we e alua ed each
es ing echnique. Fo HYBRID, he size o he seeds used is
also shown. Bo h me ics a e measu ed in e ms o HTTP calls,
and hey we e adjus ed acco ding o he SUT size (in e ms o
LOCs and numbe o endpoin s) since i s ongly de e mines
he ime equi ed o whi e-box heu is ics o e ol e owa d
good solu ions. Fo ins ance, YouTubeMock was assigned
wice he budge o RESTcoun ies because i has wice as
many LOCs, and Ohsome was assigned a minimum seed o
122 HTTP calls because i con ains 122 endpoin s. Fo BB,
we conside ed only he minimum o he wo budge s used o
WB and HYBRID, o make he expe imen s a o dable.6This
is in line wi h how black-box es ing is pe o med in p ac ice,
whe e esou ces a e o en cons ained due o es ic i e quo a
limi a ions imposed by comme cial APIs [8], [37].
Fo each SUT, we pe o med a o al o 9 expe imen s: one
o BB ( he minimum budge ), wo o WB (bo h budge s), and
six o HYBRID ( wo budge s × h ee seed sizes). In o de
o accoun o he andomness o he algo i hms, we epea ed
each expe imen 10 imes, ollowing he guidelines in [38]. As
an excep ion, some con igu a ions we e un only once, ei he
because hey ook oo much ime o comple e (e.g., 24 hou s)
o due o echnical limi a ions o he ools used (discussed in
Sec ions VI and VII).7Gi en ou SUTs, his led o 4 ×9×
6RESTes does no sh ink es sui es, and so i is ime-consuming o measu e
he co e age o millions o es cases, which mus be done manually and
dozens o imes, as explained u he on.
7Con igu a ions un only once: o Ohsome, BB; o LanguageTool, WB
wi h he high budge , HYBRID wi h he high budge and all seed sizes, and
HYBRID wi h he low budge and la ges seed.
TABLE III
RQ1: COMPARISON OF BB AND WB.
SUT Co e age (%) 5XX
BB WB ˆ
A12 p- alue BB WB ˆ
A12 p- alue
RESTcoun ies 74.9 73.6 0.68 0.136 0.0 0.5 0.25 0.014
YouTubeMock 81.9 26.0 1.00 <0.001 0.0 0.0 0.50 NaN
Ohsome 74.0 20.4 1.00 0.111 122.0 0.0 1.00 0.004
LanguageTool 46.6 35.3 1.00 <0.001 1.0 1.0 0.50 NaN
10 – (9 ×6) = 306 expe imen uns, o a o al o mo e han
40 million HTTP calls.
In o de o make a ai compa ison be ween he h ee ech-
niques, we manually execu ed he (hund eds o ) gene a ed es
sui es and measu ed he code co e age wi h In elliJ IDEA,8
a Ja a IDE. Fo he aul s ound, we elied on he numbe o
dis inc API endpoin s e u ning a leas one 5XX s a us code,
as done in [5], [7].
Tooling Se up: We pe o med he expe imen s wi h wo
open-sou ce ools, RESTes and E oMas e . Bo h ools equi e
some manual se up. In he case o RESTes , we modi ied he
es models o each SUT o include ealis ic es da a gene a-
o s o domain-speci ic pa ame e s (e.g., YouTube ideo IDs
o he YouTubeMock API). In he case o E oMas e , we
w o e he equi ed Ja a classes used by he co e componen
o s a , s op and ese he SUT. In bo h cases, we used he
de aul con igu a ion o he ools.
C. Resul s
To e alua e he e ec i eness o each es ing echnique,
we compu ed he pe cen age o code co e age achie ed (in
e ms o s a emen co e age o he whole applica ion) and
aul s ound (in e ms o he numbe o endpoin s e u ning
5XX s a us codes), ou o 10 uns. In o de o answe he
esea ch ques ions, we pe o med pai compa isons be ween
he echniques e alua ed, using he Va gha-Delaney e ec size
ˆ
A12 and he Mann-Whi ney-Wilcoxon U- es (a α= 0.05
signi icance le el) [38].
1) RQ1: Black-Box s Whi e-Box: Table III shows he
esul s achie ed by black-box and whi e-box es ing o he
lowe budge o HTTP calls (i.e., he only budge used
o BB). “NaN” alues ob ained o he p- alues mean ha
bo h echniques pe o med exac ly in he same way in all
cases (e.g., bo h BB and WB ound one aul y endpoin in
LanguageTool in he 10 expe imen uns).
In e ms o code co e age, BB ou pe o med WB wi h s ong
s a is ical e idence in 2 ou o 4 SUTs. Howe e , no e ha he
high p- alue o Ohsome is due o unning he BB echnique
only once. The la ges di e ences we e obse ed in YouTube-
Mock and Ohsome (>50% code co e age), because hese
SUTs p esen e y complex inpu cons ain s on and among
hei pa ame e s. These cons ain s need o be sa is ied in o de
o pass he inpu alida ion logic implemen ed in he sys em
and exe cise hei ac ual unc ionali y (e.g., compu ing he a ea
o a ci y in Ohsome). In ac , none o he eques s gene a ed
by E oMas e o hese SUTs ob ained a success ul esponse.
8h ps://www.je b ains.com/idea/
6
TABLE IV
CONFIGURATIONS OF HYBRID.
SUT Budge Co e age (%) 5XX
Min Med Max Min Med Max
RESTcoun ies 5K 79.5 79.7 79.4 0.9 0.9 0.8
100K 80.2 80.2 80.1 1.0 1.0 1.0
YouTubeMock 10K 68.9 81.4 86.3 0 0 0
200K 74.2 84.5 87.7 0 0.2 0.5
Ohsome 30K 68.7 74.6 75.1 45.3 116.1 122.0
600K 70.3 75.8 75.8 46.0 116.0 122.0
LanguageTool 100K 41.3 42.7 45.0 1.0 1.0 1.0
1M 43.0 43.0 45.0 1.0 1.0 1.0
In he case o RESTcoun ies and LanguageTool, hese SUTs
do no expose as complex cons ain s, and he e o e a sea ch-
based app oach can e en ually in e hose om he sou ce
code and gene a e high-co e age es cases.
In e ms o aul inding, BB and WB pe o med mo e simi-
la ly. Bo h o hem ound one aul 9in LanguageTool and none
in YouTubeMock, and each echnique ound unique bugs in
one SUT (WB unco e ed a aul y endpoin in RESTcoun ies
and BB ound aul s in all 122 endpoin s o Ohsome). An
endpoin may e u n a 500 s a us code due o mul iple aul s
in he SUT, o ice e sa, mul iple endpoin s migh ail due o
he same bug. Wi hou a manual in-dep h analysis, ou o he
scope o his pape , i is no possible o ell he exac numbe o
aul s ound by each o he (hund eds o ) es sui es gene a ed.
Howe e , an explo a o y analysis in he Ohsome API seems
o indica e ha mos o i s ailu es a e due o he same aul s.
All ope a ions ha e nea ly he same pa ame e s, he e o e all
endpoin s implemen mos ly he same alida ion logic, whe e
he aul s a e loca ed.
RQ1: Wi h a low budge o HTTP calls, black-box
cons ain -based es ing achie ed be ween 1.3% and
55.9% highe code co e age han whi e-box es ing. In
e ms o he aul s ound, bo h echniques pe o med
simila ly.
2) RQ2: Hyb id s Black-Box & Whi e-Box App oaches:
As p e iously explained, we e alua ed HYBRID wi h h ee
di e en seed sizes. Table IV shows he esul s ob ained o
each SUT, budge and he h ee seed sizes (min, med, and
9Recall ha , by “ aul ” o “bug”, we ac ually e e o “an endpoin e u ning
a leas one 5XX s a us code”, as done by p e ious au ho s [5], [7].
max) desc ibed in Table II. The highes alues on each ow a e
highligh ed in bold ace. The bes con igu a ion was ob ained
wi h he la ges seed, excep o RESTcoun ies, whe e he
medium seed yielded he bes esul s. This may be explained
by he ac ha he sea ch algo i hm may be o e i ed when
seeding many es cases, and ins ead i is necessa y o explo e
he sea ch space wi h mo e a ied es cases.
We compa ed he bes pe o ming con igu a ion o HYBRID
agains BB and WB, in pai s. Table V illus a es such com-
pa ison. Compa ed bo h o BB and WB, he imp o emen s
achie ed in he code co e age by ou no el hyb id app oach
a e s a is ically signi ican wi h s ong e ec sizes, in all
possible con igu a ions (all SUTs and budge s) excep o
LanguageTool, whe e BB pe o med bes . This di e ence may
be caused by he peculia i ies o his SUT. LanguageTool is
composed o hund eds o Ja a classes consis ing o “language
ules”, whose code is only execu ed when he ule is igge ed.
A ule is igge ed when a ex p o ided as inpu con ains a
mis ake ma ching he ule pa e n. Since BB uses a andom
ex gene a o , i may be mo e e ec i e han using a sea ch-
based app oach o e ol e s ings (which is no i ial [23],
[39]) in ways ha exe cise new code. O e all, and in aw
e ms, HYBRID co e ed 2.2% mo e code han BB and 28.8%
mo e code han WB.
Figu e 3 shows he co e age achie ed by BB (3a), WB (3b,
3d) and HYBRID (3c, 3e) a 5% in e als o he o e all budge
used. Fo WB and HYBRID, wo cha s a e shown, one o he
low budge (3b, 3c) and ano he o he high budge (3d, 3e).
This igu e e eals a signi ican di e ence in how he h ee
echniques pe o m as he budge is p og essi ely consumed.
While BB needs a small po ion o he budge o each a
co e age close o he maximum achie ed, he co e age o
WB and HYBRID e ol es mo e s eadily, because he sea ch
algo i hm keeps gene a ing es cases ha co e new a ge s
(e.g., sou ce code s a emen s and b anches).
Rega ding he aul s ound, we can say wi h high con idence
(i.e., p- alue <0.05) ha HYBRID pe o med be e han
BB in one scena io—RESTcoun ies, low budge —and be e
han WB in h ee scena ios, namely, YouTubeMock wi h he
high budge and Ohsome wi h bo h budge s. This esul in
he Ohsome SUT was expec ed, since he seeded es sui e
al eady con ained aul - e ealing es cases, which WB was
no able o gene a e. I is no ewo hy, howe e , ha HYBRID
unco e ed a bug in YouTubeMock no unco e ed by ei he
TABLE V
RQ2: COMPARISON OF HYBRID WITH BB AND WB.
SUT Budge Co e age (%) 5XX
HYBRID (h) BB (b) WB (w) ˆ
Ahb p- alue ˆ
Ahw p- alue HYBRID (h) BB (b) WB (w) ˆ
Ahb p- alue ˆ
Ahw p- alue
RESTcoun ies 5K 79.7 74.9 73.6 1.00 <0.001 0.98 <0.001 0.9 0.0 0.5 0.95 <0.001 0.70 0.064
100K 80.2 - 79.0 - - 0.88 0.002 1.0 - 1.0 - - 0.50 NaN
YouTubeMock 10K 86.3 81.9 26.0 1.00 <0.001 1.00 <0.001 0.0 0.0 0.0 0.50 NaN 0.50 NaN
200K 87.7 - 47.3 - - 1.00 <0.001 0.5 - 0.0 - - 0.75 0.014
Ohsome 30K 75.1 74.0 20.4 1.00 0.035 1.00 <0.001 122.0 122.0 0.0 0.50 NaN 1.00 <0.001
600K 75.8 - 21.2 - - 1.00 <0.001 122.0 - 0.2 - - 1.00 <0.001
LanguageTool 100K 45.0 46.6 35.3 0.00 0.111 1.00 0.195 1.0 1.0 1.0 0.50 NaN 0.50 NaN
1M 45.0 - 36.0 - - 1.00 1.000 1.0 - 1.0 - - 0.50 NaN
7
(a) BB, low budge . (b) WB, low budge . (c) HYBRID, low budge .
(d) WB, high budge . (e) HYBRID, high budge .
Fig. 3. T end o co e age e olu ion o he BB,WB and HYBRID app oaches. Fo BB, only he median es sui e (i.e., he i h bes pe o ming in e ms o
code co e age) is conside ed, since his had o be done manually. Fo WB and HYBRID, he a e age co e age o all es sui es is compu ed, since E oMas e
can ou pu hese s a s au oma ically.
BB no WB. This highligh s he po en ial o ou p oposal:
seeding an e olu iona y algo i hm wi h a ho ough es sui e
can p o ide a boo s ap o he sea ch, and e en ually make
i ind new aul s. E en in he cases whe e no new bugs we e
unco e ed, mo e code was co e ed in less ime. Fo example,
in RESTcoun ies, 5K HTTP calls su iced o co e 79.7%
code wi h HYBRID, while WB needed 100K HTTP calls o
each 79.0% co e age.
RQ2: Ou no el hyb id app oach co e ed be ween
1.2% and 60.3% mo e code han whi e-box es ing
in all SUTs, and be ween 1.1% and 4.8% mo e han
black-box es ing in 3 ou o 4 SUTs. In e ms o
aul inding, he hyb id app oach was he only one
unco e ing aul s in all SUTs.
VI. LESSONS LEARNED
In his sec ion, we p o ide lessons lea ned, de i ed om
he empi ical s udy. We may ema k ha hese lessons a e
es ic ed o he scope o ou p ac ical expe ience and u u e
esea ch will be needed o gene alizing hem u he .
Lesson 1: Black-box es ing is signi ican ly as e han whi e-
box es ing. Figu e 3 shows ha , o BB,∼5% o he budge
su ices o each ∼95% o he code co e age ob ained wi h
he whole budge . In con as , WB and HYBRID need mo e
budge , bu be e esul s could be ob ained. Mo eo e , since
WB and HYBRID apply se e al heu is ics o collec ing me ics
such as he s a emen co e age o he b anch dis ance [40],
his also ansla es in o a ime o e head. This was ex eme in
he case o LanguageTool, whe e a single HYBRID-1M-budge
expe imen un ook o e 10 hou s o comple e in a s anda d
PC.10 BB ook less han 3 hou s o gene a e a simila es sui e.
The e migh be some scena ios whe e ime can be limi ed.
Fo example, om a de elope ’s iewpoin , when de eloping
a new ea u e o an en e p ise applica ion, his mus be es ed
ho oughly be o e going in o p oduc ion. Taking in o accoun
he p e ious s a emen , ha is, ∼5% o he budge is enough
o each ∼95% o he code co e age in black-box es ing,
i is wo h conside ing he ollowing ques ion: wha would
de elope s p e e ? Ob aining a easonably ho ough es sui e
in 10 minu es, o a mo e exhaus i e es sui e in 10 hou s?
Which o he wo es sui es would be enough o e i y ha a
new ea u e did no b eak any hing? Bo h app oaches could be
complemen a y, e.g., using BB as a quick check and in eg a ing
HYBRID as a pa o a con inuous in eg a ion (CI) se up.
Lesson 2: Hyb id es ing is mo e e ec i e han black-box
and whi e-box es ing in isola ion. This is one o he main
conclusions de i ed om he e alua ion. In e ms o code
co e age, HYBRID achie ed he bes esul s in 3 ou o 4
SUTs, inc easing co e age by up o 4.8% and 60.3% compa ed
o BB and WB, espec i ely. In e ms o aul inding, HYBRID
was he only echnique unco e ing bugs in all SUTs. No e
ha HYBRID could also be applied wi h al eady exis ing es
sui es, e.g., one c ea ed by he SUT de elope s. E en small
es sui es can mean a signi ican boo s ap o he sea ch.
Fo ins ance, es sui es comp ising jus one es case pe
API endpoin in Ohsome (122 in o al) and 10 es cases in
YouTubeMock achie ed +50% code co e age han when no
seed was p o ided (Table V). C a ing such kind o es sui es
is a i ial ask, especially hanks o cu en ool suppo , which
10Technical speci ica ions: In el i5 CPU, 16GB RAM, 256GB SSD, Win-
dows 10, Ja a 8.
8
allows o do i manually (e.g., wi h he Swagge ool sui e [41]
o Pos man [32]) o au oma ically (e.g., wi h RESTes [14]).
This makes ou no el hyb id app oach eadily applicable in
p ac ice when a es sui e al eady exis s, and easily applicable
when i does no .
Lesson 3: Black-box cons ain -based es ing is gene ally
p e e ed o la ge and complex sys ems. Based on ou
empi ical s udy, WB is mo e e ec i e han BB o simple o
smalle SUTs when gi en high budge s (e.g., RESTcoun ies).
Howe e , when he SUT con ains in ica e inpu cons ain s,
he black-box app oach gene ally ou pe o ms he whi e-box,
because he gi en cus om domain knowledge in he da a
gene a o s and he handling o in e -pa ame e dependen-
cies [22] signi ican ly helps in dealing wi h hose cons ain s.
A sea ch algo i hm may s uggle o lea n how o gene a e
high-co e age inpu s in sho ime, due o complex b anch
condi ions implemen ed in he SUT code. This same limi a ion
applies o HYBRID as well (since i also le e ages he sou ce
code), and i became clea in LanguageTool. LanguageTool
is he bigges SUT, wi h 10 imes mo e LOCs han he o he
SUTs oge he . I is a CPU-bound applica ion (e.g., no CRUD
ope a ions on a da abase), doing complex compu a ions on
s ing ex inpu s. In his case, some expe imen uns o WB
and HYBRID e en ailed o inish. The E oMas e Ja a p ocess
an ou o memo y due o he p esence o e y long egula
exp essions in he code. E oMas e could no handle hose
(e.g., wi h mo e han 200K cha ac e s) and c ashed. In o he
cases, E oMas e simply imed ou due o e y long execu ion
imes (e.g., mo e han 24 hou s). All hese esul s emphasize
he ollowing: (1) whi e-box es ing is subjec o echnical
limi a ions, depending on he SUT; and (2) black-box es ing
is applicable ega dless o how he SUT is implemen ed, i s
size o i s complexi y.
Lesson 4: A manual se up is equi ed (and ecommended)
in all he s udied app oaches. In REST ul API es ing,
as in many o he disciplines, he e is a ade-o be ween
au oma ion and es ho oughness. App oaches achie ing he
highes au oma ion deg ee apply uzz o andom es ing [5],
[11] o le e age echniques ha a e simply no applicable in
all scena ios [10], [12]. The h ee echniques e alua ed in his
wo k equi e some manual se up. Wi hou a p ope empi ical
in es iga ion, i is ha d o ell which one equi es less wo k, bu
based on ou expe ience, E oMas e is quicke o se up han
RESTes . In ac , E oMas e ’s con igu a ion equi es w i ing
one Ja a class, while RESTes equi es modi ying a YAML
ile whose size depends on he numbe o API endpoin s and
pa ame e s. Howe e , as sugges ed by Table V, he manual
se up done o RESTes may play a key ole in he inal
esul s ob ained, especially o complex APIs dealing wi h
e y domain-speci ic pa ame e s. A e de elope s willing o do
some manual wo k o imp o e he es ing p ocess? O do hey
p e e 100% au oma ion? This is some hing o be empi ically
in es iga ed in u u e wo k.
Lesson 5: Whi e-box es sui es a e de e minis ic, black-
box es sui es a e usually no . One o he main di e ences
be ween whi e-box and black-box es ing is ha , in he o me ,
he s a e o he SUT can be ese be ween es cases (e.g.,
emp ying a da abase). The e o e, i is possible o c ea e a
comple ely de e minis ic es sui e, whe e he ou pu o e e y
es case does no depend a all on he es cases p e iously
execu ed. This is no possible in black-box es ing o s a e ul
APIs, whe e, o ins ance, he esponse o he 100 h API
eques depends en i ely on he p e ious 99 eques s execu ed.
This complica es debugging, since a aul - e ealing es case
may no e eal he same aul again, i he s a e o he SUT
has changed. Whi e-box and hyb id es ing can be used o
gene a e eg ession es sui es, whe eas black-box es ing is
mo e ypically applied o uzzing web se ices o gi en
pe iods o ime [11].
VII. OPEN CHALLENGES
In he nex pa ag aphs, we discuss open challenges o be
ackled by u u e esea ch in REST ul API es ing. These a e
aligned wi h wo p ima y goals: inc easing es au oma ion and
ho oughness.
Challenge 1: Highe deg ee o au oma ion. As p e iously
men ioned, he echniques e alua ed in his pape equi e some
manual se up. This is some hing ha may p e en p ac i ione s
om adop ing hem. Fo example, E oMas e equi es he
es e o w i e a d i e class de ining how o s a , s op
and ese he SUT, among o he s. Au oma ic code gene a ion
echniques could be applied o ease his ask. On he o he
hand, black-box es ing echniques in gene al may equi e he
addi ion o missing in o ma ion in he API speci ica ion, such
as in e -pa ame e dependencies. Ad anced echniques ha can
au oma ically ex ac his in o ma ion a e equi ed, e.g., by
analyzing eques - esponse pa e ns and in e ing in a ian s
p esen in he API, as hin ed in he wo k by Mi abella e
al. [42].
Challenge 2: In e ence o ealis ic es inpu s. Ve y ela ed
o he p e ious challenge, es ing REST ul APIs ho oughly
depends o a g ea ex en on he inpu s (HTTP eques s) used.
C ea ing and main aining da a dic iona ies o each pa ame e
is e y cos ly [9], [14]. Based on he API speci ica ion, which
usually desc ibes he a ailable endpoin s and pa ame e s, he e
is need o echniques ha exploi i o in e ealis ic alues
o each API pa ame e , e.g., wi h na u al language p ocessing
(NLP) capabili ies. The wo k o p e ious au ho s in GUI
es ing [43], [44] may inspi e u u e esea ch o add ess his
challenge.
Challenge 3: Ad anced whi e-box heu is ics. The e ec i e-
ness o whi e-box es ing, o e en i s applicabili y, highly
depends on he heu is ics used o le e age he sou ce code.
I hese a e no co ec ly implemen ed, hey may make whi e-
box es ing unusable (e.g., like he c ashes ha we ob ained
in E oMas e wi h LanguageTool). I he heu is ics a e oo
simple, whi e-box es ing may no be e ec i e. In E oMas e ,
ain checking and es abili y ans o ma ions, among o he s,
9