Compa ing he E ec i eness o Equi alence Pa i ioning, B anch Tes ing
and Code Reading by S epwise Abs ac ion Applied by Subjec s
N.
Ju is o1, S. Vegas1, M. Sola i2, S. Ab ahao3,1. Ramos4
1 Uni e sidad Poli ecnica de Mad id, 2Uni e sidad ORT U uguay,
3Uni e sidad Poli ecnica de Valencia, 4Uni e sidad de Se illa
^na aliajS egas^ i.upm.es, 2[email p o ec ed], 3[email p o ec ed], 4[email p o ec ed]s
Abs ac — Some e i ica ion and alida ion echniques ha e
been e alua ed bo h heo e ically and empi ically. Mos
empi ical s udies ha e been conduc ed wi hou subjec s,
passing o e any e ec es e s ha e when hey apply he
echniques. We ha e un an expe imen wi h s uden s o
e alua e he e ec i eness o h ee e i ica ion and alida ion
echniques (equi alence pa i ioning, b anch es ing and code
eading by s epwise abs ac ion). We ha e s udied how well
able he echniques a e o e eal de ec s in h ee p og ams.
We ha e eplica ed he expe imen eigh imes a di e en
si es.
Ou esul s show ha equi alence pa i ioning and
b anch es ing a e equally e ec i e and be e han code
eading by s epwise abs ac ion. The e ec i eness o code
eading by s epwise abs ac ion a ies signi ican ly om
p og am o p og am. Finally, we ha e iden i ied p ojec
con ex ual a iables ha should be conside ed when applying
any e i ica ion and alida ion echnique o o choose one
pa icula echnique.
Keywo ds-Ve i ica ion and alida ion, expe imen a ion,
combina ion o expe imen al esul s.
I. INTRODUCTION
Ve i ica ion and alida ion (V&V) is an impo an bu
expensi e pa o he so wa e de elopmen p ocess. The e
a e a wide ange o V&V echniques, each wi h di e en
ea u es. I is wo hwhile explo ing he p os and cons o he
echniques, ha is, which echniques a e be e sui ed o
pa icula so wa e p ojec cha ac e is ics, o example, o
e ealing a pa icula aul ype, o applica ion by a
pa icula de elope p o ile o o a pa icula so wa e ype.
These s eng hs and weaknesses depend on he a iables
a ec ing echnique e ec i eness [4].
Se e al empi ical s udies ha e looked a he beha io o
V&V echniques. In mos s udies, subjec s did no apply he
echniques, as we will see in he ela ed wo k sec ion. So a
hen, he onus o he esea ch has been on examining wha
we migh call pu e echnique beha io . We hink ha
s udies should add ess no only pu e echnique beha io bu
also he in luence o he human ac o on echnique
applica ion. This is especially impo an o V&V
echniques ha ha e no been ully au oma ed.
Mos empi ical s udies wi h subjec pa icipa ion ocus
on s a ic echniques [1], [7], [12], [33], [40], [48]. Only a
ew s udies ake dynamic echniques in o conside a ion [2],
[8],
[28], [34], [42]. As a esul o di e ences in con ex ual
condi ions and a low a e o in e nal and ex e nal
eplica ion, esul s a e ex emely imma u e. Fo his eason,
we canno di e en ia e o ui ous e en s om egula
pa e ns.
In his pape , we examine he e ec i eness o h ee
V&V echniques — wo dynamic echniques (equi alence
pa i ioning and b anch es ing) and one s a ic echnique
(code eading by s epwise abs ac ion)— applied o h ee
di e en p og ams. We ha e chosen hese echniques o
wo easons. We wan ed: 1) o compa e echniques wi h a
e y di e en app oach o so wa e V&V, and 2) o use
o mal enough echniques o gua an ee ha he esul s o
subjec s applying he same echnique we e compa able. We
ha e eplica ed he expe imen eigh imes a ou di e en
si es.
We ha e ound ha equi alence pa i ioning and
b anch es ing a e equally e ec i e and be e han code
eading by s epwise abs ac ion. Addi ionally, he
e ec i eness o code eading by s epwise abs ac ion a ies
signi ican ly om p og am o p og am. Finally, we ha e
iden i ied some key p ojec con ex ual a iables ha
in luence echnique e ec i eness.
The pape is o ganized as ollows. Sec ion 2 desc ibes
wo k ela ed o ou esea ch. Sec ion 3 explains he esea ch
me hod. Sec ion 4 desc ibes he expe imen and eplica ions.
Sec ions 5 and 6 p esen he esul s. Sec ion 7 ou lines he
indings. Finally, Sec ion 8 p esen s he conclusions.
II. RELATED WORK
The e a e se e al ecen s udies o V&V echnique
beha io . We di ide hese s udies in o h ee ca ego ies:
heo e ical s udies, empi ical s udies wi h subjec s, and
empi ical s udies wi hou subjec s.
The goal o heo e ical s udies is o examine
unadul e a ed echniques om he iewpoin o logic and
deduc i e easoning. They analyze echniques based on hei
heo e ical g oundwo k, s udying he e ec i eness o code-
based [9], [10], [13], [19], [23], [24], [36], [37], [41], [51],
[54],
[58] o eg ession echniques [43], [45].
Empi ical s udies wi hou subjec s aim o in es iga e
echniques om he angle o p ac ice and induc i e
easoning by simula ing echnique applica ion. They add ess
di e en aspec s o V&V echniques [27]. Some examine
es -case gene a ion and compa e he e iciency and
e ec i eness o speci ica ion-based, code-based, and aul
based echniques [6], [25], [38], [39], [52], [56]. O he s
e alua e es se s acco ding o di e en c i e ia (da a and
con ol low, mu a ion, and hei a ian s), measu e he
c i e ia co e age le el (adequacy) and ela e c i e ia
co e age and es case size o e ec i eness [15], [16], [17],
[18],
[30], [53]. Finally, some s udies e alua e es case
selec ion app oaches ( eg ession echniques [5], [46], [50]
il e ing [21], [29], [31] o p io i iza ion [14], [44], [47],
[55]).
Do e al. ha e buil an in as uc u e o help
esea che s o un such empi ical s udies [11].
Empi ical s udies wi h subjec s ake in o accoun how
he subjec in luences echnique beha io . Mos o hese
s udies e alua e s a ic echniques [1], [7], [12], [33], [40],
[48].
S udies e alua ing dynamic echniques ha e add essed
es -case gene a ion, compa ing he e iciency and
e ec i eness o speci ica ion-based and code-based
con ol low echniques applied by subjec s [2], [8], [28],
[34],
[42].
Empi ical s udies wi h and wi hou subjec s a e
necessa y. S udies wi h subjec s a e close o eal si ua ions
whe e echniques a e applied by es e s, bu , hey a e unable
o sepa a e ac ual echnique pe o mance om he in luence
ha he subjec migh be ha ing on echnique e ec i eness.
III.
RESEARCH
METHOD
We ha e conduc ed empi ical s udies wi h subjec s o
u he ou knowledge o h ee code e alua ion echniques.
We ha e un a se ies o expe imen eplica ions. The esul s
o a single expe imen could be a me e one-o coincidence.
An expe imen needs o be epea ed o check whe he he
obse ed esul s ollow a egula pa e n and a e
ep oducible [20].
We ha e eplica ed he expe imen eigh imes. Fi e
eplica ions we e un a he Uni e sidad Poli ecnica de
Mad id (UPM01, UPM02, UPM03, UPM04 and UPM05,
espec i ely). Addi ionally, he expe imen was conduc ed
a ano he h ee di e en si es: Uni e sidad de Se illa in
Spain (UdS05), Uni e sidad Poli ecnica de Valencia in
Spain (UPV05) and he Uni e sidad ORT in U uguay
(ORT05).
The numbe o subjec s in he expe imen eplica ions
was 42, 39, 29, 35 and 31 a UPM; 31 a UPV; 172 a UdS;
and 76 a ORT.
Addi ionally, we ha e held mee ings wi h he
esea che s p esen du ing he expe imen al ope a ion a
each si e o discuss de ia ions om he planned execu ion.
These mee ings ha e helped us o be e unde s and he
condi ions unde which subjec s apply he echniques.
To syn hesize he esul s o he eigh eplica ions, we
i s examine he combined esul s o all he eplica ions.
We iden i y he signi icance le el o he ac o s (and
in e ac ions), and we examine he beha io o he signi ican
ac o s iden i ied in each eplica ion agains he esponse
a iable means. To do his, we may need o e-examine
some o he esul s analyzed in each sepa a e expe imen o
un al e na i e analyses. Second, we s udy con ex ual
di e ences among he eplica ion si es. Finally, we in eg a e
he esul s o he wo s ages.
IV.
DESCRIPTION
OF THE
EXPERIMENT
A.
Expe imen al Design
Ou expe imen and i s eplica ions uses he expe imen
eplica ion packages buil by Kams ies and Lo [28], and
Rope and colleagues [42], al hough he ma e ial is adap ed
o ou expe imen al goals and con ex . The null hypo hesis
o he expe imen is:
H0: The e is no di e ence in he e ec i eness o
equi alence pa i ioning, b anch es ing and code eading
by s epwise abs ac ion.
Fo equi alence pa i ioning and b anch es ing, we
measu e he esponse a iable e ec i eness as he
pe cen age o subjec s ha a e able o gene a e a es case
ha unco e s he ailu e associa ed wi h a gi en aul . Fo
code eading by s epwise abs ac ion, we measu e
e ec i eness as he pe cen age o subjec s ha epo a
gi en aul .
The expe imen has a ac o ial design [26]. The
echnique is an expe imen ac o wi h h ee ea men s (o
le els):
equi alence pa i ioning (EP), b anch es ing (BT)
and code eading by s epwise abs ac ion (CR). The e
ollows a b ie eminde o hese echniques:
• Tes case design by equi alence pa i ioning1 [35] is a
wo-s ep p ocedu e: (1) iden i y he equi alence classes
and (2) de ine he es cases.
Equi alence classes a e iden i ied by aking each inpu
condi ion (usually a sen ence o s a emen in he
speci ica ion) and pa i ioning i in o wo o mo e g oups.
The e a e wo ypes o equi alence classes: alid classes
ep esen alid p og am inpu s, and in alid classes
ep esen e oneous inpu alues. Bo h alid and in alid
equi alence classes a e iden i ied acco ding o a se o
p ede ined heu is ics based on whe he he inpu condi ion
speci ies a ange o alues, a numbe o alues, a se o
inpu alues o a "mus be" si ua ion. I he e is any eason
o belie e ha he p og am does no handle elemen s in an
equi alence class iden ically, he equi alence class should
be spli in o smalle equi alence classes.
The second s ep is o use equi alence classes o iden i y
es cases. Tes cases ha co e as many o he unco e ed
alid equi alence classes as possible a e w i en un il all
alid equi alence classes ha e been co e ed by es cases.
New es cases ha co e one, and only one, o he
unco e ed in alid equi alence classes a e w i en un il all
in alid equi alence classes ha e been co e ed by es
cases.
• Whi e-box es ing is conce ned wi h he ex en o which
es cases exe cise he p og am logic. One possible logic-
co e age c i e ion is decision co e age o b anch es ing.
This c i e ion s a es [3] ha enough es cases mus be
w i en o assu e ha e e y b anch al e na i e is exe cised
a leas once. I enough es s a e un o mee his
1
Some
au ho s
belie e
he
e m
is a
misnome
and he
echnique
should
be
named
jus
pa i ioning.
We use he
e m
equi alence
pa i ioning
in
his
pape ,
as i is he
name
used
in he
e e enced book
ha
we use o
each
he
echnique
o he
s uden s.
p esc ip ion hen 100% b anch co e age is achie ed.
The es case design s a egy is as ollows. Fi s , an ini ial
es case is gene a ed ha co esponds o he simples ,
unc ionally sensible en y/exi pa h2. Ex a es cases a e
hen gene a ed. They should di e sligh ly om p e ious
pa hs.
Some pa hs should be a ou ed o e o he s: pa hs
ha do no ha e loops o e pa hs ha do, sho pa hs o e
long pa hs, simple pa hs o e complica ed pa hs, and pa hs
ha make sense o e pa hs ha do no . As he es cases
a e gene a ed, a able showing he co e age s a us o each
decision is buil .
• The p ocess o eading a p og am bo om up is called code
eading by
s epwise
abs ac ion [32]. Subjec s begin a he
lowes (mos de ailed) le el and eplace each p ime
(consecu i e lines o code) in he p og am by an
abs ac ion (o speci ica ion) ha summa izes i s possible
ou comes, i espec i e o i s in e nal con ol s uc u e and
da a ope a ions. In e media e abs ac ions a e successi ely
disco e ed a highe le els by using hose al eady ound.
Subjec s epea he p ocess un il hey ha e abs ac ed all
o he sou ce code. Nex , subjec s compa e he p og am
speci ica ion wi h hei abs ac ion (own speci ica ion) o
obse e inconsis encies be ween speci ied and expec ed
p og am beha io .
The expe imen uses h ee di e en p og ams, w i en
inC:
• cmdline is a pa se ha eads he inpu line and ou pu s a
summa y o
i s
con en s. I has 209 LOC and a cycloma ic
complexi y o
61.
• name bl implemen s he da a s uc u e and ope a ions o
a
symbol able. I has 172 LOC and a cycloma ic
complexi y o
29.
• n ee implemen s he da a s uc u e and ope a ions o an n-
a y ee. I has 146 LOC and a cycloma ic complexi y o
21.
Each p og am con ains se en di e en aul s. They a e
de ined acco ding o he six aul ypes iden i ied in [2].
Each aul ype is classed as omission (some hing ha is
missing) o commission (some hing ha is inco ec ). The
aul ypes used in ou expe imen a e:
•
Ini ializa ion.
Inco ec ini ializa ion o a da a s uc u e.
Fo example, assigning an inco ec alue o a a iable
when en e ing a module is a commission aul , whe eas
ailu e o ini ialize when necessa y is an omission aul .
Ou expe imen co e s ini ializa ion aul s o bo h
commission (F4) and omission (F3).
• Con ol. The p og am ollows an inco ec con ol low
pa h in a gi en si ua ion. Fo example, an inco ec
p edica e in an i - hen-else sen ence is a commission aul ,
whe eas a missing p edica e is an omission aul . Ou
expe imen co e s con ol aul s o bo h commission (F5)
and omission (F6).
•
Compu a ion.
These aul s lead o an inco ec calcula ion.
2 Sequence o ins uc ions o s a emen s ha s a s a he ou ine's en y
poin and ends a i s exi . A pa h may pass h ough se e al junc ions,
p ocesses, o decisions, one o mo e imes.
Fo example, an inco ec a i hme ic ope a o on he igh -
hand side o an assigna ion is a commission aul . The
expe imen will co e compu a ion aul s o commission
(F7).
•
Cosme ic:
Commission aul s can esul , o example, in a
spelling mis ake in an e o message. Omission aul s a e
aul s whe e an e o message should and does no appea .
The expe imen co e s cosme ic aul s o bo h
commission (F2) and omission (Fl).
We use wo e sions o each p og am in o de o
eplica e e e y aul wice o each p og am. The p og ams
a e small in size, and we canno seed as many aul s as we
would like o wi hou iola ing he aul masking p emise.
The only di e ence be ween wo e sions o he same
p og am is he aul ins ance ha hey con ain. The
p og ams always con ain he same numbe and ype o
aul s.
All de ec s ha e a 100% p obabili y o being de ec ed
by all h ee echniques.
The expe imen al p ocedu e consis s o se e al
sessions. Subjec s apply e e y echnique once. Subjec s
wo k on e e y p og am once. The e o e, a subjec applies
each echnique o one p og am. Subjec s pe o m he
ollowing asks:
• Subjec s applying equi alence class pa i ioning a e gi en
he p og am speci ica ion o design es cases. A e wa ds,
hey a e gi en an execu able e sion o he p og am and
hey un es cases. Subjec s iden i y ailu es in e ms o
inco ec ou pu s.
• Subjec s a e expec ed o ge as close as possible o 100%
b anch co e age. They a e gi en he sou ce code wi hou
i s speci ica ion o design es cases. A e wa ds, hey a e
gi en an execu able e sion o he p og am o un he es
cases.
They a e hen gi en he p og am speci ica ion o
check he es case ou pu s. Subjec s iden i y ailu es in
e ms o in alid ou pu s.
• Subjec s eading he code by s epwise abs ac ion a e
gi en he p og am sou ce code o iden i y abs ac ions and
gene a e a p og am speci ica ion. They a e hen gi en he
o iginal speci ica ion o iden i y he aul s in he p og am.
Subjec s iden i y aul s om inconsis encies be ween he
abs ac ed and o iginal speci ica ions.
B.
Desc ip ion o he Replica ions
Each si e has some con ex ual condi ions which a ec
expe imen al design. TABLE I shows each si e's con ex ual
condi ions, along wi h he design decisions. The e a e some
mino di e ences among he eplica ions a each si e:
• UPM: As he e a e no ime cons ain s, he expe imen
examines all h ee echniques and uses all h ee p og ams.
As he e a e enough compu e s o all subjec s, h ee
sessions a e held whe e subjec s indi idually apply he
echniques and, o dynamic echniques, un he es cases.
Subjec s apply one echnique o one p og am in each
session. Subjec s apply di e en echniques in each
session. This way, all six possible echnique applica ion
o de s (EP-BT-CR, EP-CR-BT, BT-EP-CR, BT-CR-EP,
CR-EP-BT and CR-BT-EP) a e co e ed ac oss he h ee
sessions. All h ee echniques a e exe cised du ing each
session, he echniques being applied o he same p og am.
No ice ha he day and p og am a e con ounded. Fo his
eason, subjec s apply p og ams
in
di e en o de s
in
UPM eplica ions (cmdline-n ee-name bl
in
UPM01,
UPM02
and
UPM04,
and
n ee-cmdline-name bl
in
UPM03
and
UPM05). Each session
has a
ou -hou
du a ion. This
is
equi alen o he e being no ime limi ,
as
i does
no
ake subjec s
as
long
o
comple e
he
ask.
Subjec s ecei e h ee ou -hou aining sessions
o
lea n
how o apply he echniques be o e he expe imen is un.
TABLE
I.
EXPERIMENT ADAPTATION TO SITES.
VARIABLE
Time
Compu e
a ailabili y
Subjec
p o ile
T aining
sequence
Session
leng h
CONDITION
Uncons ained
Cons ained
Yes
Limi ed
No
Familia wi h
echnique
Un amilia wi h
he echnique
Sequen ial
In e lea ed
Unlimi ed
Limi ed
SITE
UPM, UdS
UPV, ORT
UPM, UPV
UdS
ORT
UPV, UdS
UPM, ORT
UPM, ORT
UPV, UdS
UPM, ORT
UPV, UdS
DESIGN DECISION
3 echniques and
3
p og ams
2 echniques, and 2
o
3 p og ams
Wo k indi idually
Wo k
in
pai s
No es cases
un
Re eshe u o ial
Full cou se
on
echniques
Fi s aining, hen
expe imen
T aining in e lea ed
wi h expe imen
Tes cases un wi h
echnique applica ion.
Applica ion
o
all
echniques
in
1 session
by di e en subjec s
Tes cases
un in a
sepa a e session
o
only 1 p og am
• UdS:
As
he e
a e no
ime cons ain s,
he
expe imen
examines all h ee echniques and uses all h ee p og ams.
The expe imen
is
un
in
ou wo-hou sessions, because
session leng h
is
limi ed. Du ing
he
i s h ee sessions,
subjec s apply
one
echnique
o one
p og am. Subjec s
wo k
in
pai s
as
he e
a e no
enough compu e s
o all
subjec s. Subjec s apply
he
same echnique
o
di e en
p og ams
in
each session, since aining
has o be
in e lea ed wi h expe imen ope a ion.
As
session leng h
is limi ed, subjec s un es cases
o
one
o
he p og ams
ha hey ha e es ed wi h
a
dynamic echnique
in he
ou h session. As subjec s a e al eady acquain ed wi h he
echniques, aining consis s
o
h ee b ie wo-hou
u o ials. Each u o ial
is
held be o e echnique
applica ion.
• UPV:
As
he e
a e
ime cons ain s, code eading
by
s epwise abs ac ion
is
omi ed, bu all h ee p og ams a e
used.
The
expe imen
is un in
h ee wo-hou sessions
because session leng h
is
limi ed.
As
he e
a e
enough
compu e s, subjec s indi idually apply
one
echnique
o
one p og am du ing he i s wo sessions. As aining has
o
be
in e lea ed wi h expe imen ope a ion, subjec s
apply
he
same echnique
o
di e en p og ams
in
each
session. Subjec s
un
es cases
o
one
o
he p og ams
ha hey ha e es ed wi h
a
dynamic echnique
in
session
3.
As subjec s a e al eady acquain ed wi h he echniques,
aining consis s
o
wo b ie wo-hou u o ials. Each
u o ial is held be o e echnique applica ion.
• ORT:
As
he e
a e
ime cons ain s, code eading
by
s epwise abs ac ion
and he
n ee p og am
a e
omi ed.
The expe imen is un in one session. As session leng h
is
unlimi ed, subjec s indi idually apply he wo echniques
o
he wo
p og ams. Subjec s apply echniques
o
di e en p og ams and in di e en o de s o co e all ou
possible applica ion o de s (EP/cmdline-BT/name bl,
EP/name bl-BT/cmdline, BT/cmdline-EP/name bl,
and
BT/name bl-EP/cmdline). Subjec s ne e
un
es cases,
since hey
do no
ha e access
o
compu e s. Subjec s
ecei e h ee ou -hou aining sessions
o
lea n how
o
apply
he
echniques be o e
he
expe imen
is un.
Subjec s a e no e y skilled wi h p og amming issues.
TABLE II gi es a b ie desc ip ion o each eplica ion.
TABLE
II.
DESCRIPTION OF EACH REPLICATION.
Si e
Techniques
P og ams
Subjec s
Technique
applica ion
T aining
Sessions
Techniques
& p og ams
pe session
Session
wo kload
Tes case
execu ion
UPM
EP, BT, CR
cmdline
name bl
n ee
S^-yea
compu ing
s uden s
Indi idual
Full cou se
on
echniques
3
1 p og am
3 echniques
1 echnique
Same
session
UdS
EP, BT, CR
cmdline
name bl
n ee
S^-yea
compu ing
s uden s
Pai s
Re eshe
u o ial
4
1 echnique
3 p og ams
1 echnique
Sepa a e
session
UPV
EP, BT
cmdline
name bl
n ee
S^-yea
compu ing
s uden s
Indi idual
Re eshe
u o ial
3
1 echnique
3 p og ams
1 echnique
Sepa a e
session
ORT
EP, BT
cmdline
name bl
2nd-yea
compu ing
s uden s
Indi idual
Full cou se
on
echniques
1
2 echniques
2 p og ams
2 echniques
None
SPSS s a is ical package
has
been used
o he
da a
analysis. We use analysis o a iance (ANOVA)
o
analyze
he da a collec ed
in
he expe imen [26]. We wan
o
ind
ou i
a
gi en ac o (o combina ion o ac o s) is signi ican
a
a
0.05 signi icance le el. We can apply ANOVA because
he samples
o
all he eplica ions mee he equi emen s
o
no mali y
and
homogenei y
o
a iances3.
We use he
Bon e oni mul iple compa ison es
o
s udy he e ec
o
he ac o le els
on he
esponse a iable
a a 0.05
signi icance le el. Finally, based
on
he esul s
o
he i e
UPM eplica ions (whe e he e we e
no
con ex ual
a ia ions),
we se a
e e ence alue
o he
mean
and
s anda d de ia ion
o
de e mine whe he
he
alues
o a
eplica ion
a e
abno mally high/low.
As
wi h ou lie
de ec ion [57], alues ha do no all wi hin he ange o he
mean ±3 s anda d de ia ions
a e
conside ed
o be
di e en
(highe /lowe ) om he e e ence alue.
The comple e esul s
o he
s a is ical analyses
o can be
ound
a
h p://www.g ise.upm.eS/si es/ex as/4.
TABLE III. ANOVA SIGNIFICANCE LEVELS (P-
Model
Model powe
P og am
Technique
Ve sion
Faul
P og am * Faul
Technique * Faul
Ve sion * Faul
P og am * Technique
P og am * Ve sion
Technique * Ve sion
P og am * Technique * Faul
P og am * Ve sion * Faul
Technique * Ve sion * Faul
P og am * Technique "Ve sion
UPM01
0.000
1.000
0.041
0.000
0.153
0.071
0.000
0.001
0.491
0.013
0.295
0.074
0.094
0.125
0.348
0.410
UPM02
0.005
0.993
0.390
0.000
0.095
0.316
0.041
0.269
0.152
0.017
0.590
0.539
0.564
0.708
0.888
0.646
UPM03
0.004
0.995
0.163
0.000
0.123
0.882
0.057
0.032
0.044
0.005
0.811
0.459
0.102
0.242
0.847
0.752
VALUES) FOR ALL REPLICATIONS.
UPM04
0.014
0.979
0.907
0.000
0.323
0.009
0.002
0.472
0.361
0.048
0.156
0.904
0.113
0.075
0.779
0.800
UPM05
0.002
0.998
0.002
0.000
0.361
0.215
0.000
0.028
0.670
0.040
0.061
0.236
0.060
0.245
0.674
0.275
UdS05
0.008
0.982
0.009
0.000
0.070
0.046
0.023
0.037
0.078
0.141
0.760
0.034
0.172
0.212
0.210
0.230
UPV05
0.036
0.862
0.026
0.765
0.387
0.013
0.021
0.255
0.014
0.022
0.054
0.039
0.066
0.412
0.516
0.999
ORT05
0.004
0.992
0.003
0.003
0.635
0.000
0.001
0.264
0.092
0.014
0.710
0.221
0.028
0.019
0.309
0.896
Finally, a a mee ing wi h he esea che s esponsible o
unning he expe imen s, we examined possible de ia ions
o eal om planned expe imen execu ion and inciden s.
Ou s udy has some alidi y h ea s ha indica e ha
we ha e o be ca e ul abou how he esul s a e used, as:
• Resul s ha e been ob ained om junio es e s. I he
subjec s we e expe ienced p ac i ione s, esul s could be
di e en .
• Resul s ha e been ob ained om h ee p og ams w i en in
C. We e he p og ams o be la ge , o w i en in a di e en
p og amming language, esul s could be di e en .
• Resul s ha e been ob ained om a ce ain mode o
applying he echniques. Fo ins ance, no dynamic
analyse is used o apply b anch es ing.
• The a iables iden i ied du ing he mee ing wi h
esea che s unning he expe imen s ha could ha e a
possible in luence on esul s should be u he explo ed in
u u e expe imen s o check whe he hey do o do no
ha e an in luence.
V. EXPERIMENT RESULTS
TABLE III illus a es he signi icance le el (p- alue) o
he ANOVA o each eplica ion, as ou pu ed by SPPSS.
The i s wo ows in TABLE III espec i ely deno e ha
he ANOVA model is alid (i s alue is less han 0.05 in all
cases) and i s s a is ical powe is high (i s alue is g ea e
han 0.8 in all cases). The shaded cells indica e he
signi ican ac o s o combina ion o ac o s, ha is, ac o s
whose alue is lowe han 0.05.
To in e p e he esul s o he eigh eplica ions join ly,
we label each ac o o combina ion o ac o s wi h a alue
ha summa izes i s signi icance end. A ac o o
combina ion o ac o s has a signi ican end i i is
signi ican in 67%-100% o he eplica ions (6-8); has an
ambiguous end when i is signi ican in 34%-66% o he
eplica ions (3-5); and does no ha e a signi ican end i i
is signi ican in 0%-33% o he eplica ions (0-2). TABLE
IV shows he signi icance ends. We decided o use his
app oach and no me a-analysis, because he aw da a o he
expe imen was a ailable. This app oach would allow us o
ge mo e insigh s on he esul s han jus examining means
and s anda d de ia ions (as me a-analysis does). Le us
discuss signi ican and ambiguous ac o s o combina ions
o ac o s.
TABLE
IV.
SIGNIFICANCE
TRENDS.
Signi ican
(67%-100%)
Technique
Technique*P og am
P og am*Faul
Ambiguous
(34%-66%)
P og am
Faul
Technique*Faul
No signi ican
(0%-33%)
Ve sion
P og am*Ve sion
Technique*Ve sion
Faul *Ve sion
Th ee-way in e ac ions
A.
Fac o s and
Combina ions
wi h a Signi ican T end
The ac o s ha ha e a signi ican end a e echnique,
and echnique*p og am and p og am* aul combina ions.
Figu e 1 shows he mean e ec i eness alue o each
echnique a a 95% con idence le el in each eplica ion.
The s a is ical analyses show ha :
Figu e 1. Technique e ec i eness alues.
• E en in a bes -case scena io, none o he echniques
comes close o 100% e ec i eness, whe e he mean
e ec i eness among eplica ions is 79.26% o
equi alence pa i ioning,
78.43%
o b anch es ing and
47.23%
o code eading by s epwise abs ac ion.
• Code eading by s epwise abs ac ion is less e ec i e han
equi alence pa i ioning and b anch es ing in all he
eplica ions ha exe cise he eading echnique.

Cmdline
UPMDl UPM02 UPMD3 UPMD4 UPMD5 UDSD5 UPVD5 0RTD5
Name bl
UPMDl UPM02 UPMD3 UPMD4 UPMD5 UDSD5 UPVD5 0RTD5
N ee
UPMDl UPM02 UPMD3 UPMD4 UPMD5 UDSD5 UPVD5
•Equi alence Pa i ioning •B anch Tes ing
»CR
S epwise Abs ac ion
Figu e 2. Technique e ec i eness alues by p og am.
• The e ec i eness o equi alence pa i ioning and b anch
es ing is simila , excep in ORT05.
Some low e ec i eness alues (calcula ed as explained
in Sec ion 4) a e ci cled in Figu e 1: equi alence
pa i ioning in UdS05 and UPV05, and b anch es ing in
UPV05 and ORT05. This could be a sign ha he si e has
some so o in luence on echnique e ec i eness.
Figu e 2 shows he mean alue o echnique
e ec i eness by p og am in each eplica ion. The
echnique *p o g am in e ac ion is signi ican in all
eplica ions excep UdS05. The in e ac ion is o dinal
( ea men e ec s a e no equal ac oss all le els o ano he
ea men , bu hey a e always in he same di ec ion [22]),
meaning ha he main e ec s can be in e p e ed. The
s a is ical analyses show ha :
• Code eading by s epwise abs ac ion beha es wo se han
equi alence pa i ioning and b anch es ing o all
p og ams, whe e e ec i eness is much wo se o cmdline
and name bl and only sligh ly wo se o n ee.
• Equi alence pa i ioning and b anch es ing end o beha e
equally o all p og ams, excep o cmdline in UPM05
and UPV05, and name bl in ORT05.
• Some low e ec i eness alues o cmdline a e ci cled in
Figu e 2: b anch es ing in UdS05 and ORT05, and
equi alence pa i ioning in UPV05.
The p og am* aul in e ac ion is diso dinal ( he
di e ences be ween he le els o one ea men change
depending on how hey a e combined wi h le els o ano he
ea men [22]), and he main e ec s o he in e ac ion
canno be in e p e ed sepa a ely. The diso dinal in e ac ion
signi ies ha wo iden ical aul s do no ollow he same
pa e n in wo di e en p og ams. None heless, we looked
o a beha io al pa e n o he p og am* aul in e ac ion
ac oss he di e en eplica ions.
Again we we e unsuccess ul. The only plausible
explana ion o his diso dinal in e ac ion is ha he aul
classi ica ion used in he expe imen is no disc imina i e.
E en hough we ake ca e o use he same ype o aul s
(e.g., omission, con ol), hey beha e di e en ly in
combina ion wi h he echniques ac oss p og ams. The
echniques a e able o de ec he de ec s in some cases and
no in o he s, o which he e is no plausible aul ype-
ela ed explana ion.
B.
Fac o s and
Combina ions
wi h an Ambiguous T end
The ac o s ha do no ha e a clea ly signi ican o no
signi ican end a e p og am, aul and he echnique* aul
in e ac ion.
Figu e 3 shows he mean e ec i eness alue o each
p og am o a 95% con idence in e al in each o he
eplica ions. As he p og am* aul in e ac ion is diso dinal,
he indings a e limi ed. The p og am is signi ican in he
UPM01,
UPM05, UdS05, UPV05 and ORT05 eplica ions.
Howe e , he mul iple compa ison es s e eal ha he h ee
p og ams beha e equally in UPM01 and UPV05. E en
hough he signi icance le el is less han 0.05, he high
a iabili y in p og am beha io a he abo e si es means ha
no di e ences a e ound among he p og ams. Consequen ly
we ind di e ences in p og am beha io in only h ee ou o
he eigh eplica ions: UPM05, UdS05 and ORT05. We ind
ha :
85,00
50,00
UPM01 UPM02 UPM03 UPM04 UPM05 UDS05 UPV05 ORT05
•^^cmdline ""^"name bl ""^"n ee
Figu e 3. P og am e ec i eness alues.
• The e does no appea o be a beha io al pa e n in
eplica ions in which he ela i e e ec i eness o he
p og ams is signi ican .
• The e is a low e ec i eness alue ci cled in Figu e 3:
cmdline in UdS05.
Faul was signi ican in he UPM04, UdS05, UPV05
and ORT05 eplica ions. As he p og am* aul in e ac ion is
diso dinal, he indings a e again limi ed. Howe e , he
mul iple compa ison es s e eal ha aul s beha e equally
in UPM04 and
UdS05.
E en hough he signi icance le el is
less han 0.05, he high a iabili y in p og am beha io a
Fl F2 F3
N^
F4
K^V <y:
F5 F6 F7 • Equi alence Pa i ioning
• B anch Tes ing
•CR S epwise Abs ac ion
Figu e 4. Technique e ec i eness alues by aul .
he abo e si es means ha no di e ences a e ound among
he p og ams. Consequen ly, we ind di e ences in p og am
beha io in only wo ou o he eigh eplica ions: UPV05
and ORT05. No e ha he aul end is no longe
ambiguous, i is no signi ican . The e o e i makes no sense
o analyze his end.
Figu e 4 illus a es he mean alue o echnique
e ec i eness by aul in each eplica ion. The in e ac ion is
signi ican in he
UPM01,
UPM03, UPM05, and UdS05
eplica ions. F om mul iple compa ison es s we ind ha :
• Equi alence pa i ioning and b anch es ing a e equally
e ec i e i espec i e o he aul ype.
• Code eading by s epwise abs ac ion is causing he
in e ac ion, as i does no beha e equally o all aul s in all
eplica ions. We ind ha he e is a endency o aul s Fl
(cosme ic, omission), F3 (ini ializa ion, omission), F5
(con ol, commission) and F7 (compu a ion, commission)
o be mo e e ec i e han aul s F2 (cosme ic, commission),
F4 (ini ializa ion, commission) and F6 (con ol, omission).
• The e a e some ex eme e ec i eness alues o all aul s
excep F3 and F4. Howe e , he e is no clea beha io al
pa e n.
TABLE V summa izes he ele an esul s.
TABLE
V.
SUMMARY
OF
RELEVANT
RESULTS.
Tech.
Tech. x
P og.
Tech. x
Faul
UPM
(EP=BT)
>CR
(EP=BT) > CR
n : CR
UPM05: BT>EP
UdS
(EP=BT) > CR
EP P
(EP=BT) > CR
n : CR
cm: BT4*
UPV
EP=BT
EP l>, BT4<
cm: EP<BT
na, n : EP=BT
cm: EP4*
ORT
EP>BT
BTVP
cm: EP=BT
na: EP>BT
cm: BT4*
EP-BT i espec i e o aul ype
CR:
(Fl,
F3,
F5,
F7)
>
(F2,
F4, F6)
LEGEND: cm cmdline, na name bl, n n ee,
EP Equi alence pa i ioning, BT B anch es ing, CR: CR s epwise abs.
NAd ops,
^inc eases, = equal, > be e , < wo se
VI. DIFFERENCES OF CONTEXT
F om he analysis o he changes in he eplica ion
designs, we ind ha he ollowing a iables could be
in luencing he esul s.
•
Copying.
Subjec s apply he same echnique o di e en
p og ams in each expe imen al session a UdS and UPV.
As a esul , subjec s migh swap in o ma ion abou he
p og ams and hei aul s a he end o each session. As a
esul o copying, he echniques applied in sessions 2 and
3 could be mo e e ec i e a UdS and UPV han a o he
si es.
• Expe ience
in
p og amming issues. A ORT, expe imen al
subjec s ha e ha dly any p og amming expe ience. B anch
es ing gene a es es cases om sou ce code, whe eas
equi alence pa i ioning gene a es hem om he
speci ica ion. As a esul o p og amming expe ience,
b anch es ing could be less e ec i e a ORT han a o he
si es.
•
Ti edness.
A ORT, subjec s comple e he whole
expe imen in a single session, whe e hey ha e o apply
wo echniques. Subjec s could be i ed by he ime hey
come o apply he second echnique. This e ec is
cancelled ou by conside ing all possible applica ion
o de s. As a esul o
i edness,
howe e , echniques could
be less e ec i e a ORT han a o he si es.
• Wo k in pai s. A UdS, subjec s apply he echniques in
pai s a he han indi idually. As a esul o pai wo k,
echniques could be mo e e ec i e a UdS han a o he
si es.
F om he pos -execu ion mee ings held wi h esea che s
p esen du ing he expe imen execu ion, we iden i ied he
ollowing a iables ha could be in luencing he obse ed
esul s. Fo a de ailed desc ip ion o how all hese a iables
ha e been ob ained, see [49]:
•
T aining.
A UdS and UPV, he subjec s a e amilia wi h
he echnique and ecei e only a e eshe u o ial. A UPM
and ORT, subjec s ecei e a ull cou se. We disco e ed
ha he u o ials did no ensu e ha subjec s we e equally
knowledgeable abou he echniques. As a esul o
aining, echniques could less e ec i e a UdS and UPV
han a o he si es.
•
Mo i a ion.
Subjec s a UdS and UPV a e no as highly
mo i a ed because he expe imen has ha dly any impac
on he g ade o he cou se hey a e aking. A UPM and
ORT subjec s a e g aded on he expe imen al asks. As a
esul o mo i a ion, echniques could be less e ec i e a
UdS and UPV han a o he si es.
•
Time
p essu e. UPV subjec s did no ha e enough ime o
apply b anch es ing. I was he i s echnique applied, and
some ime was spen on explaining he expe imen
p ocedu e o he subjec s. Subjec s we e wo king unde
p essu e o ge he ask done on ime. As a esul o ime
p essu e, b anch es ing could be less e ec i e a UPV han
a o he si es.
TABLE VI summa izes he alues a each si e o he
possible in luencing a iables.
TABLE
VI.
POSSIBLE
INFLUENTIAL
VARIABLES.
T aining
Mo i a ion
Time
p essu e
Copying
Expe ience
in
p og amming
Ti edness
Wo k
in
pai s
UPM
Full
cou se
High
No
No
Yes
No
Indi idual
UdS
Re eshe
u o ial
Low
No
Yes
Yes
No
Pai
UPV
Re eshe
u o ial
Low
Yes (BT)
Yes
Yes
No
Indi idual
ORT
Full
cou se
High
No
No
No
Yes
Indi idual
These a iables can be g ouped in o h ee ca ego ies
depending on he ype o in luence hey ha e on he esul s:
a iables ha in luence all echniques; a iables ha
in luence a pa icula echnique, and a iables ha ha e no
in luence. Gene al-pu pose guidelines on how o apply V&V
o imp o e pe o mance can be o mula ed om a iables
ha in luence ei he all o none o he echniques. Guidelines
abou how o selec he bes echnique(s) o a gi en p ojec
con ex can be o mula ed om a iables ha in luence a
pa icula echnique.
VII. FINDINGS
We g oup ou indings by hei sou ce: expe imen
esul s, and combina ion o expe imen esul s wi h
con ex ual a iables. Depending on he eliabili y o he
esul s, hey a e also classed as e idence o signs.
A.
Expe imen al Resul s
We ha e ound s a is ical e idence ha :
• Equi alence pa i ioning and b anch es ing a e equally
e ec i e. E ec i eness is simila in se en ou o he eigh
eplica ions ( he excep ion being 0RT05).
• Equi alence pa i ioning and b anch es ing a e mo e
e ec i e when di e en subjec s apply he same
echnique han when he same subjec applies di e en
echniques. Techniques a e no 100% e ec i e ( heo e ical
likelihood) in any o he eplica ions.
• Equi alence pa i ioning and b anch es ing
e ec i eness is no dependen on he p og am. This is
ue in i e ou o he eigh eplica ions ( he excep ions
being
UPV05,
UPV05 and ORT05).
• Equi alence pa i ioning and b anch es ing a e mo e
e ec i e han code eading by s epwise abs ac ion.
Code eading by s epwise abs ac ion beha es wo se han
equi alence pa i ioning and b anch es ing in all six
eplica ions e alua ing his echnique.
• Code eading by s epwise abs ac ion should be applied
in conjunc ion wi h equi alence pa i ioning o b anch
es ing, as he e ec i eness o code eading by s epwise
abs ac ion is aul dependen . Fl (cosme ic, omission), F3
(ini ializa ion, omission), F5 (con ol, commission) and F7
(compu a ion, commission) gene ally end o beha e be e
han F2 (cosme ic, commission), F4 (ini ializa ion,
commission) and F6 (con ol, omission) in all eplica ions,
al hough he di e ence a ies om one eplica ion o
ano he .
• Faul de ec ion e ec i eness appea s o be p og am
dependen . The p og am was in luen ial in se en ou o
he eigh eplica ions, al hough we we e unable o ind a
beha io al pa e n. This sugges s ha he aul
classi ica ion ha we used was no disc imina i e, as
supposedly iden ical aul s (all Fls, all F2s, e c.) did no
beha e equally in all he p og ams. An al e na i e aul
classi ica ion scheme is equi ed o s udy e ec i eness.
Nei he is aul beha io consis en ac oss eplica ions.
Faul s ha subjec s ound e y easy ( e y ha d) o de ec in
one eplica ion we e no in o he s. This sugges s ha aul
beha io s a e un epea ed and e a ic, and he e is no
pa e n.
• P og am appea s o in luence he beha io o code
eading by s epwise abs ac ion. In all six eplica ions
examining his echnique, i is mo e e ec i e wi h n ee
han wi h he o he wo p og ams. A de ailed analysis o
n ee does no e eal any special ea u e ha migh single i
ou . Whe eas cmdline migh be expec ed o beha e
di e en ly, name bl and n ee a e in many espec s alike:
bo h a e p og ams ha implemen a da a s uc u e, e c. The
di e ence is mo e likely o be due o he aul s seeded in
n ee code, which, o some, as ye unknown, eason, a e
easie o subjec s o ind han o he s. I is su p ising ha
he e ec i eness esul s o code eading by s epwise
abs ac ion we e di e en e en wi h ex emely simila
p og ams. This would sugges ha he echnique is highly
sensi i e o p og am o aul cha ac e is ics.
• Technique e ec i eness is independen o he aul ype
in all eplica ions. Howe e , subjec s do no de ec he
same aul s, since all aul s a e being de ec ed by some
subjec s.
B.
Expe imen al Resul s and
Con ex
The e a e signs ha :
• P og amming inexpe ience dec eases he e ec i eness
o b anch es ing. In ORT05, b anch es ing was less
e ec i e han equi alence pa i ioning. ORT05 subjec s
we e no expe ienced wi h p og amming issues. This
di e ence is g ea e o he cmdline p og am, which is he
p og am wi h he g ea es cycloma ic complexi y.
• Technique knowledge could a ec echnique
e ec i eness. Subjec aining in UPV05 and UdS05 is no
as ho ough as a UPM and in ORT05. This could explain
why bo h echniques a e less e ec i e in UPV05, and
equi alence pa i ioning unde pe o ms in UdS05.
• Techniques migh be less e ec i e i subjec s a e no
mo i a ed. The subjec s in UPV05 and UdS05 a e less
mo i a ed han UPM and ORT05 subjec s. This could
explain why equi alence pa i ioning and b anch es ing
a e less e ec i e in UPV05, and equi alence pa i ioning
unde pe o ms in UdS05.
• Techniques a e mo e e ec i e wi h pai wo k. Al hough
mo i a ion and aining could be ha ing a nega i e
in luence, only equi alence pa i ioning beha es wo se in
UdS05 han in he o he eplica ions. This sugges s ha
pai wo k migh be o se ing any e ec s ha mo i a ion
and aining ha e on b anch es ing and code eading by
s epwise abs ac ion ha we e applied in pai s (no so in
he case o equi alence pa i ioning).
• Time p essu e does no appea o in luence b anch
es ing e ec i eness. The e ec i eness o equi alence
pa i ioning and b anch es ing is simila in UPV05. I
wo k unde p essu e had been in luen ial, b anch es ing
should ha e been less e ec i e since subjec s had less ime
o apply his echnique han equi alence es ing.
• Ti edness does no appea o in luence echnique
e ec i eness. I i edness we e o a ec e ec i eness, he
echniques should ha e been less e ec i e in ORT05, bu
hey we e no .
• P i ileged in o ma ion abou he p og am does no
appea o in luence echnique e ec i eness. In UPV05
and UdS05 subjec s had he chance o swap in o ma ion on
he p og ams a he end o each session. I p i ileged
in o ma ion we e o be in luen ial, equi alence pa i ioning
should ha e beha ed be e han b anch es ing in UPV05
and UdS05, and code eading by s epwise abs ac ion
be e han he dynamic echniques in UdS05.
VIII. CONCLUSIONS
We ha e un eigh eplica ions o he same expe imen
(some a o he si es) o y o unde s and he e ec i eness o
h ee V&V echniques: wo dynamic echniques
(equi alence pa i ioning and b anch co e age) and one
s a ic echnique (code eading by s epwise abs ac ion). We
analyzed he esul s by join ly in e p e ing esul s. We also
conduc ed a s udy o con ex ual di e ences be ween he
di e en si es a which he eplica ions we e un.
The indings e eal ha equi alence pa i ioning and
b anch es ing a e simila ly e ec i e and mo e e ec i e han
code eading by s epwise abs ac ion. Likewise, we
disco e ed ha equi alence pa i ioning and b anch es ing
a e independen o
he
aul ype. This sugges s ha i migh
be wo hwhile combining subjec s using he echniques o
inc ease echnique e ec i eness.
On he o he hand, we iden i ied con ex ual a iables ha
could be in luencing he esul s. T aining, mo i a ion, and
pai wo k could ha e an e ec on echnique e ec i eness,
and p og amming language expe ience could ha e an e ec
on b anch es ing's e ec i eness. T ansla ing hese esul s o
indus y, aining, mo i a ion and pai wo k could imp o e
es ing e ec i eness, whe eas language expe ience could
in luence he decision on whe he o use equi alence
pa i ioning o b anch es ing.
Mo e expe imen s wi h o he echniques would show
whe he he beha io o equi alence pa i ioning and b anch
es ing can be gene alized o black-box and whi e-box es ing
echniques.
REFERENCES
[I] V. R. Basili, S. G een, O. Lai enbe ge , F. Lanubile, F. Shull, S.
So umga d, M. V. Zelkowi z, The Empi ical In es iga ion o
Pe spec i e Based Reading. Jou nal o Empi ical So wa e
Enginee ing. 1(2):133-164, 1996.
[2] V.R. Basili, R.W. Selby. Compa ing he E ec i eness o So wa e
Tes ing S a egies. IEEE T ansac ions on So wa e Enginee ing,
13(2):1278-1296. 1987.
[3] B. Beize . So wa e Tes ing Techniques. In e na ional Thomson
Compu e P ess, Second Edi ion. 1990.
[4] A. Be olino. Guide o he Knowledge a ea o So wa e Tes ing.
So wa e Enginee ing Body o Knowledge. IEEE Compu e Socie y.
2004.
[5] J. Bible, G. Ro he mel, D. Rosenblum. A Compa a i e S udy o
Coa se- and Fine-G ained Sa e Reg ession Tes Selec ion. ACM
T ansac ions on So wa e Enginee ing and Me hodology
10(2):
149-
183.2001.
[6] J.M. Bieman, J.L. Schul z. An Empi ical E alua ion (and
Speci ica ion) o he All-du-pa hs Tes ing C i e ion. So wa e
Enginee ing Jou nal, Janua y 1992, pp.
43-51.
[7] S. Bi l, Analysis o he Impac o Reading Technique and Inspec o
Capabili y on Indi idual Inspec ion Pe o mance. 7*1 Asia-Paci ic
So wa e Enginee ing
Con e ence,
pp. 136-145. 2000.
[8] L.C. B iand, M. Pen a, Y. Labiche. Assessing and Imp o ing S a e-
Based Class Tes ing: A Se ies o Expe imen s. IEEE T ansac ions on
So wa e
Enginee ing.
30(ll):770-793. 2004.
[9] T.Y. Chen, Y.T. Yu. On he Rela ionships be ween Pa i ion and
Random Tes ing. IEEE T ansac ions on So wa e Enginee ing.
20(12):
977-980, 1994.
[10] L.A. Cla ke, A. Podgu ski, D.J. Richa dson, S.J. Zeil. A Fo mal
E alua ion o Da a Flow Pa h Selec ion C i e ia. IEEE T ansac ions
on So wa e Enginee ing. 15(11):1318-1332, 1989.
[II] H. Do, S. Elbaum, G. Ro he mel. Suppo ing Con olled
Expe imen a ion wi h Tes ing Techniques: An In as uc u e and i s
Po en ial Impac . Empi ical So wa e
Enginee ing.
10:405-435, 2005.
[12] A. Dunsmo e, M. Rope , M. Wood. Fu he In es iga ions in o he
De elopmen and E alua ion o Reading Techniques o Objec -
o ien ed Code Inspec ion. 24 h In e na ional Con e ence on So wa e
Enginee ing, pp. 47-57.2002.
[13] J.W. Du an, S.C. N a os. An E alua ion o Random Tes ing. IEEE
T ansac ions on So wa e Enginee ing. SE-10(4): 438—444, 1984.
[14] S. Elbaum, A.G. Mailshe sky, G. Ro he mel. P io i izing Tes Cases
o Reg ession Tes ing. In e na ional Symposium on So wa e Tes ing
and Analysis. 2000, pp. 102-112.
[15] P.G. F ankl, Y. Deng. Compa ison o Deli e ed Reliabili y o B anch,
Da a Flow and Ope a ional Tes ing, a case s udy. In e na ional
Symposium on So wa e Tes ing and Analysis. 2000.
[16] P. F ankl, O. Iakounenko. Fu he Empi ical S udies o Tes
E ec i eness. In e na ional Symposium on Founda ions on So wa e
Enginee ing. Pp 153-162. 1998
[17] P.G.F ankl, S.N. Weiss. An Expe imen al Compa ison o he
E ec i eness o B anch Tes ing and Da a Flow Tes ing. IEEE
T ansac ions on So wa e Enginee ing, 19(8):774-787. 1993.
[18] P.G. F ankl, S.N. Weiss, C. Hu. All-Uses s Mu a ion Tes ing: An