Con en s lis s a ailable a ScienceDi ec
In o ma ion and So wa e Technology
jou nal homepage: www.else ie .com/loca e/in so
E alua ing subse selec ion me hods o use case poin s es ima ion
Radek Silha y
⁎
, Pe Silha y, Zdenka P okopo a
Tomas Ba a Uni e si y in Zlin, Facul y o Applied In oma ics, Nad S anemi 4511, Zlin 76001, Czech Republic
ARTICLE INFO
Keywo ds:
So wa e De elopmen Effo Es ima ion
So wa e size es ima ion
Clus e ing echniques
Spec al Clus e ing
K-means
Mo ing Window
Use Case Poin s
ABSTRACT
When he Use Case Poin s me hod is used o so wa e effo es ima ion, use s a e aced wi h low model accu acy
which impac s on i s p ac ical applica ion. This s udy in es iga es he significance o using subse selec ion
me hods o he p edic ion accu acy o Mul iple Linea Reg ession models, ob ained by he s epwise app oach. K-
means, Spec al Clus e ing, he Gaussian Mix u e Model and Mo ing Window a e e alua ed as app op ia e
subse selec ion echniques. The me hods we e e alua ed acco ding o se e al e alua ion c i e ia and hen
s a is ically es ed. E alua ion was pe o ming on wo independen da ase s - which diffe in p ojec ypes and
size. Bo h we e cu by he hold-ou me hod. I clus e ing we e used, he aining se s we e clus e ed in o 3
classes; and, o each o class, an independen eg ession model was c ea ed. These we e la e used o he
p edic ion o es ing se s. I Mo ing Window was used, hen window o sizes 5, 10 and 15 we e es ed.
The esul s show ha clus e ing echniques dec ease p edic ion e o s significan ly when compa ed o Use
Case Poin s o mo ing windows me hods. Spec al Clus e ing was selec ed as he bes -pe o ming solu ion,
because i achie es a Sum o Squa ed E o s educ ion o 32% o he fi s da ase , and 98% o he second
da ase . The Mean Absolu e Pe cen age E o is less han 1% o he second da ase o Spec al Clus e ing; 9%
o mo ing window; and 27% o Use Case Poin s. When he fi s da ase is used, hen p edic ion e o s a e
significan ly highe –53% o Spec al Clus e ing, bu Use Case Poin s p oduces a 165% esul .
I can be concluded ha his s udy p o es subse selec ion echniques as a significan me hod o imp o ing
he p edic ion abili y o linea eg ession models - which a e used o so wa e de elopmen effo p edic ion. I
can also be concluded ha he clus e ing me hod pe o ms be e han he mo ing window me hod.
1. In oduc ion
So wa e De elopmen Effo Es ima ion me hods ep esen ac ual
opics ha a e c ucial o so wa e p ojec planning. I effo es ima ion
is no pe o med, i causes inco ec p ojec planning, which is an im-
po an isk in so wa e enginee ing p ojec managemen .
A he ea ly s age, only limi ed knowledge abou a so wa e p ojec
is a ailable. The p ojec scope is desc ibed in a limi ed way, and
usually, he e is no possibili y o ob aining a complex and alid in-
o ma ion - which should be use ul o es ima ion pu poses. The e o e,
he Use Case Poin (UCP) me hod [1], can be used o pe o m es ima-
ions. UCP is based on Use Case Model (UCM) s uc u ed scena io and
ac o s analysis. S uc u ed scena ios o ansac ions based ep esen a-
ion is a p e equisi e o UCP. The ac o s and use cases a e e alua ed;
he so wa e size is es ima ed based on an e alua ion o he UCM. UCP
conside s so wa e de elopmen effo s as linea co ela ed alue o
so wa e size [2].
In p e iously published s udies au ho s p esen s imp o emen s
[2–5] and simplifica ion [6–8] o an es ima ion p ocess.
In his pape , he ocus is on he issue o “how can we au oma ically
come up wi h he op imum subse o his o ical da a-poin s?”Me hods
o finding simila i ies o analogies be ween his o ical da a-poin s can
lead o educing an es ima ion e o .
The es o he a icle is s uc u ed as ollows: Sec ion 1 is an In-
oduc ion. Sec ion 2 defines he esea ch ques ions. Sec ion 3 desc ibes
he me hods p oposed o add ess he esea ch ques ions. Sec ion 4 in-
oduces he p ojec da ase s and expe imen al p ocedu e. Sec ion 5
p esen s he esul s ob ained. Finally, Sec ion 6 summa ises he con-
clusions and u u e wo k.
1.1. Rela ed wo k
Jo gensen and Sheppe d in [9], iden i y 11 es ima ion app oaches,
which a e mos ly based on s a is ical o nume ical models. Many o
hem a e based on linea eg ession. Many hese algo i hms a e based
on his o ical da ase s, whe e all a ailable p ojec s a e conside ed o
new es ima ion. Silha y e al. [2], p esen he Algo i hmic Op imisa ion
Me hod (AOM), which is based on linea eg ession and b ings
h ps://doi.o g/10.1016/j.in so .2017.12.009
Recei ed 30 June 2017; Recei ed in e ised o m 18 Decembe 2017; Accep ed 21 Decembe 2017
⁎
Co esponding au ho .
E-mail add esses: [email p o ec ed],[email p o ec ed] (R. Silha y).
In o ma ion and So wa e Technology 97 (2018) 1–9
A ailable online 28 Decembe 2017
0950-5849/ © 2017 The Au ho s. Published by Else ie B.V. This is an open access a icle unde he CC BY-NC-ND license (h p://c ea i ecommons.o g/licenses/BY-NC-ND/4.0/).
T
significan imp o emen s in compa ison o UCP. The AOM me hods
ou pe o m many o he me hods because o i s simplici y; only he
a iables known om UCP a e used in AOM. The second g oup e-
p esen s Machine Lea ning Algo i hms. Wen e al. [10], conduc ed a
sys ema ic li e a u e e iew in which hey iden i y 84 p ima y s udies
and 8 me hods o applied machine lea ning algo i hms. In his pape ,
he ocus is on me hods which a e applicable in conjunc ion wi h he
AOM me hod –o , mo e gene ally, wi h Leas Squa ed Reg ession
models (LSR), o Mul iple Linea Reg ession (MLR).
In [11], Silha y e al., ecen ly discussed a iable alidi y and
model selec ion. They decla e ha all a iables om he UCP me hod
a e unde s ood as explo a o y a iables, and he s epwise p ocedu e
allows one o selec he bes pe o ming model.
Id i e al. [12], p o ide a sys ema ic mapping s udy o Analogy-
based So wa e De elopmen Effo Es ima ion (ASEE). This pape ´s
au ho s in es iga e 65 s udies om 1990 o 2012. Mos o hese s udies
a e aimed a subse me hod selec ion. Id i e al., conclude ha ASEE
ou pe o ms mos algo i hms which used all his o ical da a-poin s
a ailable. ASEE me hod looks o simila i ies in his o ical p ojec s.
Clus e ing helps o find analogy among p ojec s - and is a b oadly in-
es iga ed me hod o educing he numbe o his o ical da a-poin s and
selec ing he mos simila subse . Azzeh and Nassi [13], deal wi h
se ing he numbe o nea es p ojec s. These au ho s ecommend a
me hod called Bisec ing k-medoids Clus e ing and ha e claimed ha
his me hod is be e han common ASEE me hods.
Azzeh e al. in hei pape [14], p esen a hyb id model ha consis s
o classifica ion and p edic ion s ages using a Suppo Vec o Machine
and Radial Basis Neu al Ne wo ks. They compa e he said model wi h k-
medoids. They ecommend ha ECF be omi ed om he es ima ion
and o ocus all es ima ion on he p oduc i i y ac o - which ep esen s
he a io be ween UCP, and de elopmen effo in pe son-hou s.
Ba disi i e al. [15], decla e ha clus e ing has a significan effec
on he accu acy o de elopmen effo es ima ion because i allows one
o omi i ele an p ojec s om his o ical da a-poin s.
P okopo a e al. [16], compa e k-means, hie a chical and densi y-
based clus e ing echniques wi h h ee diffe en dis ance me ics. The
esul s show ha all es ed clus e ing echniques imp o e es ima ion
accu acy and ha he numbe o clus e s plays a significan ole. I is
impo an o selec he clus e ing ype and dis ance me ic p ope ly.
The au ho s show ha hie a chical clus e ing has p oduced in-
app op ia e dis ibu ion o clus e s –and he e o e, canno be used. The
k-means clus e ing echnique wi h 4 clus e s and he cosine dis ance
me ic appea s o be he mos app op ia e me hod o a es ed da ase
and AOM model.
In [17], Ba disi i e al., imp o ed hei me hod by he combina ion
o ASEE and he Pa icle Swa m Op imiza ion (PSO) [18] algo i hm.
They in oduce a weigh ing sys em in which he p ojec a ibu es o
diffe en clus e s a e gi en diffe en weigh s. This app oach suppo s
he compa ison o a new p ojec only wi h p ojec s loca ed in ela ed
clus e s, based on he simila i y measu es. This me hod, like o he
me hods whe e a subse is selec ed, deals wi h se ing he co ec alue
o k o he nea es p ojec . Hihn e al. [19], desc ibed ha he nea es
neighbou me hod has significan ly mo e ou lie s han Spec al Clus-
e ing does.
Lokan and Mendes [20], in es iga ions showed ha mo ing win-
dows a e help ul as a subse selec ion echnique. Using 75 mos ecen
p ojec s o a new es ima ion makes his es ima ion mo e accu a e han
using all a ailable da a poin s. Amasaki and Lokan [21], la e compa e
mo ing windows o ASEE and MLR models. They ound MLR benefi s
mo e om windowing han analogy-based me hods.
2. P oblem s a emen
His o ical p ojec s in da ase s, e en i hey a e om iden ical o -
ganisa ion, should be dissimila , which should nega i ely influence he
model capabili y.
This esea ch s udy add esses a p oblem o selec ing a subse om
his o ical da ase s, which allows o cons uc a p edic ion model mo e
ele an o newly es ima ed so wa e de elopmen p ojec . The incon-
sis ence o his o ical da ase has nega i e impac o model pe o mance.
Es ima ion pe o mance and accu acy o he chosen subse selec-
ion echniques, oge he wi h an MLR model based on UCP a iables, is
compa ed.
Model ini ial configu a ion was de i ed om Silha y e al. [11].
Silha y e al., es ed se e al MLR models by using a s epwise linea
eg ession app oach and he Bes Pe o ming Reg ession Model (BPRM)
was selec ed. The ollowing me hods we e e alua ed and unde s ood as
subse selec ion echniques:
•k-means Clus e ing (k-means)
•Gaussian Mix u e Model Clus e ing (GMM)
•Spec al Clus e ing (SC)
•Mo ing Window (MW)
These echniques we e selec ed o demons a e he diffe ences be-
ween classical me hods (k-means, GMM), and he mode n one (SC),
wi h al e na i e app oach (MW). Selec ed clus e ing me hods a e a-
ou o nume ical da a [22] and we e p e iously used in he field
[16,19]. MW me hod was p e iously s udied by [23–25] an was in-
oduced as subse selec ion me hod o inc easing simila i y among
his o ical da a.
In Silha y e al. [11], he au ho s confi m ha all UCP a iables/
ea u es a e significan o so wa e de elopmen effo es ima ion.
The e o e, his s udy deals wi h UCP a iables. The UCP me hod
iden ifies:
•Unadjus ed Ac o s´ Weigh s (UAW)
•Unadjus ed Use Case Weigh s (UUCW)
•Technical Complexi y Fac o s (TCF)
•En i onmen al Complexi y Fac o s (ECF)
O he a ibu es a ailable in da ase s - including nominal a ibu es
we e no used, because nominal a ibu es, in gene al, canno be used
o new p ojec classifica ion o selec ed clus e .
The UCP me hod is based on assigning weigh s o clus e ed ac o s
and use cases. I employs h ee clus e ypes: simple, a e age, and
complex. The sum o he weigh ed ac o s c ea es a alue called UAW;
he UUCW alue is calcula ed simila ly. Two coefficien s, echnical
ac o s and en i onmen al ac o s, a e used o desc ibe he p ojec ,
ela ed in o ma ion, and he expe ience le el o he de elopmen eam.
Ac o s play oles in he UAW a iables [2,26]. A simple ac o y-
pically ep esen s an applica ion p og amming in e ace and a complex
ac o ep esen s a human using a g aphical use in e ace.
UAW and UUCW a e de i a e om UCM and he so wa e size is
based on ac o s o scena ios om use cases. TCF and ECF a e used o
desc ibe he p ojec , ela ed in o ma ion and he expe ience le el o he
de elopmen eam. An Adjus ed UCP (AUCP) sco e is ob ained by
summing he UAW and he UUCW, and hen mul iplying he esul ing
alue by he TCF and ECF (1). See [11,27] o a de ailed desc ip ion o
he UCP me hod.
=+ ××AUCP UAW UUCW TCF EC
F
() (1)
2.1. Resea ch ques ions and hypo hesis o mula ion
RQ1: Can be a p edic ion abili y o MLR models imp o ed by using
subse selec ion echnique?
RQ2: Which subse selec ion echnique is he bes , when compa ed
o UCP?
RQ3: A e mo ing windows equal o clus e ing echniques when
p edic ion accu acy is compa ed?
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
2
Le us assume ha , he Squa ed P edic ion E o s (SE) o Tes ed
Models (TM) - when he da a subse echnique is used, will be sig-
nifican ly lowe han UCP's SE. To decide whe he he model employing
subse selec ion echnique is mo e capable o p edic ion, a s a is ical
hypo hesis was es ed:
H
0
:=
μ
SE μSE
TM UCP; he e is no diffe ence in p edic ion capabili y
be ween UCP and TM wi h subse echniques. No diffe ence in he
mean o such p edic ion e o s.
Al e na i e hypo hesis:
H
1
:μSE
TM
<μSE
UCP
; he e is diffe ence in p edic ion capabili y
be ween UCP and he TM wi h subse echniques. The mean o
squa ed p edic ion e o s is significan ly lowe o TM han o he
UCP me hod.
This pape compa es he accu acy o he es ed models wi h ha o
he UCP me hod using a pai wo sample - es . The pai - es o wo
samples is used as a es o he null hypo hesis - ha he means o wo
no mally dis ibu ed popula ions ( wo sample) a e equal. The - es will
be used o he e alua ion o squa ed p edic ion e o s. Usage o - es
was p oo ed by analysis in [11].
2.2. E alua ion c i e ia
All he es ed me hods we e e alua ed acco ding o (2) Mean Ab-
solu e Pe cen age E o (MAPE), (3) he Sum o Squa ed E o s (SSE),
(4) Mean Squa ed E o (MSE), (5) Roo Mean Squa ed E o (RMSE)
and (6) No malised Mean Squa ed E o (NRMSE). MAPE was selec ed,
because o in [28] au ho s p oo s ha MAPE has p ac ical and heo-
e ical ele ance o e alua ion o eg ession models and i s in ui i e
in e p e a ion in e ms o ela i e e o . Whe eas Mean Rela i e E o
(MRE) is no always op imal o desc ibing he es ima ion accu acy [7].
MRE some ime omi s o e es ima ion, which lowe s usabili y o so -
wa e de elopmen effo es ima ion [29,30]. SSE is used because o i s
abili y o ep esen e o s o selec ed da ase s. C oss-da ase compa -
abili y is weak (numbe o n mus be handled), bu o e all i illus a es
he model p edic ion pe o mance.
MSE is seen as he bes pe o ming cos unc ion in [8], which guide
us o include i as possible pe o mance measu emen . RMSE and
NRMSE a e included o illus a ion pu poses o u u e compa abili y
o o he models es ed on gi en da ase s. The equa ions a e gi en as
ollows:

∑
=−×
=
n
yy
y
M
APE 1100
i
n
ii
i
1
(2)
∑
==
SSE ɛ
i
n
i
1
2
(3)
∑
==
M
SE n
1ɛ
i
n
i
1
2
(4)
=∑
=
R
MSE n
ɛ
i
n
i
1
2
(5)
=
N
RMSE RMSE
n(6)
Whe e nis he numbe o obse a ions, y
i
is he known eal alue,

yi
is he p edic ed alue and ɛis he p edic ion e o alue.
3. Me hods used
3.1. K-means Clus e ing
K-means is a me hod o finding clus e s in unlabelled da a poin s.
The numbe o clus e s has o be se be o e he algo i hm s a s. This
app oach is based on finding da a cen es, whe e he ini ial cen es a e
se andomly. The cen es a e i e a i ely mo ed o minimize da a a -
iance in one clus e [31]. The mean alue o each ea u e is calcula ed,
and hose means a e se as new clus e cen es. In ac , i finds a subse
o each cen e o which he Cos Func ion (CF) o dissimila i y is
minimized. In (7), he cos unc ion in i s gene ic o m can be seen:
∑∑
=
=∈
CF d x c(,
)
k
K
xG
k
1k
(7)
Whe e, K ep esen s he p edefined numbe o clus e s; G
k
e-
p esen s a clus e ; d(8) is he dis ance/dissimila i y unc ion; xis a
ec o ; and c
k
ep esen s a clus e cen e:
=− × −
d
xc x c Mx c(, ) ( ) (
)
kk
Tk(8)
Whe e, M is a dis ance ma ix. The K-means app oach does no
gua an ee ha one will ob ain an op imal solu ion. In many cases, a
sub-op imal solu ion is ound. The e is a endency in many i e a i e
algo i hms - including k-means, no o be con e gen wi h he global
minima. The e o e, i is used o un he k-means p ocess se e al imes –
and, as esul , a solu ion esul ing om se e al uns is aken.
3.2. Gaussians Mix u e Model Clus e ing
The Gaussian Mix u e Model [32], (GMM), p esen s a model-based
app oach o clus e ing. GMM is unde s ood as P obabilis ic Clus e ing.
In model-based clus e s, a model o clus e and op imisa ion be ween
da a and model is esea ched. Each clus e is ep esen ed by i s pa a-
me ic dis ibu ion (componen dis ibu ion). GMM can be used in ad-
ance in dominan pa e n ecogni ions since hese dominan pa e ns
a e ela ed o he componen dis ibu ion. (9), shows a mul i a ia e
Gaussian equa ion:
N
=−−−
−
{}
xμ πxμ x(,Σ) 1
(2 ) Σ exp 1
2()Σ(Σ)
d
T
/2 1/2
1
(9)
Whe e, μis, a componen mean; Σis he co a iance ma ix; and x
ep esen s a componen . Mixing confiden s can be in e p e ed by using
(10). Fo GMM Clus e ing, he p ocess is in e ed and pa ame e s like
mixing confiden s, means and co a iance a e esea ched:
∑
==
p
xpkpxk() () (
)
k
K
1(10)
Then, a maximum likelihood unc ion is used. The maximum o log-
likelihood (11), is esol ed by using an Expec a ion-Maximiza ion (EM)
algo i hm, which desc ibes an i e a i e scheme o he maximisa ion o
he log-likelihood unc ion. Fi s ly, he ini ial ques o he pa ame e s is
needed:
N
∑∑
=⎧
⎨
⎩⎫
⎬
⎭
==
Dπ μ ln π π μ
l
np( , ,Σ) ( ,Σ)
n
N
k
K
kk
kk
11 (11)
EM needs an e-s eps app oach o he pa ame e s - which a e la en
a iables and m-s eps o he expec ed comple e log-likelihood. In each
i e a ion, he pa ame e s a e upda ed. The algo i hm i e a es un il he
log-likelihood o he da a emains fixed. EM ends o s ay wi hin he
local op imal alue. The e o e, a new ini ial se ing is ecommended o
unning he algo i hm again; and e en se e al uns emain wi hin he
same op imum, and he global op imum migh be ound.
3.3. Spec al Clus e ing
The Spec al Clus e ing [33] algo i hm is an Unsupe ised Clus-
e ing me hod, based on a g aphical ep esen a ion, whe e each da a
poin is a node and he edges be ween da a poin s ep esen simila i y,
see G aph G (12). This ep esen s a Deg ee Diagonal Ma ix, in which a
cell ep esen s a sum o weigh s co esponding o each node om he
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
3
g aph - o espec i ely, a cell o ma ix W:
=
G
VE(,
)
(12)
Whe e, se V con ains e exes
i
and se E he edges e
i
, which e-
p esen da a poin s. Two e exes a e connec ed i he simila i y s
ij
be ween he co esponding da a poin s x
i
and x
j
a e la ge o equal o
he h eshold; and he edge is weigh ed by s
ij
. This means ha pa o
he g aph whe e edges wi h e y low weigh s a e ound.
The K-nea es neighbou g aph, ɛ-neighbo hood g aph and he ully-
connec ed g aph a e ypically used in Spec al Clus e ing [34]. The k-
nea es neighbou g aph connec s
i
and
j
e exes, whe e
j
is one o
he k-nea es e exes o
i
. The ɛ-neighbo hood g aph connec s all da a
poin s whe e pai wise dis ances a e smalle han ɛ. La e , he Adjacency
Ma ix W (13) is c ea ed:
=
W
w(
)
ij
(13)
Whe e,
=
i
j
n
,1..
and each cell in he ma ix co espond o he
edge’weigh be ween wo da a poin s. I he weigh is 0, hen he e is
no connec ion be ween he edges. Finally, a Laplacian Ma ix (14) is
calcula ed:
=−LD
W
(14)
Whe e, Dis he diagonal ma ix o he deg ee o e ex
i
. The L
ma ix is used o spec um calcula ion - which is a key poin in spec al
clus e ing algo i hms. The Lma ix is used in un-no malized algo-
i hms; when a no malized Laplacian algo i hm is used, he e a e wo
possibili ies (15) –a symme ic ma ix and a andom-walk:
==−
==−
−− − −
−−
LDLD IDWD
LDLIDW
:
:
sym
w
1/2 1/2 1/2 1/2
11 (15)
The spec um is a so ed lis o he eigen ec o s o a L,L
sym
o L
w
ma ix. In ac , he eigen ec o s ep esen a da a-poin o a da a-se and
an eigen alue o a L,L
sym
o L
w
ma ix. Spec al Clus e ing uses hese
eigen ec o s as a ea u e. The clus e ing o ea u es can be pe o med
by any known algo i hm. In his pape , he k-means algo i hm is used.
3.4. Classi ying in o clus e s
Mul iclass Linea Disc iminan Analysis, (MLDA), is used o he
classifica ion o new da a-poin s o exis ing clus e s. Each new p ojec ,
which is ep esen ed as a new ow in he es ing se , is classified ac-
co ding o he clus e s ha we e ob ained om a aining se . The
es ing se and aining se mus be ma ices, wi h he same numbe o
columns. In his case, mo e han wo g oups a e used, which leads o
seeking ou mo e p ojec ions. The e o e, he WP (16) p ojec ion ma ix
is a anged as ollows:
=
W
PW
X
T(16)
Whe e, W
T
ep esen s a T ansposed Adjacency Ma ix and Xis he
es ing se da a poin ma ix. The fi s s ep, (17), is used o he in e -
class sca e ma ix

Σ
b
[35]:

∑
=−×−
′
=mx x x x
Σ
()()
b
i
n
ii i
1(17)
Whe e, m
i
is he numbe o da a poin s in he aining se o each
clus e ;
x
i
is he mean o each clus e ; and
x
is he mean ec o . The
linea ans o ma ion is hen pe o med - whe e he key poin is o
maximize (17), and sol e as a gene alized eigen alue p oblem.
The classifica ion i sel , is hen pe o med in he ans o med space
by using me ics like Euclidean Dis ance o Cosine Dis ance. Newly-
ob ained ins ances a e classified by minimizing a dis ance unc ion wi h
espec o a k h clus e cen oid; and a mean alue:
x
k
.
3.5. Mo ing Window
Mo ing Window is a subse echnique ha can be unde s ood in wo
ypes. The fi s ype o MW is based on du a ion - and he second is
based on he numbe o p ojec s [20]. The Du a ion-based app oach
means, a aining se p ojec , a finished in a specific ime ame, a e
used as pa ame e s. I he e is a la ge numbe o finished p ojec s, hen
his is he mos use ul app oach. O he wise, i he e is a low numbe o
finished p ojec s, hen he “numbe o p ojec s”is only an op ion.
Fig. 1 illus a es he window unc ion applica ion is. I e a ion 0 is
desc ibed as an ini ial pa i ioning o a da ase - (n= 70), o a aining
se - (n= 49), and he es ing pa , (n= 21). The aining da ase ex-
pands a e a p ojec is finished, (= da a-poin in es ing se ), and 5
mos ecen da a-poin s a e selec ed (see window).
In I e a ion 1, he 5 mos - ecen p ojec s, (44-49), a e selec ed - and
used o c ea e he model. The model is used o he es ima ion o P ojec
50 - ( he fi s one in he es ing se ).
In I e a ion 2, P ojec 51 is es ima ed, and he model is cons uc ed
based upon P ojec s 45-49 om he aining se and 50 om he es ing
se . This esul , once again, in 5 he mos ecen p ojec s. In I e a ion 3,
P ojec 52 is es ima ed, by means o a model which is based on P ojec s
46-49 om he aining se and on P ojec s 50 and 51 om he es ing
se . In his illus a i e example, i is expec ed ha he p ojec s a e
comple ed one by one.
I mo e han 1 p ojec is finished be o e he nex es ima ion is
pe o med, hen all finished p ojec s a e appended o he end o he
aining da ase .
Simila ly, i mo e han 1 p ojec mus be es ima ed in each i e a ion
a window can be used, which means ha all p ojec s a e es ima ed by
using an iden ical model.
Window-size ep esen s he numbe o mos ecen p ojec s in he
aining se . I window-size is n= 5, hen he 5 mos - ecen da a-poin s
a e used o he MLR model cons uc ion:
Au ho s o [36] ecommend ha he size o he aining se , ( he
numbe o finished p ojec s) should be a ound 30. Amasaki e al. [21]
discuss windows sizes and conclude ha 75 da a poin s p o ides bes
a ailable esul s. Window size depends on da ase size, his is a eason
why in his s udy only smalle windows a e es ed. I is expec ed ha
he ele ancy o ea lie p ojec s dec eases o e ime. This claim can
only be made i he e is an assump ion o inc easing es ima ion accu-
acy, o i changes in size o p ojec ypes a e expec ed.
3.6. S epwise Linea Reg ession
SLR is employed o selec a BPRM by means o an au oma ic p o-
cedu e. SLR is some imes biased o he significance le el in s a is ical
assump ions. The bes model is selec ed when he e alua ion c i e ia
a e ulfilled. On he o he hand, SLR should also help o educe he
numbe o p edic o s used in he selec ed model.
Fig. 1. Mo ing Window –p ojec selec ion.
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
4
S epwise eg ession is based on he o wa d - and backwa d selec-
ion ha in ol es an au oma ic p ocess o he selec ion o independen
a iables; and can be b iefly desc ibed as ollows:
(1) Se a s a ing model, which con ains p edefined e ms (backwa d);
o se a null model ( o wa d);
(2) Se limi s o he final model—wha ype o model is needed?,
whe he linea e ms a e used?, squa ed e ms?, o ice- e sa?;
(3) Se an e alua ion h eshold, (p alue o x, Sum o Squa ed E o s, o
simila s a is ics) e o is significan ly dec eased);
(4) Adding o emo ing e ms; e es ing he model;
(5) S epwise eg ession hal s when no u he imp o emen in es ima-
ion occu s.
Fig. 2 depic s a schema o he s epwise p ocess. The e is a mod-
ifica ion o o wa d selec ion - such ha a e each s ep in which a
a iable xis added, all he candida e a iables in he model a e checked
o see whe he hei significance phas been educed below a specified
h eshold p
en e
(uppe bound), o p
emo e
(lowe bound). Fo wa d se-
lec ion s a s as a null model and hen i e a es o add each a iable
which mee a condi ion. When a non-significan a iable is ound, i is
emo ed om he model. Backwa d selec ion wo ks in a simila
manne , bu emo es a iables when hey a e ound o be non-sig-
nifican . The e o e, s epwise eg ession equi es wo significance le els:
he fi s , o adding a iables; and he second, o emo ing a iables.
SLR only is a me hod o building many models om a combina ion
o p edic o s, he e o e he mul iple linea eg ession assump ion has o
be ulfilled. The mul iple linea eg ession model, (18),isdefined as:
=+ + +…+ +yβ βX βX βX ɛ
iii pip i
01
122
(18)
Whe e,
=…
i
ny1, ,
i
is he dependen a iable;
…XX
ii
p
1
a e in-
dependen a iables, (p edic o s); β
0
is an in e cep ; and, …
β
β
n
1a e
eg ession coefficien s. The alue o ɛ
i
ep esen s he esiduals. The
model is designed as a ma ix - whe e each ow ep esen s a da a-poin .
When MLR is a polynomial eg ession, (19); hen he ela ionship
be ween he dependen a iables and he independen a iable is
modelled as an m h deg ee polynomial:
=+ + +…+ +yβ βX βX βX ɛ
iiipip
mi
01
122
2(19)
I o dina y Leas Squa e Es ima ion is used, he ec o o he es i-
ma ed eg ession coefficien s is in a ma ix o m - and can be defined as
ollows (20):

=
−
X
β
XXy()
TT1
(20)
4. Expe imen design
4.1. P ojec da ase s
Me hods, p e iously desc ibed abo e, we e e alua ed using wo
da ase s. Da ase 1 - (DS1), was ob ained om [2] –in which he da-
ase was based on [6] and [37]. Da ase 2 - (DS2), was collec ed by he
au ho s and was fi s published - (in pa ), in [11].
Fo he pu pose o his s udy, we do no agg ega e da a by da a-
dona o s o p oblem-domains. All p ojec s we e o de ed by de elop-
men comple ion da e, o each dona o . Fig. 3 shows a boxplo o he
da ase s. As shown, he wo da ase s ep esen g oupings o diffe en
p ojec sizes.
Table 1 shows he cha ac e is ics o he da ase s used in he ex-
pe imen s. As shown, bo h da ase s ha e a simila s anda d de ia ion,
bu DS2 con ains la ge p ojec s. All he alues in Table 1 a e based on
he Real_P20 [11], which desc ibes he eal p ojec size in poin s (UCP).
Da ase s con ain ollowing a iables:
•P ojec _No - p ojec ID o iden ifica ion pu poses
•Simple Ac o s - Numbe o ac o classi y acco ding UCP - simple
ac o s
•A e age Ac o s - Numbe o ac o classi y acco ding UCP - a e age
ac o s
•Complex Ac o s - Numbe o ac o classi y acco ding UCP - complex
ac o s
•UAW - Unadjus ed Ac o weigh , compu ed by using UCP equa ion
•Simple UC - Numbe o use cases classified as simple - UCP numbe
o s eps is used A e age UC - Numbe o use cases classified as
a e age - UCP numbe o s eps is used
•Complex UC - Numbe o use cases classified as complex - UCP
numbe o s eps is used
•UUCW - Unadjus ed Use Case Weigh - compu ed by using UCP
equa ion
•TCF - Technical Complexi y Fac o
•T1-T13 - Technical ac o s significance
•ECF - En i onmen al Complexi y Fac o s
•ENV1 - ENV8 - En i onmen al ac o s significance alue
•Real_P20 - Real Effo in Pe son hou s, di ided by p oduc i i y
ac o (pe son hou s pe 1 UCP) (PF = 20)
•Real_Effo _Pe son_Hou s - Real Effo (de elopmen ime) in
pe son-hou s
•Sec o - P oblem domain o p ojec
•Language - P og amming language used o p ojec
•Me hodology - De elopmen me hodology used o p ojec de el-
opmen
•Applica ionType - Classifica ion o p ojec ype - p o ided by do-
na o
•Da aDona o - Anonymized ac onym o da a dona o
In Table 1 Da ase s’s a is ical cha ac e is ics can be seen. Median
Pe son-Hou s illus a es man alue o p ojec de elopmen ime, which
Fig. 2. The S epwise Reg ession P ocess.
Fig. 3. Boxplo s o p ojec size, ans o med in o UCP.
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
5

was consumed om p ojec 's s a ing da e o accep ance da e. Median
Real_P20 p esen s same alue di ided by PF = 20. I supposes ha 20
pe son-hou s co esponds o 1 UCP. This ans o ma ion was done be-
cause o da a dona o s do no pe o m es ima ion using UCP.
Range Real_P20 desc ibes diffe ence be ween he smalles
(Minimum Real_P20) and la ges p ojec (Maximum Real_P20). As can
be seen a iance o Da ase 1 is highe han o Da ase 2. Las column
(n) s ands o numbe o p ojec s in each da ase .
4.2. Expe imen p ocedu e
Fo he pu poses o his s udy, he BPRM [11] model is used. BPRM
is buil by SLR - wi h cons ain s applied. The model con ains an in-
e cep , linea e ms, and squa ed e ms. Bo h da ase s - (DS1 and DS2),
we e di ided in a a io o 2:1, by using a hold-ou me hod, which hen
was used o c ea e aining and es ing se s. The es ing se s a e un-
de s ood as ime-o de ed p ojec s, which en e an es ima ion p ocess.
The aining se was s anda dised (23) o clus e ing and used as non-
s anda dised o ob ain models o each subse /clus e , o as he sou ce
o he window ange.
The expe imen al p ocedu e o clus e ing algo i hms is as ollows:
(1) C ea ing aining and es ing se s by using he hold-ou me hod
(2) S anda diza ion o he aining se , using he z-sco e me hod
(3) Selec ing ea u es o clus e ing (UAW, UUCW, TCF, ECF and
Real_P20)
(4) Applying a clus e ing app oach on he s anda dized aining se
(5) Only solu ions whe e each clus e con ains mo e han 5 p ojec s
a e selec ed
(6) Compu ing a BPRM o each o he clus e s
(7) P ojec s in he es ing se a e classified in o clus e s
(8) Es ima ion o AUCP is pe o med by using a clus e -specific model
(9) PE o p ojec s in he aining se a e compu ed
(10) SSE and o he e alua ion c i e ia a e compu ed
The expe imen p ocedu e o he windows-based algo i hm is as
ollows:
(1) C ea ing aining and es ing da ase s by using he hold-ou me hod
(2) The aining se is unde s ood as a his o ical p ojec , in ch on-
ological o de
(3) Compu ing a BPRM o p ojec s in window-sizes: 5, 10 and 15
(4) Es ima ing a AUCP alue o he fi s p ojec in he aining se
(5) Appending he p ojec om S ep 4 o he aining se
(6) Repea ing he p ocedu e upon he es ing se un il i is emp y
(7) PE o es ima ed p ojec s in he aining se a e compu ed
(8) SSE and o he e alua ion c i e ia a e compu ed
SLR - o gene ally, a MLR model; is sensi i e o a minimum numbe
o p ojec s accep able o aining a model. When MLR is used in con-
nec ion wi h clus e ing, a minimum numbe o p ojec s mus be se
when a solu ion is selec ed. In his pape , only solu ions whe e each
clus e con ains 5 o mo e da a-poin s a e used. This condi ion is used
o ob ain an app op ia e le el o machine p ecision when MLR models
a e calcula ed.
All algo i hms es ed in his expe imen ha e no na u al way as o
how o apply his cons ain , he e o e he aining se was clus e ed
i e a i ely, wi h a p edefined maximum alue k,(21):
=⎛
⎝
⎞
⎠
k ound n
2. . 5(21)
Whe e, kis he numbe o clus e s, and nis he numbe o p ojec s in
he aining se . Clus e ing algo i hms - (k-means, GMM clus e ing and
Spec al Clus e ing), we e used in combina ion wi h cosine simila i y
measu emen (22). Cosine simila i y was selec ed acco ding o he e-
sul s o he simula ion s udy - [16], whe e se e al dis ance unc ions
we e es ed.
=−
d
xx1cos(,)
ij
(22)
Whe e, x
i
,x
j
a e ea u e se s ha desc ibe da a-poin s in he aining
se . The k-nea es neighbou g aph, was used o simila i y measu e-
men in Spec al Clus e ing; and k-means wi h cosine simila i y we e
used o Eigen ec o s Clus e ing –whe e a cons ain o 5 da a-poin s in
each clus e was applied.
UAW, UUCW, TCF, ECF and Real_P20 we e used as ea u e se s o
all o he clus e ing algo i hms. This ea u e se was selec ed because all
o hose a ibu es con ibu e o effo es ima ion. Using his se allows
o clus e da ase s no only acco ding a p ojec size, bu also ela ion-
ships among a ibu es can be included when he simila i y is measu ed.
T aining se s da a we e s anda dized. S anda diza ions we e o med
by using z-sco e. Z-sco es measu e he dis ance o a da a poin om he
mean in e ms o he s anda d de ia ion. This is also called s anda di-
za ion o da a. The s anda dized da a se has mean 0 and s anda d
de ia ion 1, and e ains he shape p ope ies o he o iginal da a se
(same skewness and ku osis). Applying s anda diza ion equal weigh s
a e gi en o all o a ibu es, which a e used in dis ance measu emen s.
=−
zxμ
σ
()
(23)
Whe e, xis a a iable, μs ands o mean and σ ep esen s a s anda d
de ia ion.
5. Resul s
5.1. K-means Clus e ing
Only a solu ion in which a maximum o 3 clus e s a e c ea ed, can
be accep ed wi h espec o all cons ain s. The algo i hm beha es he
same o DS1 and DS2. Using mo e clus e s is no possible, because o
he ini ial cons ain o 5 da a-poin s in each clus e .
Table 2 shows ha he 3-clus e solu ion allows one o achie e
be e p edic ion han a 2-clus e solu ion. No e: SSE o he DS2, 2
clus e s: 2700.96 poin s; and 3 clus e s: 289.36 only. P edic ion E o is
mo e han 9 imes lowe ; when MAPE is compa ed, hen 3 clus e s a e 3
imes be e han 2 clus e s.
5.2. Spec al Clus e ing
In Table 3, he Spec al Clus e ing esul s a e shown. Spec al
Clus e ing achie es 5 clus e s o DS2; and 3 clus e s o DS1. The bes
DS2 esul s we e achie ed by 3 clus e s. The SSE o 4 clus e s is
552.04; and 5 clus e s achie ed: 3739.35. A simila diffe ence is alid
o all o he c i e ia.
Table 1
Da ase s’cha ac e is ics.
Median Median Range SD Minimum Maximum n
pe son-hou s Real_P20 Real_P20 Real_P20 Real_P20 Real_P20
Da ase 1 1952.500 97.625 183.650 57.063 13.850 197.500 28
Da ase 2 6406.000 320.300 109.750 33.212 288.750 398.500 70
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
6
5.3. GMM Clus e ing
In Table 4, GMM shows he wo s clus e ing abili y since only wo
clus e s we e iden ified o DS2, and no clus e s o DS1. Fo DS1, his
means ha he e was only one clus e wi h a majo i y o da a-poin s
and he second con ains less han 5 da a-poin s.
5.4. Mo ing Window
Mo ing Window, (Table 5), p oduced esul s which show i s p e-
dic ion abili y - bu also, e o limi s in i s p ac ical abili y. Mo ing
Window achie ed a MAPE 27.26 o DS2, which means ha he a e age
p edic ion is abou 27% lowe - o highe , han eal known alue.
6. Discussion
In his s udy, ou subse selec ion echniques me hods we e es ed
on he wo da ase s. Winning app oaches ( he bes pe o ming
me hods) a e p esen ed in Table 6, (DS1); and Table 7, (DS2). The bes
p edic ion abili y was achie ed by using k-means o spec al clus e ing,
when 3 clus e s o bo h da ase s (DS1, DS2) we e used. As can be seen
om Table 3, Spec al Clus e ing is able o clus e a DS2 da ase in o a
maximum o 5 clus e s - i he 5 da a-poin cons ain is applied. The
o e all esul s show - (see Table 3) ha 3 clus e s ou pe o m he 5-
clus e solu ion. The winning applica ion is Spec al Clus e ing, because
i shown a be e abili y o clus e and i da ase is la ge , han mo e
clus e ing can be used i any minimal numbe o da a-poin s in each
clus e is eques ed.
Fo DS1, applying Spec al Clus e ing leads o dec eases in e o s by
40% as compa ed o UCP, and MAPE is 53%; o he me hods, (e.g.
GMM, MW), a e ou pe o med by SC o k-means. MW is ou pe o med
by UCP.
When DS2 is e alua ed; hen simila esul s and beha iou can be
obse ed. The esul s show ye again ha SC is he winning solu ion –
bu in absolu e alues; all me hods beha e be e han in DS1, which
could be caused by highe o clus e consis ency le els. This is d i en by
he sou ce da a - when DS2 con ains mo e simila p ojec s.
An in e es ing aspec can be seen in compa ing DS2 and DS1 esul s.
The diffe ence be ween how all me hods beha e is clea ly seen by using
he MAPE c i e ion o by NRMSE - which a e he only wo c i e ia no
biased ega ding he numbe o da a-poin s. As can be seen, he effec s
o clus e ing a e highe o a bigge da ase (DS2).
Answe ing a RQ1 i can be shown, ha UCP and BPRM (when
iden ical MLR model is applied) a e ou pe o med. Subselec ion ech-
nique o his o ical da ase s impac ed es ima ion accu acy. Applying
clus e ing wi h 3 (k-means, SC) and 2 (GMM) clus e s allows o inc ease
a pe o mance when compa ed o UCP o BPRM. MW, when window
size equals 15 ou pe o ms UCP bu esul s a e no significan o BPRM
(Table 8b). The esul s we e analysed using a “pai - es ”. This analysis
e ealed a significan diffe ence be ween k-means and SC –as agains
UCP and BPRM. The es esul s a :
=
α
0.0
5
, can be seen in Table 8a
and in Table 8b. The esul s a e s a is ically significan o DS2 only.
Fig. 4 illus a es he p edic ion e o s o each da a-poin se in DS1 and
DS2.
Table 2
K-means clus e ing esul s.
Numbe o clus e s SSE MSE RMSE NRMSE MAPE
DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2
2 96,237.00 2700.96 12,030.00 128.62 109.68 11.34 0.62 0.13 95.78 2.52
3 42,001.00 289.36 5250.10 13.77 72.46 3.71 0.41 0.04 53.05 0.97
Table 3
Spec al Clus e ing esul s.
Numbe o clus e s SSE MSE RMSE NRMSE MAPE
DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2
2 125,560.00 6451.49 15,695.00 307.21 125.28 17.53 0.71 0.20 69.50 2.30
3 42,001.00 289.36 5250.10 13.78 72.46 3.71 0.41 0.04 53.05 0.98
4 NaN 552.04 NaN 26.29 NaN 5.13 NaN 0.06 NaN 1.25
5 NaN 3739.35 NaN 178.06 NaN 13.34 NaN 0.15 NaN 2.39
Table 4
GMM Clus e ing esul s.
Numbe o clus e s SSE MSE RMSE NRMSE MAPE
DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2
2 51 773.00 2700.96 6471.70 128.62 80.45 11.34 0.45 0.13 96.58 2.52
Table 5
Mo ing Window esul s.
Window size SSE MSE RMSE NRMSE MAPE
DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2 DS1 DS2
5 135,760.00 1,537,772.33 1697.00 73,227.25 130.26 270.61 0.76 3.09 394.31 27.26
10 617,570.00 10,851,000.00 77,196.00 516,700.00 277.84 718.81 1.63 8.21 509.61 59.61
15 95,361.00 40,287.00 11,920.00 1918.50 109.18 43.80 0.64 0.50 333.98 9.35
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
7
The RQ2 asking abou i all subse selec ion echniques a e equal. As
can be seen k-means and SC a e only significan ly be e (see Tables 8a
and 8b). GMM o MW canno be unde s ood as be e when nume ical
no s a is ical significance is s udied.
The esiduals show a significance ha he MW app oach is capable
o p edic ing effo - when p ojec s a e na i ely simila . Whe eas,
clus e ing me hods a e less sensi i e o changes in p ojec ypes o size.
When RQ3 is discussed –i MW equals o clus e ing echniques.
When MW a e used oge he wi h MLR, hey a e no as capable as
spec al clus e ing o k-means.
I all p ojec s in a window a e simila - and newly-p edic ed, his is
significan ly diffe en ha he MLR model and can p o ide an accu a e
p edic ion. Clus e ing me hods excel because o he classifica ion o
p ojec s in ad ance and because o using MLR models - which we e
cons uc ed on simila p ojec s.
Small window-size espec s a p ac ical poin -o - iew because i is
usually no possible o use la ge windows in p ac ice; bu does ela e o
his o ical da ase size. The influenceo MWsizecanbeseeninFig. 5.La ge
windows p oduce be e p edic ion - as p e iously confi med in o he s u-
dies [23–25]. Window sizes la ge han 15 da a-poin s b ing significan
imp o emen s. Bu , SC is a mo e capable me hod. When SSE is compa ed,
hen SC achie es ( o DS2): 289.36 s 40,287.00; and MAPE is 0.98%
compa ed o 9.35%. In Fig. 5, a de elopmen o SSE o a selec ed window
size is shown. Fo be e o e iew, he SSE is shown on a log- ans o med y
axis; while MAPE is shown on a no mal y-axis (pe cen age).
6.1. Th ea s o alidi y
In his s udy, wo da ase s - which bo h a e publicly a ailable, a e
used. Bo h da ase s a e ela ed o he UCP es ima ion app oach, which
makes i a bias- o-es ima ion me hod. All conclusions a e alid o he
expe imen se up p esen ed he ein. No o he da ase s o UCP me hods
a e a ailable, he e o e only men ioned da ase s a e used.
A da ase is used in he hold-ou se up, which allows one o di ide i
in o aining and es ing se s. Hold-ou is only an op ion due o limi ed
size o DS1. This di ision is based on andom gene a ion; he e o e,
eplica ion is limi ed i he same andom gene a o configu a ion is no
used.
Sub-selec ion Techniques, (Clus e ing), a e sensi i e o ini ial seed
se up. Seed is used o ini ial cen oid selec ion o clus e ing me hods.
The ule o a minimum o 5 obse a ion (his o ical da a-poin s) in
each clus e is applied. This ule allows o a oid c ea ion o oo many
small clus e s, which an imp op ia e o MLR model cons uc ion.
In he MW app oach, he windows size akes on an impo an ole.
In his s udy, a window o size o 15 is used because he DS1 is small
and la ge windows canno be es ed.
7. Conclusion
The effec o subse selec ion echniques was s udied, and h ee
clus e ing echniques: (k-means, SC, GMM), as well as he MW me hod,
we e e alua ed agains he UCP me hod. All echniques we e used as
da a-p epa a ion me hods o he MLR models. Clus e ing allows
Table 6
Selec ing a winning app oach o DS1.
Me hod SSE MSE RMSE NRMSE MAPE
K-means - 3clus 42,001.00 5250.10 72.46 0.41 53.05
SC –3clus 42,001.00 5250.10 72.46 0.41 53.05
GMM –2clus 51,773.00 6471.70 80.45 0.45 96.58
MW –size 15 95,361.00 11,920.00 109.18 0.64 333.61
UCP 69,344.24 8667.60 93.10 0.54 165.52
BPRM [11] 29,499.90 3687.74 60.72 0.34 56.32
Table 7
Selec ing a winning app oach o DS2.
Me hod SSE MSE RMSE NRMSE MAPE
K-means - 3clus 289.36 13.78 3.71 0.04 0.98
SC –3clus 289.36 13.78 3.71 0.04 0.98
GMM –2clus 2700.96 128.62 13.34 0.13 2.52
MW 40,287.00 1918.50 43.80 0.50 9.35
UCP 203,140.00 9673.20 98.35 1.12 26.82
BPRM [11] 7865.10 374.52 19.35 0.18 4.44
Table 8a
S a is ical Significance o Tes ed Me hods s. UCP.
Me hod DS1 DS2
k-means - 3clus 0.227 <0.001
SC –3clus 0.227 <0.001
GMM –2 clus 0.310 <0.001
MW –size 15 0.661 <0.001
Table 8b
S a is ical Significance o Tes ed Me hods s. BPRM.
Me hod DS1 DS2
k-means - 3clus 0.709 <0.001
SC –3clus 0.709 <0.001
GMM –2 clus 0.981 0.052
MW –size 15 0.849 0.963
Fig. 4. Residuals o indi idual da a-poin s (DS1 –le , DS2 – igh ).
Fig. 5. Mo ing Window Size Impac on SSE and MAPE (on DS2).
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
8
selec ing mo e simila p ojec s - which leads o be e es ima ion, be-
cause o mo e capable is used o p edic ion. MW is simila in pe o -
mance, only i a la ge size is used, (15 da a-poin s). This means MW is
no accep able o smalle da a-se s. MW is sensi i e o significan
changes in p ojec -size, o cha ac e is ics. The MW solu ion is no a
obus solu ion, due o i s da a sensi i i y.
When RQ1 is discussed, i can be concluded ha Subse Selec ion
Techniques imp o e o e all p edic ion and MLR pe o mance is sig-
nifican ly imp o ed. GMM Clus e ing p o ides small imp o emen s -
when compa ed o UCP o BPRM, ( o DS1). MW - when window-size
equals 15, is be e han UCP; bu , smalle window sizes a e no as
capable.
In RQ2, we can decla e ha he bes pe o ming subse selec ion
echnique is SC, which we can assume is he mos influen ial me hod. In
his s udy, SC pe o mance is equal o k-means; bu , as can be seen in
Table 3, SC is able o clus e da a be e , (in o mo e clus e s), i he
da ase is la ge . This is suppo ed by he esul s, when DS2 is em-
ployed.
To answe RQ3; i can be concluding ha MW is usable –only o
la ge da ase , bu is s ill no as good as SC when windows size is la ge -
(15 da a-poin s). O he wise, MW caused ises in p edic ion e o s. MW
is sensi i e o changes in p ojec cha ac e is ics; i a newly p edic ed
p ojec is dissimila o hose in he his o ical da ase , hen MW can
in oduce a significan e o .
In conclusion clus e ing me hods educe he p edic ion e o o
linea eg ession me hods significan ly, whe e SC is a winning me hod
o subse selec ion echniques. SC can educe a p edic ion e o by up o
98%, when compa ed o UCP and by 30% when compa ed o BMRM
[11]. MW p oduces inconsis en esul s and i is a da a sensi i e
me hod.
In u u e esea ch, a hyb id me hod - which will combine a MW and
SC app oach, will be in es iga ed and weigh ed windows based on
windows unc ion will be e alua ed.
Supplemen a y ma e ials
Supplemen a y ma e ial associa ed wi h his a icle can be ound, in
he online e sion, a doi:10.1016/j.in so .2017.12.009.
Re e ences
[1] G. Ka ne , Me ics Fo Objec o y', Diploma, Uni e si y o Linkoping, Sweden,
Decembe 1993, p. 21 No. LiTH-IDA-Ex-9344.
[2] R Silha y, P Silha y, Z P okopo a, Algo i hmic op imisa ion me hod o imp o ing
use case poin s es ima ion, PloS ONE 10 (11) (2015) e0141887.
[3] M. Ochodek, B. Alchimowicz, J. Ju kiewicz, J. Naw ocki, Imp o ing he eliabili y
o ansac ion iden ifica ion in use cases, In . So w. Technol. 53 (8) (2011)
885–897, h p://dx.doi.o g/10.1016/J.In so .2011.02.004 PubMed PMID:
WOS:000292176300007.
[4] A.B. Nassi , D Ho, L.F. Cap e z, Towa ds an ea ly so wa e es ima ion using log-
linea eg ession and a mul ilaye pe cep on model, J. Sys . So w. 86 (1) (2013)
144–160.
[5] V. Anandhi, R.M. Chezian, Reg ession echniques in so wa e effo es ima ion
using cocomo da ase , In e na ional Con e ence on In elligen Compu ing
Applica ions (Icica 2014), 2014, pp. 353–357, , h p://dx.doi.o g/10.1109/Icica.
2014.79 PubMed PMID: WOS:000358253500072.
[6] M. Ochodek, J. Naw ocki, K. Kwa ciak, Simpli ying effo es ima ion based on use
case poin s, In . So w. Technol. 53 (3) (2011) 200–213, h p://dx.doi.o g/10.
1016/j.in so .2010.10.005.
[7] M. Jo gensen, Reg ession models o so wa e de elopmen effo es ima ion accu-
acy and bias, Empi . So w. Eng. 9 (4) (2004) 297–314, h p://dx.doi.o g/10.
1023/B:EMSE.0000039881.57613 cb. PubMed PMID: WOS:000224569200003.
[8] T. U banek, Z. P okopo a, R. Silha y, V. Vesela, P edic ion accu acy measu emen s
as a fi ness unc ion o so wa e effo es ima ion, Sp inge Plus 4 (2015) 17, h p://
dx.doi.o g/10.1186/s40064-015-1555-9 PubMed PMID: WOS:000368718000002.
[9] M. Jo gensen, M. Sheppe d, A sys ema ic e iew o so wa e de elopmen cos es-
ima ion s udies, IEEE T So w. Eng. 33 (1) (2007) 33–53, h p://dx.doi.o g/10.
1109/Tse.2007.256943 PubMed PMID: WOS:000242312200003.
[10] J.F. Wen, S.X. Li, Z.Y. Lin, Y. Hu, C.Q. Huang, Sys ema ic li e a u e e iew o
machine lea ning based so wa e de elopmen effo es ima ion models, In . So w.
Technol. 54 (1) (2012) 41–59, h p://dx.doi.o g/10.1016/j.in so .2011.09.002
PubMed PMID: WOS:000297871500003.
[11] R. Silha y, P. Silha y, Z. P okopo a, Analysis and selec ion o a eg ession model
o he use case poin s me hod using a s epwise app oach, J. Sys . So w. 125 (2017)
1–14 h p://dx.doi.o g/10.1016/j.jss.2016.11.029.
[12] A. Id i, F.A. Amazal, A. Ab an, Analogy-based so wa e de elopmen effo es i-
ma ion: a sys ema ic mapping and e iew, In . So w. Technol. 58 (2015) 206–230,
h p://dx.doi.o g/10.1016/j.in so .2014.07.013 PubMed PMID:
WOS:000347022800012.
[13] A. Nassi , M. Azzeh, L. Cap e z, D. Ho, Neu al ne wo k models o so wa e de-
elopmen effo es ima ion: a compa a i e s udy, Neu al Compu . Appl. (2015)
1–13, h p://dx.doi.o g/10.1007/s00521-015-2127-1.
[14] M. Azzeh, A.B. Nassi , A hyb id model o es ima ing so wa e p ojec effo om
use case poin s, Appl. So Compu . (2016).
[15] V.K. Ba dsi i, D.N.A. Jawawi, S.Z.M. Hashim, E. Kha ibi, Inc easing he accu acy o
so wa e de elopmen effo es ima ion using p ojec s clus e ing, Ie So w. 6 (6)
(2012) 461–473, h p://dx.doi.o g/10.1049/ie -sen.2011.0210 PubMed PMID:
WOS:000310517200001.
[16] Z. P okopo a, R. Silha y, P. Silha y, The effec s o clus e ing o so wa e size es-
ima ion o he use case poin s me hods, Ad . In ell. Sys . Compu . (2017)
479–490.
[17] V.K. Ba dsi i, D.N.A. Jawawi, S.Z.M. Hashim, E. Kha ibi, A flexible me hod o es-
ima e he so wa e de elopmen effo based on he classifica ion o p ojec s and
localiza ion o compa isons, Empi . So w. Eng. 19 (4) (2014) 857–884, h p://dx.
doi.o g/10.1007/s10664-013-9241-4 PubMed PMID: WOS:000336388500003.
[18] J. Kennedy, R. Ebe ha , Pa icle swa m op imiza ion, 1995 IEEE In e na ional
Con e ence on Neu al Ne wo ks P oceedings, 1–6 1995, pp. 1942–1948, , h p://dx.
doi.o g/10.1109/Icnn.1995.488968 PubMed PMID: WOS:A1995BF46H00374.
[19] J. Hihn, L. Jus e , J. Johnson, T. Menzies, G. Michael, Imp o ing and expanding
NASA so wa e cos es ima ion me hods, Ae ospace Con e ence, 2016 IEEE, IEEE,
2016.
[20] C. Lokan, E. Mendes, Applying mo ing windows o so wa e effo es ima ion, In .
Symp. Emp. So w. (2009) 111–122 PubMed PMID: WOS:000274866100011.
[21] S. Amasaki, C. Lokan, The effec s o mo ing windows o so wa e es ima ion:
compa a i e s udy on linea eg ession and es ima ion by analogy, P oceedings o
he 2012 Join Con e ence o he 22nd In e na ional Wo kshop on So wa e
Measu emen and he 2012 Se en h In e na ional Con e ence on So wa e P ocess
and P oduc Measu emen (Iwsm-Mensu a 2012), 2012, pp. 23–32, , h p://dx.doi.
o g/10.1109/Iwsm-Mensu a.2012.13 PubMed PMID: WOS:000317102600006.
[22] A. Saxena, M. P asad, A. Gup a, N. Bha ill, O.P. Pa el, A. Tiwa i, e al., A e iew o
clus e ing echniques and de elopmen s, Neu ocompu ing 267 (Supplemen C)
(2017) 664–681 h ps://doi.o g/10.1016/j.neucom.2017.06.053.
[23] C. Lokan, E. Mendes, In es iga ing he use o mo ing windows o imp o e so wa e
effo p edic ion: a eplica ed s udy, Empi . So w. Eng. 22 (2) (2017) 716–767,
h p://dx.doi.o g/10.1007/s10664-016-9446-4 PubMed PMID:
WOS:000399891400004.
[24] S. Amasaki, C. Lokan, E alua ion o mo ing window policies wi h CART, In .
Wo ksh. Empi . Eng. (2016) 24–29, h p://dx.doi.o g/10.1109/Iwesep.2016.10
PubMed PMID: WOS:000381744800005.
[25] S. Amasaki, C. Lokan, On applicabili y o fixed-size mo ing windows o ANN-based
effo es ima ion, P oceedings o 2016 Join Con e ence o he In e na ional
Wo kshop on So wa e Measu emen and he In e na ional Con e ence on So wa e
P ocess and P oduc Measu emen (Iwsm-Mensu a), 2016, pp. 213–218, , h p://dx.
doi.o g/10.1109/IWSM-Mensu a.2016.31 PubMed PMID:
WOS:000399139200029.
[26] M. Azzeh, A. Nassi , S. Bani aan, F. Almasalha, Pa e o efficien mul i-objec i e
op imiza ion o local uning o analogy-based es ima ion, Neu al Compu . Appl.
(2015) 1–25, h p://dx.doi.o g/10.1007/s00521-015-2004-y.
[27] R. Silha y, P. Silha y, Z. P okopo a, Applied leas squa e eg ession in use case
es ima ion p ecision uning, So wa e Enginee ing in In elligen Sys ems, Sp inge
In e na ional Publishing, 2015, pp. 11–17.
[28] A. de My enae e, B. Golden, B. Le G and, F. Rossi, Mean absolu e pe cen age e o
o eg ession models, Neu ocompu ing 192 (Supplemen C) (2016) 38–48 h ps://
doi.o g/10.1016/j.neucom.2015.12.114.
[29] B.A. Ki chenham, L.M. Picka d, S.G. MacDonell, M.J. Sheppe d, Wha accu acy
s a is ics eally measu e [so wa e es ima ion], So w. IEE P oc. 148 (3) (2001)
81–85, h p://dx.doi.o g/10.1049/ip-sen:20010506.
[30] M. Sheppe d, M. Ca w igh , G. Kadoda, On building p edic ion sys ems o so -
wa e enginee s, Empi . So w. Eng. 5 (3) (2000) 175–182, h p://dx.doi.o g/10.
1023/a:1026582314146.
[31] G. James, D. Wi en, T. Has ie, R. Tibshi ani, An In oduc ion o S a is ical
Lea ning, Sp inge , 2013.
[32] D. Reynolds, Gaussian Mix u e Models, in: SZ Li, A Jain (Eds.), Encyclopedia o
Biome ics. Bos on, MA: Sp inge US, 2009, pp. 659–663.
[33] U. on Luxbu g, A u o ial on spec al clus e ing, S a . Compu . 17 (4) (2007)
395–416, h p://dx.doi.o g/10.1007/s11222-007-9033-z.
[34] M. Sol anolko abi, E. Elhami a , E.J. Candes, Robus subspace clus e ing, Ann. S a .
42 (2) (2014) 669–699, h p://dx.doi.o g/10.1214/13-Aos1199 PubMed PMID:
WOS:000336888400014.
[35] T. Li, S. Zhu, M. Ogiha a, Using disc iminan analysis o mul i-class classifica ion:
an expe imen al in es iga ion, Knowl. In . Sys . 10 (4) (2006) 453–472, h p://dx.
doi.o g/10.1007/s10115-006-0013-y.
[36] B. Ki chenham, S.L. Pfleege , B. McColl, S. Eagan, An empi ical s udy o main-
enance and de elopmen es ima ion accu acy ( ol 64, pg 57, 2002), J. Sys . So w.
74 (2) (2005) 227, h p://dx.doi.o g/10.1016/j.jss.2004.07.010 PubMed PMID:
WOS:000224874700010.
[37] A Sub iadi, P Ning um, C i ical e iew o he effo a e alue in use case poin
me hod o es ima ing so wa e de elopmen effo , J. Theo e ical Appl. In .
Technol. 59 (3) (2014) 735–744.
R. Silha y e al. In o ma ion and So wa e Technology 97 (2018) 1–9
9