

Ci a ion: Hai, V.V.; Nhung, H.L.T.K.;
P okopo a, Z.; Silha y, R.; Silha y, P.
A New App oach o Calib a ing
Func ional Complexi y Weigh in
So wa e De elopmen E o
Es ima ion. Compu e s 2022,11, 15.
h ps://doi.o g/10.3390/
compu e s11020015
Academic Edi o : Robe as
Damaše iˇcius
Recei ed: 15 Decembe 2021
Accep ed: 19 Janua y 2022
Published: 22 Janua y 2022
Publishe ’s No e: MDPI s ays neu al
wi h ega d o ju isdic ional claims in
published maps and ins i u ional a il-
ia ions.
Copy igh : © 2022 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license (h ps://
c ea i ecommons.o g/licenses/by/
4.0/).
compu e s
A icle
A New App oach o Calib a ing Func ional Complexi y Weigh
in So wa e De elopmen E o Es ima ion
Vo Van Hai * , Ho Le Thi Kim Nhung , Zdenka P okopo a, Radek Silha y and Pe Silha y
Depa men o Compu e and Communica ion Sys ems, Tomas Ba a Uni e si y in Zlin, Nam. T.G.M. 5555,
76001 Zlin, Czech Republic; [email p o ec ed] (H.L.T.K.N.); p [email p o ec ed] (Z.P.); [email p o ec ed] (R.S.);
[email p o ec ed] (P.S.)
*Co espondence: [email p o ec ed]
Abs ac :
Func ion poin analysis is a widely used me ic in he so wa e indus y o de elopmen
e o es ima ion. I was p oposed in he 1970s, and hen s anda dized by he In e na ional Func ion
Poin Use s G oup, as accep ed by many o ganiza ions wo ldwide. While he so wa e indus y has
g own apidly, he weigh alues speci ied o he s anda d unc ion poin coun ing ha e emained
he same since i s incep ion. Ano he p oblem is ha so wa e de elopmen in di e en indus y
sec o s is peculia , bu basic ules apply o all. These aise impo an ques ions abou he alidi y o
weigh alues in p ac ical applica ions. In his s udy, we p opose an algo i hm o calib a ing he
s anda dized unc ional complexi y weigh s, aiming o es ima e a mo e accu a e so wa e size ha i s
speci ic so wa e applica ions, e lec s so wa e indus y ends, and imp o es he e o es ima ion o
so wa e p ojec s. The esul s show ha he p oposed algo i hms imp o e e o es ima ion accu acy
agains he baseline me hod.
Keywo ds:
so wa e de elopmen e o es ima ion; unc ion poin analysis; unc ional
complexi y weigh
1. In oduc ion
So wa e es ima ion has long been conside ed a co e issue ha di ec ly a ec s success
o ailu e. Acco ding o he S andish G oup [
1
], he ailu e a e o a pa o a p ojec o
o a whole p ojec is likely o be up o 83.9% (as o 2019). One o he easons o his
ailu e is inaccu a e cos and e o es ima es. In ac , o ob ain so wa e p ojec s, companies
pa icipa ing in ende s mus submi bids ha include cos , manpowe , and so wa e
de elopmen ime. To be able o win he ende , he companies pa icipa ing need o gi e
a easonable es ima e o he cos , manpowe , and ime equi ed o ca y ou he p ojec .
Reasonabili y he e does no mean unde es ima ing he p ice, because in so doing he
company will no gain (i no lose) when comple ing he p ojec . I is also no easonable
o o e es ima e he p ice, because hen i is ce ain ha he company will no win he
bid. The e o e, a p ojec es ima e is conside ed easonable only i i accu a ely e lec s he
p ojec ’s ac ual alue.
Th oughou he so wa e de elopmen p ocess, no ma e wha so wa e manage-
men model a company uses, p ojec leade s o en ha e o plan he wo k o so wa e
de elopmen miles ones, plan he nex miles one, and ecalcula e he wo k done in he
p e ious miles one. All o hese asks equi e so wa e es ima ion skills. Many me hods
ha e p e iously been p oposed o sol e he so wa e es ima ion p oblem. Due o he
inc easing demand o mo e e icien and accu a e es ima ion me hods ha can wo k wi h
mo e complex so wa e p ojec s, such es ima ion me hods need o be e ined. So wa e
es ima ion me hods can be classi ied in o h ee main g oups: non-algo i hmic, algo i hmic,
and machine lea ning app oaches [
2
,
3
]. In he non-algo i hmic ca ego y, he e a e wo
ep esen a i e me hods: expe judgemen (EJ) [
4
] and analogy [
5
]; wi h hese me hods,
expe s play he mos signi ican ole in judgemen . O cou se, p e ious samples (his o ical
da ase ) also play ano he impo an ole. In he algo i hmic ca ego y, an algo i hm akes
Compu e s 2022,11, 15. h ps://doi.o g/10.3390/compu e s11020015 h ps://www.mdpi.com/jou nal/compu e s
Compu e s 2022,11, 15 2 o 20
he i s ole. so wa e li ecycle managemen (SLIM) [
6
], he Cons uc i e Cos Model (CO-
COMO) [
7
], and unc ion poin analysis (FPA) and he In e na ional Func ion Poin Use
G oup (IFPUG FPA) [
8
,
9
] a e ep esen a i e models. The IFPUG FPA me hod a ose as an
al e na i e o o he solu ions. O he me hods based on IFPUG FPA —such as COSMIC [
10
],
FiSMA [
11
], Ma kII [
12
], and NESMA [
13
]—we e p oposed o imp o e some aspec s o he
o iginal me hod. In he las ca ego y, machine lea ning echniques ha e been used in ecen
yea s o supplemen o eplace he o he wo echniques. Mos so wa e cos es ima ion
echniques use s a is ical me hods, which canno p o ide s ong a ionales and esul s.
Machine lea ning app oaches may be app op ia e in his a ea because hey can inc ease
he accu acy o es ima es by aining he es ima ion ules and epea ing he unning cy-
cles. Examples include uzzy logic modelling, eg ession ees, a i icial neu al ne wo ks
(ANNs), and case-based easoning (CBR). These me hods ha e explo ed he applicabili y o
machine lea ning echniques o so wa e e o es ima ion; hey ha e objec i e and analy i-
cal o mulas ha a e no limi ed o a speci ic numbe o ac o s. Howe e , mos me hods
only e alua e he limi a ions o modelling echniques on a pa icula da ase , educing he
gene alisabili y o he obse ed esul s. Some esea che s also o e look s a is ical es ing o
he esul s ob ained o e alua ion o he models agains he same da ase used o gene a e
he models [14].
This pape is o ganised as ollows: Sec ion 1is he in oduc ion. The p oblem o mula-
ion and he con ibu ions a e desc ibed in Sec ion 2. Sec ion 3illus a es he ela ed wo ks.
Sec ion 4p esen s he backg ound, wi h a b ie o e iew o he FPA me hod, Bayesian
idge eg ession model, and o ing eg esso model. The p oposed esea ch me hodology
is exp essed in Sec ion 5, in which we in oduce he da ase and da a p ocessing, expe -
imen al se up, and e alua ion c i e ia. Sec ion 6p esen s he expe imen al esul s and
discussion. Sec ion 7desc ibes he h ea s o alidi y. We inalise ou pape in Sec ion 8
wi h he conclusion.
2. P oblem Fo mula ion
FPA has been used o o e ou decades, and has p o en o be a dependable and con-
sis en me hod [
15
] o sizing so wa e o p ojec es ima ion and p oduc i i y o e iciency
compa isons. Al hough i has made signi ican con ibu ions in he so wa e indus y, i
s ill has many p oblems. In an ea lie sys ema ic li e a u e e iew [
16
], we men ioned
some limi a ions o FPA. The inadequacy o complexi y weigh is s ill a majo p oblem. In
addi ion, he locali y o he da ase ha builds he FPA app oach (IBM p ojec s) does no
e lec he en i e global so wa e indus y. Many p e ious s udies ha e sugges ed a new
unc ional complexi y weigh [
17
,
18
] in di e en ways. In [
18
], Xia e al. p oposed a new
unc ional complexi y able based on an IFPUG FPA calib a ion model called Neu o-Fuzzy
Func ion Poin Calib a ion Model (NFFPCM), which in eg a es he powe o neu al ne -
wo ks and uzzy logic. Ne e heless, he me hod needs o be changed in line wi h changes
in he mode n so wa e indus y.
Ano he issue ha needs o be men ioned is he speci ici y o each piece o so wa e.
Di e ences in he pu pose o he so wa e being de eloped lead o di e en app oaches.
Wi h FPA, a me hod ha applies o all so wa e es ima ion cases needs o be e isi ed; his
was he mo i a ion o us o p opose a new, up- o-da e, nonlocal unc ional complexi y
weigh ha e lec s he so wa e indus y.
A e coun ing he unc ion poin s (FPs), we can calcula e he e o and hen w i e
a epo [
9
] based on hese FPs. A his poin , he FPA coun ing p ocess is conside ed
comple e. Howe e , one ques ion is whe he we can u he imp o e he e o a e
he calcula ion o he coun ing p ocess. Recen s udies [
19
–
23
] show ha combining an
ensemble model and o he app oaches p o ides be e esul s han using a single model.
This s udy applies an ensemble model o he esul a e he coun ing and calcula ing
e o o imp o e he accu acy gained om FP coun ing based on he p oposed unc ional
complexi y weigh .
Compu e s 2022,11, 15 3 o 20
This s udy aims o p opose an algo i hm called he calib a ion o unc ional complexi y
weigh (CFCW) algo i hm. The p oposed algo i hm is based on he FPA me hod combined
wi h eg ession me hods implemen ed in he In e na ional So wa e Benchma king S an-
da ds G oup (ISBSG) da ase Release 2020 R1 [24].
Many companies a ound he wo ld con ibu e o he ISBSG da ase , so he locali y
p oblem can be sol ed. In addi ion, he 2020 da abase upda e add esses he ou -o -da e is-
sue. Many ecen s udies ( o example, [
25
–
27
]) used he indus y sec o (IS) as a ca ego ical
a iable o da ase segmen ing in hei esea ch. Ou s udy es ed a second app oach— he
calib a ion o unc ional complexi y weigh wi h op imisa ion based on an ensemble model
called he o ing eg esso (CFCWO).
Based on he abo e issues, we p opose h ee esea ch ques ions:
RQ1:Is he accu acy o he p oposed CFCW algo i hm be e han ha o he IFPUG FPA o
NFFPCM me hods?
RQ2:
Does he ad anced CFCWO algo i hm ou pe o m he CFCW algo i hm?
RQ3:
How accu a e is he es ima ion o each sec o compa ed o an ung ouped da ase ?
To answe hese esea ch ques ions, we conduc ed an expe imen al s udy o e alua e
he es ima ion accu acy o he p oposed app oaches.
Con ibu ions
The main con ibu ions o his esea ch a e as ollows:
•
In he i s phase, a new CFCW algo i hm o he calib a ion o unc ional complexi y
weigh is p oposed;
•
In he second phase, he esul om he i s phase is op imised by using a o ing
eg esso o es ima e he inal so wa e e o — he CFCWO algo i hm is p oposed;
•
The IFPUG FPA me hod is compa ed o he CFCW algo i hm o ung ouped da a and
da a g ouped by IS;
•
The CFCW algo i hm is compa ed o he CFCWO algo i hm o ung ouped da a and
da a g ouped by IS.
3. Rela ed Wo k
FPA is a s anda dised me hod o de e mining he size o so wa e based on i s
unc ional equi emen s; i is designed o be applicable ega dless o p og amming language
o implemen a ion echnology. Alb ech [
8
] ecommends FPA o measu e he size o a
sys em ha p ocesses da a om end-use s. Since i s in oduc ion, much esea ch has been
ca ied ou o imp o e i s accu acy.
Al-Haj i e al. [
17
] in oduced a modi ica ion weigh ing sys em o measu ing FP
using an ANN model (backp opaga ion echnique). In hei s udy, a weigh ing sys em
was buil based on ou s eps: (1) using he o iginal weigh ing sys em as a baseline o
es ablish new weigh s; (2) using he DETs/RETs o he o iginal sys em’s FPA o calcula e
he new alues, aining hese new alues wi h an ANN, and hen p edic ing he alues
o he new weigh s; (3) applying he new weigh s and he o iginal weigh s in he FPA
model; and (4) calcula ing he size o FPs as a unc ion o he o iginal weigh s and he
new weigh s. Wei e al. [
28
] p oposed a di e en sizing app oach by in eg a ing he new
calib a ed FP weigh p oposed in [
18
] in o a complexi y assessmen sys em o objec -
o ien ed de elopmen e o es ima ion. Mis a e al. [
29
] p oposed a me ics sui e ha
helps in de e mining he complexi y o objec -o ien ed p ojec s by e alua ion o message
complexi y, a ibu e complexi y, weigh ed class complexi y, and code complexi y.
Dewi e al. [
30
] p oduced a o mula o es ima ing he cos o so wa e de elopmen
p ojec s, especially in he ield o public se ice applica ions; he au ho s modi ied he
complexi y adjus men ac o o 16 ins ead o he 14 used in he s anda d FPA me hod and;
as a esul , he accu acy was imp o ed by 7.19%.
In he s udy o Leal e al. [
31
], he au ho s in es iga ed he use o nea es -neighbou s
linea eg ession me hods o es ima ion in so wa e enginee ing. These me hods we e
compa ed wi h mul ilaye pe cep on neu al ne wo ks, adial basis unc ion neu al ne -
Compu e s 2022,11, 15 4 o 20
wo ks, suppo - ec o eg ession, and bagging p edic o s. The da ase used in he s udy
was a NASA so wa e p ojec . Based on he ela i e e o and he es ima ion a e, he
nea es -neighbou s linea eg ession me hods ou pe o med he o he s.
In a su ey o applying ANNs o so wa e es ima ion, Hamza e al. [
32
] p o ided an
o e iew o he use o ANN me hods o es ima e de elopmen e o o so wa e de el-
opmen p ojec s; he au ho s o e ed ou main ANN models, including (1) eed o wa d
neu al ne wo ks; (2) ecu en neu al ne wo ks; (3) adial basis unc ion ne wo ks; and
(4) neu o- uzzy ne wo ks. The su ey also explains why hose me hods a e used and how
accu a e hey a e.
In he endea ou o es ima ing he e o needed o he nex phase o he emaining
e o needed o inish a p ojec , Lena duzzi e al. [
33
] conduc ed an empi ical s udy on he
es ima ion o so wa e de elopmen e o . The es ima ion was b oken down by phase so
ha es ima ion could be used h oughou he so wa e de elopmen li ecycle. This means
ha he e o needed o he nex phase a any gi en poin in he so wa e de elopmen
li ecycle is es ima ed. Addi ionally, hey es ima ed he e o equi ed o he emaining
pa o a so wa e de elopmen p ocess. The ISBSG da ase was used in he s udy. The
esul s show s a is ically signi ican co ela ions be ween e o expended in one pe iod
and e o expended in he ollowing pe iod, e o expended in one pe iod and o al e o
expended, and accumula ed e o up o he p esen s age and emaining e o . The esul s
also indica e ha hese es ima ion models ha e di e en deg ees o goodness o i . Fu he
a ibu es, such as he unc ional size, do no signi ican ly imp o e es ima ion quali y.
In [
25
,
34
], he au ho s p esen ed an in luence analysis o selec ed ac o s (FP coun
app oach, business a ea, IS, and ela i e size) on he es ima ion o he wo k e o o which
he FPA me hod is p ima ily used. They also s udied he ac o s ha in luence p oduc i i y
and he p oduc i i y es ima ion capabili y in he FPA me hod. Based on hese selec ed
ac o s and expe imen ally, he au ho s p o ed ha he selec ed ac o s ha e speci ic e ec s
on wo k e o es ima ion accu acy.
In [
35
], om so wa e ea u es, he au ho s used a ious machine lea ning algo i hms
o build a so wa e e o es ima ion model. ANNs, suppo - ec o machines, K-s a ,
and linea eg ession machine lea ning algo i hms we e app aised on a PROMISE da ase
(called Usp05- ) wi h ac ual so wa e e o s. The esul s e ealed ha he machine lea ning
app oach could be applied o p edic so wa e e o . In he s udy, he esul s om he
suppo - ec o machines we e he bes .
In [
36
], he au ho s conduc ed a compa ison be ween so compu ing and s a is ical
eg ession echniques in e ms o a so wa e de elopmen es ima ion eg ession p oblem.
Suppo - ec o eg ession and ANNs we e used as so compu ing me hodologies, and
s epwise mul iple linea eg ession and log-linea eg ession we e used as s a is ical eg es-
sion me hods. Expe imen s we e pe o med using he NASA93 da ase om he PROMISE
so wa e eposi o y, wi h mul iple da ase p e-p ocessing s eps pe o med. The au ho s
elied on he holdou echnique associa ed wi h 25 andom epe i ions wi h con idence
in e al calcula ion wi hin a 95% s a is ical con idence le el. The 30 p e-e alua ion c i e ia
we e used o compa e he esul s. The esul s o he s udy show ha he suppo - ec o
eg ession model has a signi ican impac on p ecision.
4. Backg ound
4.1. IFPUG FPA
Alb ech [
8
] i s in oduced FPA in 1979, and p esen ed he FP me ic o measu e he
unc ionali y o a p ojec . This was p oposed in esponse o a numbe o p oblems wi h
o he sys em size measu es, such as lines o code. In 1986, he In e na ional Func ion Poin
Use G oup (IFPUG) [
37
] p omo ed and popula ised e ec i e so wa e de elopmen and
main enance managemen h ough FPA.
The IFPUG is cu en ly he go e ning body o FPA, and is esponsible o imp o ing
and de eloping coun ing ules and o he ela ed ma e s. Since he IFPUG was c ea ed,
he o iginal FPA me hod has been known as he IFPUG’s FPA. In his s udy, he s anda d
Compu e s 2022,11, 15 5 o 20
FPA me hod concep e e s o he IFPUG FPA me hod. FPA is cu en ly s anda dised by
ISO/IEC 20926:2010 [
38
]; his s anda d speci ies a se o de ini ions, ules, and s eps o
applica ion [
9
]. The e a e six phases in coun ing s anda ds; in his s udy, we a e only
in e es ed in wo phases: (1) da a unc ion and ansac ional unc ion measu emen , and (2)
unc ional size measu emen .
The i s -phase esul s a e he unadjus ed unc ion poin s (UFPs) and he alue adjus -
men ac o (VAF) alues. The UFP alue can be de e mined based on es ima ions o he
numbe o ansac ional unc ions (ex e nal inpu (EI), ex e nal ou pu (EO), o ex e nal
inqui y (EQ)) and da a unc ions (in e nal logic iles (ILFs), and ex e nal in e ace iles
(EIFs)). These componen s a e called base unc ional componen s. Each o hese, in u n,
is judged as low (L), a e age (A), o high (H), and assigned a weigh acco dingly. Table 1
shows he a ailable complexi y weigh o he componen s.
Table 1. Da a and ansac ional unc ion complexi y.
Componen
EI EO EQ EIF ILF
Complexi y
weigh
Low 3 4 3 5 7
A e age 4 5 4 7 10
High 6 7 6 10 15
The UFP o al se s he numbe o ypes in g oups, mul iplies hem by complexi y
weigh s, and inds he sum o all ields, as in Equa ion (1):
UFP =
n
∑
i=1
m
∑
j=1
(Sij ×Wij)(1)
whe e
Sij
ep esen s he o al o each unc ional componen ,
Wij
ep esen s he complexi y
weigh s, nis he numbe o ypes, and mis he numbe o complexi y g oups.
The VAF coun is based on he a e o 14 gene al sys em cha ac e is ics (GSCs): da a
communica ions, dis ibu ed da a p ocessing, pe o mance, hea ily used con igu a ion,
ansac ion a e, online da a en y, end use e iciency, online upda e, complex p ocessing,
eusabili y, ins alla ion ease, ope a ional ease, mul iple si es, and acili a e change.
The e a e six in luence le els o GSC c i e ia, wi h he sys em being de e mined as
a alue om 0 o 5 con ingen on he le el: 0—no in luence; 1—inciden al in luence; 2—
mode a e in luence; 3—a e age in luence; 4—signi ican in luence; and 5—s ong in luence
h oughou . The VAF coun is adjus ed as ollows:
VAF =0.65 +0.01 ×
14
∑
i=1
(Fi× a ing)(2)
The second-phase esul is he adjus ed unc ion poin s (AFPs) alue, which can be
ob ained using Equa ion (3):
AFP =UFP ×VAF (3)
To es ima e he e o a e AFP coun ing, we should use ano he pa ame e . The
p oduc i i y ac o (PF) was desc ibed as he ela ionship be ween one FP and he numbe
o hou s needed o i s de elopmen by one pe son. P oduc i i y and PF we e s udied
in [39,40]. The ollowing o mula can be used o calcula e he e o using he PF:
E o =AFP ×PF (4)
ISBSG uses he p oduc i i y deli e y a e (PDR) as a me ic o e iciency. The PDR
is measu ed in pe son-hou s pe FP. F om he PDR, we can de i e he PF in FPs pe
pe son-hou . We can see ha he PDR is he in e ed alue o he PF (and ice e sa) [9].

Compu e s 2022,11, 15 6 o 20
In ou s udy, he IFPUG FPA me hod is he base me hod o p oposing he new model;
i is also used o he base compa ed wi h he p oposed model. Addi ionally, he e ms FPA
and IFPUG FPA ha e he same meaning, and a e in e changeable.
4.2. Bayesian Ridge Reg ession Model
The ull Bayesian eg ession in e ence uses he Ma ko chain Mon e Ca lo algo i hm
o cons uc models [
41
]. The Bayesian modelling amewo k has been epu ed o i s abili y
o deal wi h a hie a chical da a s uc u e. In Bayesian eg ession echniques, egula isa ion
pa ame e s can be included in he es ima ion p ocedu e. A egula isa ion pa ame e is
no ha d se , bu is uned o he da a a hand. This can be done by in oducing non-
in o ma i e p io s o e he hype pa ame e s o he model. The
l2
egula isa ion used in
idge eg ession and classi ica ion is equi alen o inding a maximum a pos e io i es ima e
unde a Gaussian p io o e he coe icien s wi h p ecision. Ins ead o se ing he lambda
manually, his a iable can be andomly es ima ed om he a ailable da a [
42
]. To acqui e
a ully p obabilis ic model, he ou pu yis assumed o be Gaussian dis ibu ed a ound
Xω
:
p(y|X,ω,α)=ℵ(y|Xω,α)(5)
whe e αis ea ed as a andom a iable ha is o be es ima ed om he da a.
Bayesian idge eg ession (BRR) is a p obabilis ic me hod ha builds a eg ession
model using Bayesian in e ence; i combines p io in o ma ion abou pa ame e s ( he
coe icien o so wa e ea u es) wi h he obse ed aining da a in o de o acqui e he
pa ame e s’ pos e io dis ibu ion [
42
]. The p io o he coe icien
ω
is speci ied by a
sphe ical Gaussian:
p(ω|λ)=ℵω

0, λ−1Ip(6)
The p io s o e
α
and
λ
a e picked o be gamma dis ibu ions [
43
]— he conjuga e
p io o he p ecision o he Gaussian dis ibu ion.
In ou s udy, he BRR plays a signi ican ole in he calib a ion phase (see Figu e 1).
Compu e s 2022, 11, 15 7 o 20
Figu e 1. Expe imen al p ocess.
4.3. Vo ing Reg esso Model
The ensemble is a lea ning me hod ha uses a speci ic agg ega ion mechanism o
c ea e a collec ion o p edic ion models, and hen uses a weigh ed o e o hei ini ial
esul s o ob ain he inal solu ion [44–48]. The p incipal p emise is ha i echniques wo k
oge he as a commi ee wi h eliable me hods, hey may be imp o ed and gene a e mo e
signi ican esul s [44]. As a esul , his me hod is excellen o p edic ing so wa e e o ,
since each model has i s assump ions and se up pa ame e s, allowing he ensemble o
pe o m excep ionally well wi h some desi able s a is ical quali ies [45]. Id i e al. [22,23]
conduc ed a sys ema ic li e a u e e iew and mapping s udy, and disco e ed ha (1)
ensemble e o es ima ion echniques a e mo e accu a e han solo me hods, (2)
homogeneous ensembles a e he mos in es iga ed, (3) machine lea ning echniques a e
he mos used solo echniques o cons uc ensembles, and (4) he e a e wo ypes o
combine ules used o es ima e ensemble e o es ima ion: linea and nonlinea .
A o ing eg esso [46] is based on he idea o in eg a ing a ious machine lea ning
app oaches o e u n uni o m a e age p ojec ed alues. A o ing eg esso is a echnique
ha i s each o he base eg esso s o he en i e da ase . A eg esso such as his can help
a g oup o es ima o s wi h simila pe o mance le els o balance ou hei indi idual
laws. When he p edic o s a e as independen as possible, ensemble app oaches pe o m
bes . In gene al, each eg esso is ained using a dis inc echnique, in o de o make each
p edic ion mo e independen o he o he s. This inc eases he likelihood ha hey will
make a a ie y o blunde s, which will imp o e he ensemble’s pe o mance. A o ing
eg esso can be applied o classi ica ion o eg ession. Each label’s p edic ions a e
combined wi h ega d o classi ica ion, and he label wi h he mos o es is chosen. In he
case o eg ession, his en ails compu ing he mean o he p edic ions om he models.
Acco ding o Wi en e al. [47], a o ing ensemble is app op ia e when all applicable
models should pe o m well on a p edic i e modelling ask; in o he wo ds, he models
used in he ensemble mus mos ly ag ee.
In ou s udy, he o ing eg esso was used in he CFCWO algo i hm in he
op imiza ion phase (Figu e 1).
Figu e 1. Expe imen al p ocess.
Compu e s 2022,11, 15 7 o 20
4.3. Vo ing Reg esso Model
The ensemble is a lea ning me hod ha uses a speci ic agg ega ion mechanism o
c ea e a collec ion o p edic ion models, and hen uses a weigh ed o e o hei ini ial esul s
o ob ain he inal solu ion [
44
–
48
]. The p incipal p emise is ha i echniques wo k oge he
as a commi ee wi h eliable me hods, hey may be imp o ed and gene a e mo e signi ican
esul s [
44
]. As a esul , his me hod is excellen o p edic ing so wa e e o , since
each model has i s assump ions and se up pa ame e s, allowing he ensemble o pe o m
excep ionally well wi h some desi able s a is ical quali ies [
45
]. Id i e al. [
22
,
23
] conduc ed
a sys ema ic li e a u e e iew and mapping s udy, and disco e ed ha (1) ensemble e o
es ima ion echniques a e mo e accu a e han solo me hods, (2) homogeneous ensembles a e
he mos in es iga ed, (3) machine lea ning echniques a e he mos used solo echniques
o cons uc ensembles, and (4) he e a e wo ypes o combine ules used o es ima e
ensemble e o es ima ion: linea and nonlinea .
A o ing eg esso [
46
] is based on he idea o in eg a ing a ious machine lea ning
app oaches o e u n uni o m a e age p ojec ed alues. A o ing eg esso is a echnique
ha i s each o he base eg esso s o he en i e da ase . A eg esso such as his can
help a g oup o es ima o s wi h simila pe o mance le els o balance ou hei indi idual
laws. When he p edic o s a e as independen as possible, ensemble app oaches pe o m
bes . In gene al, each eg esso is ained using a dis inc echnique, in o de o make
each p edic ion mo e independen o he o he s. This inc eases he likelihood ha hey
will make a a ie y o blunde s, which will imp o e he ensemble’s pe o mance. A
o ing eg esso can be applied o classi ica ion o eg ession. Each label’s p edic ions
a e combined wi h ega d o classi ica ion, and he label wi h he mos o es is chosen. In
he case o eg ession, his en ails compu ing he mean o he p edic ions om he models.
Acco ding o Wi en e al. [
47
], a o ing ensemble is app op ia e when all applicable models
should pe o m well on a p edic i e modelling ask; in o he wo ds, he models used in he
ensemble mus mos ly ag ee.
In ou s udy, he o ing eg esso was used in he CFCWO algo i hm in he op imiza-
ion phase (Figu e 1).
5. Resea ch Me hodology
In his sec ion, we p esen he esea ch me hodology. This includes desc ibing he
da a o be used, along wi h he da a p ocessing and he expe imen al se up. In addi ion,
e alua ion c i e ia a e in oduced he e.
5.1. Expe imen al Se up
In his sec ion, we desc ibe he expe imen al p ocess, which is g aphically illus a ed
in Figu e 1.
In he da a p e-p ocessing phase (Figu e 1), da a il e ing and cleaning we e pe o med
o c ea e he wo king da ase (see ollowing sec ion). This da ase was used o wo b anches
o expe imen s: expe imen s on ung ouped da a (all sec o s), and expe imen s on g ouped
da a, whe e IS ca ego ical a iables we e used o g ouping. The i e old c oss- alida ion
was used o c ea e a aining/ es ing old. The da ase we used in ou expe imen s was
he ISBSG eposi o y Augus 2020 R1 [
24
]. In ou s udy, he c i e ia o da a il e ing we e
as ollows:
1.
We selec ed eco ds wi h he IFPUG coun ing app oach (including IFPUG Old and
IFPUG 4+);
2. Only he eco ds whe e he da a quali y a ing is A o B has been selec ed;
3. The de elopmen ype was new de elopmen ;
4. The ows wi h an emp y alue o base unc ional componen s we e elimina ed;
5. Rows wi h emp y alues in he indus y sec o column we e emo ed;
6.
The ows wi h emp y alues in no malised p oduc i i y deli e y a e and summa y
wo k e o (SWE) we e also e ased;
7. We illed he VAF blank cells wi h he alues ob ained om Equa ion (3).
Compu e s 2022,11, 15 8 o 20
Acco ding o Lich enbe g and ¸Sim¸sek [
49
], he numbe o eco ds in a da ase is la ge
enough o be eligible o a gi en aining se o a ain he mos sa is ac o y esul s. M.
Hammad [
35
] also p o ed ha some algo i hms lea n pe ec ly as he size o he aining
se inc eases. In ou case, a e many es s and e alua ions, he esul s om ISs wi h o e
30 eco ds ga e he bes esul s. Fo he ISs ha did no sa is y his condi ion ( he numbe
o eco ds is less han 30), we ga he ed hem in o a g oup named “O he s”. Figu e 2shows
a his og am o he da ase a e being p ocessed.
Compu e s 2022, 11, 15 8 o 20
5. Resea ch Me hodology
In his sec ion, we p esen he esea ch me hodology. This includes desc ibing he
da a o be used, along wi h he da a p ocessing and he expe imen al se up. In addi ion,
e alua ion c i e ia a e in oduced he e.
5.1. Expe imen al Se up
In his sec ion, we desc ibe he expe imen al p ocess, which is g aphically illus a ed
in Figu e 1.
In he da a p e-p ocessing phase (Figu e 1), da a il e ing and cleaning we e
pe o med o c ea e he wo king da ase (see ollowing sec ion). This da ase was used o
wo b anches o expe imen s: expe imen s on ung ouped da a (all sec o s), and
expe imen s on g ouped da a, whe e IS ca ego ical a iables we e used o g ouping. The
i e old c oss- alida ion was used o c ea e a aining/ es ing old. The da ase we used in
ou expe imen s was he ISBSG eposi o y Augus 2020 R1 [24]. In ou s udy, he c i e ia
o da a il e ing we e as ollows:
1. We selec ed eco ds wi h he IFPUG coun ing app oach (including IFPUG Old and
IFPUG 4+);
2. Only he eco ds whe e he da a quali y a ing is A o B has been selec ed;
3. The de elopmen ype was new de elopmen ;
4. The ows wi h an emp y alue o base unc ional componen s we e elimina ed;
5. Rows wi h emp y alues in he indus y sec o column we e emo ed;
6. The ows wi h emp y alues in no malised p oduc i i y deli e y a e and summa y
wo k e o (SWE) we e also e ased;
7. We illed he VAF blank cells wi h he alues ob ained om Equa ion (3).
Acco ding o Lich enbe g and Şimşek [49], he numbe o eco ds in a da ase is la ge
enough o be eligible o a gi en aining se o a ain he mos sa is ac o y esul s. M.
Hammad [35] also p o ed ha some algo i hms lea n pe ec ly as he size o he aining
se inc eases. In ou case, a e many es s and e alua ions, he esul s om ISs wi h o e
30 eco ds ga e he bes esul s. Fo he ISs ha did no sa is y his condi ion ( he numbe
o eco ds is less han 30), we ga he ed hem in o a g oup named “O he s”. Figu e 2 shows
a his og am o he da ase a e being p ocessed.
Figu e 2. His og am o he p ocessed da ase .
We no ice ha he e is a possibili y ha he da a may be noisy because some SWE
alues a e oo a om he mean g oup. In his s udy, we used he in e qua ile ange
(IQR) me hod [50,51] on hese ea u es o de e mine and emo e ou lie s, wi h he lowe
bounda y being 0.15 and he uppe bounda y being 0.85. Figu es 3 and 4 show he
desc ip ion o he da ase be o e and a e he emo al o ou lie s, espec i ely.
Figu e 2. His og am o he p ocessed da ase .
We no ice ha he e is a possibili y ha he da a may be noisy because some SWE
alues a e oo a om he mean g oup. In his s udy, we used he in e qua ile ange (IQR)
me hod [
50
,
51
] on hese ea u es o de e mine and emo e ou lie s, wi h he lowe bounda y
being 0.15 and he uppe bounda y being 0.85. Figu es 3and 4show he desc ip ion o he
da ase be o e and a e he emo al o ou lie s, espec i ely.
Compu e s 2022, 11, 15 9 o 20
Figu e 3. Boxplo o he da ase be o e emo ing ou lie s.
Figu e 4. Boxplo o he da ase a e emo ing ou lie s.
The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks
as ollows:
1. Bayesian idge eg ession (Sec ion 4.2) is employed;
Figu e 3. Boxplo o he da ase be o e emo ing ou lie s.
Compu e s 2022,11, 15 9 o 20
Compu e s 2022, 11, 15 9 o 20
Figu e 3. Boxplo o he da ase be o e emo ing ou lie s.
Figu e 4. Boxplo o he da ase a e emo ing ou lie s.
The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks
as ollows:
1. Bayesian idge eg ession (Sec ion 4.2) is employed;
Figu e 4. Boxplo o he da ase a e emo ing ou lie s.
The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks
as ollows:
1. Bayesian idge eg ession (Sec ion 4.2) is employed;
2.
CFCW elici s he complexi y weigh s o he EI, EO, EQ, EIF, and ELF a iables using
Bayesian idge eg ession;
3.
The UFP is calcula ed by using a newly es ima ed complexi y weigh o each o he
a iables (EI, EO, EQ, EIF, and ELF);
4.
Es ima ed e o is ob ained by mul iplying UFP by he VAF and, inally, by mul iply-
ing by PF.
The op imisa ion phase (Figu e 1) ep esen s he CFCWO algo i hm, which wo ks
as ollows:
1. E o om CFCW (calib a ion phase) is used as inpu ;
2. The o ing eg esso (Sec ion 4.3) algo i hm is employed;
3. Vo ing eg esso is an ensemble model, consis ing o ou es ima o s (Table 2);
4.
CFCWO op imises es ima ed e o by CFCW by minimising he e o o SWE (know
e o om da ase ).
Table 2. Base es ima o s o he o ing ensemble model.
Algo i hm Implemen a ion Pa ame e s
Random o es s
sklea n.ensemble.RandomFo es Reg esso
n_es ima o s = 200,
andom_s a e = 0
Bayesian idge sklea n.linea _model. BayesianRidge n_i e = 300
ANN sklea n.neu al_ne wo k. MLPReg esso ol = 0.00001, max_i e =10000,
momen um = 0.000001
Lasso sklea n.linea _model. Lasso
alpha = 0.01,
selec ion = ‘ andom’,
andom_s a e = 63
Compu e s 2022,11, 15 16 o 20
(se ice indus y); he mean pe cen age di e ences o he indi idual sec o s compa ed o
he ung ouped da ase a e MAE = 49.98%, MAPE = 49.03%, and RMSE = 38.25%.
Pai ed-samples - es s we e used o e alua ing s a is ical signi icance compa isons
[58,59]
o see whe he he CFCWO me hod is signi ican ly di e en om he o he me hods, in
o de o con i m he e alua ion conclusions (see Table 10). The no a ions

,

, and
≈
e-
lec he s a is ical supe io i y, in e io i y, and simila i y o he CFCWO app oach compa ed
o each o he o he me hods (FPA and NFFPCM), espec i ely. We can conclude ha he
di e ence in es ima ing accu acy be ween he CFCWO and each al e na i e app oach is
signi ican when he p- alue is less han 0.05.
Table 10. The s a is ical - es s based on he inal e alua ion esul s.
Pai s o Me hods CFCW VS. FPA CFCWO VS.
CFCW
CFCWO VS.
NFFPCM
MAE esul s
Mean MAE 280.77 s. 362.4 249.67 s. 280.77 10.06 s. 611.12
Mean p- alue 0.00787 0.00013 0.00156
S a is ical
conclusion >> >> >>
MAPE esul s
Mean MAPE 8.42 s. 10.06 7.61 s. 8.42 636.52 s. 17.41
Mean p- alue 0.00475 0.00124 0.00017
S a is ical
conclusion >> >> >>
RMSE esul s
Mean SE 459.87 s. 636.52 403.42 s. 459.87 403.42 s. 769.26
Mean p- alue 0.00215 0.00287 0.00445
S a is ical
conclusion >> >> >>
All used e alua ion c i e ia esul s in his s udy we e used as he sample es se o
each me hod in his s udy.
7. Th ea s o Validi y
In e nal alidi y in his s udy, which can a ec he alidi y o conclusions d awn
om expe imen al esea ch, is an inco ec /inaccu a e e alua ion me hod o assess he
p oposed me hod; speci ically, i e e s o he echnique o s a is ical sample alida ion.
The h ea o he alidi y was con olled using he k- old c oss- alida ion me hod, which
gua an ees ha he p oposed me hod is accu a ely assessed. Ano he in e nal h ea ha
may a ec he alidi y o he ob ained esul s is he choice o pa ame e s in he machine
lea ning echnique. In his s udy, we use he de aul pa ame e se ings o he Bayesian
idge eg esso echnique o he p oposed algo i hm.
Ex e nal alidi y in his s udy is conce ned wi h he ange o alidi y o he esul s
ob ained, and whe he he esul s ob ained could be applied in a di e en con ex . The
ISBSG eposi o y Augus 2020 R1 da ase was used o assess he p edic i e abili y o he
p oposed me hod. This da ase con ains many so wa e p ojec s collec ed om di e en
o ganisa ions wo ldwide ha di e in e ms o ea u es, ields, size, and numbe o ea u es.
Unbiased e alua ion c i e ia a e used o e alua e he pe o mance accu acy o he
p oposed me hod. This s udy used e alua ion c i e ia such as he MAE, MAPE, and RMSE,
which a e unbiased e alua ion c i e ia acco ding o p e ious esea ch [
60
,
61
]. The e o e,
we can conclude ha he expe imen al esul s o his s udy a e highly gene alizable.
8. Conclusions and Fu u e Wo k
A s anda d IFPUG FPA me hod calib a ion algo i hm based on he Bayesian idge
eg esso model o calib a ion (CFCW) and he o ing eg esso model o op imising e o
es ima ion (CFCWO) wi h and wi hou da ase g ouping is p esen ed in his s udy. This
pape aimed o answe h ee esea ch ques ions: In answe o RQ1, we can see a pe cen age
accu acy imp o emen wi h he p oposed CFCW algo i hm compa ed o he IFPUG FPA
me hod, depending on he e alua ion c i e ia and whe he a g ouped o ung ouped

Compu e s 2022,11, 15 17 o 20
da ase was used. Fo he ung ouped da ase , he pe cen age accu acy imp o emen o
MAE = 5.46%, MAPE = 4.10%, and RMSE = 10.39%. The mean pe cen age di e ence o he
indi idual sec o s compa ed o he ung ouped da ase was MAE = 22.52%, MAPE = 16.26%,
and RMSE = 27.75%, showing an e en g ea e imp o emen in he accu acy o he es ima es.
This demons a es ha he IFPUG FPA me hod needs calib a ion, and can be calib a ed.
When CFCW is compa ed o NFFCMP, MAE = 15.39%, MAPE = 28.41%, and RMSE = 19.9%
o all sec o s.
The second p oposed algo i hm, CFCWO, b ings u he imp o emen , and ou pe -
o ms he CFCW algo i hm, answe ing RQ2. The pe cen age imp o emen a ies acco ding
o he e alua ion c i e ia and da ase . Fo he ung ouped da ase , he pe cen age accu acy
imp o emen is MAE = 11.89%, MAPE = 6.56%, and RMSE = 24.62%. The mean pe cen age
di e ence o he indi idual sec o s compa ed o he ung ouped da ase is MAE = 11.08%,
MAPE = 9.59%, and RMSE = 12.28%. The esul s also show ha i makes sense o wo k
wi h da a belonging o a speci ic g oup. In ou case, we g ouped he da a acco ding o
he IS. The answe o RQ3 is ha he es ima e’s accu acy in all indi idual sec o s o all
e alua ion c i e ia is highe han o an ung ouped da ase .
The unc ional complexi y weigh alues e lec he mode n so wa e indus y end o
imp o ing wo k pe o mance hanks o he de elopmen o compu e echnology, p og am-
ming languages, and CASE ools. This mani es s i sel in unc ional complexi y weigh
alues ha a e smalle han he o iginal alue. In addi ion, he demand o sophis ica ion
and complexi y o so wa e unc ions also inc eases o e ime in ce ain a eas, mani es -
ing in calib a ed unc ional complexi y weigh alues ha a e mo e signi ican han he
o iginal alues.
IFPUG FPA is a calcula ion me hod ha es ima es he size, cos , and e o in he ield
o so wa e de elopmen ; i plays a signi ican ole in oday’s so wa e indus y. Howe e ,
so wa e enginee ing is a apidly e ol ing ield; oday’s ac ual alues may no accu a ely
e lec omo ow’s so wa e alues. The e o e, he weigh s p oposed in his pape need
o be upda ed acco ding o he new end. The ISBSG da ase is an up- o-da e da abase
o companies a ound he globe; i e lec s he mode n so wa e indus y ha is cons an ly
upda ed. The e o e, in he u u e, when p ojec da a a e upda ed, he IFPUG FPA weigh ing
alues should be ecalib a ed o e lec he la es so wa e indus y ends.
Au ho Con ibu ions:
Concep ualiza ion, V.V.H., P.S. and H.L.T.K.N.; me hodology, V.V.H., R.S. and
Z.P.; so wa e, V.V.H. and H.L.T.K.N.; alida ion, V.V.H., P.S. and H.L.T.K.N.; in es iga ion, V.V.H.,
H.L.T.K.N., R.S. and Z.P.; esou ces, P.S., Z.P. and R.S.; da a cu a ion, V.V.H., P.S. and H.L.T.K.N.;
w i ing—o iginal d a p epa a ion, V.V.H., R.S. and Z.P.; w i ing— e iew and edi ing, V.V.H. and
R.S.; isualiza ion, V.V.H., P.S. and H.L.T.K.N.; supe ision, R.S. and Z.P.; p ojec adminis a ion,
P.S., R.S. and Z.P.; unding acquisi ion, P.S., R.S. and Z.P. All au ho s ha e ead and ag eed o he
published e sion o he manusc ip .
Funding:
This esea ch was unded by he Facul y o Applied In o ma ics, Tomas Ba a Uni e si y in
Zlin, unde P ojec No.: RVO/FAI/2021/002.
Ins i u ional Re iew Boa d S a emen : No applicable.
In o med Consen S a emen : No applicable.
Da a A ailabili y S a emen :
The ISBSG da a used o suppo he indings o his s udy may be
eleased upon applica ion o he ISBSG, which can be con ac ed a [email p o ec ed]g o h p://isbsg.
o g/academic-subsidy (accessed on 20 Sep embe 2021).
Con lic s o In e es : The au ho s decla e no con lic o in e es .
Compu e s 2022,11, 15 18 o 20
Re e ences
1. S andish G oup Repo . A ailable online: h ps://www.s andishg oup.com (accessed on 20 Sep embe 2021).
2.
Ve a, T.; Ochoa, S.F.; Pe o ich, D. Su ey o So wa e De elopmen E o Es ima ion Taxonomies; Technical Repo ; Compu e Science
Depa men , Uni e si y o Chile: San iago, Chile, 2017.
3.
Khan, B.; Khan, W.; A shad, M.; Jan, N. So wa e cos es ima ion: Algo i hmic and non-algo i hmic app oaches. In . J. Da a Sci.
Ad . Anal. 2020,2, 1–5.
4.
Fa ia, P.; Mi anda, E. Expe judgmen in so wa e es ima ion du ing he bid phase o a p ojec —An explo a o y su ey. In
P oceedings o he 2012 Join Con e ence o he 22nd In e na ional Wo kshop on So wa e Measu emen and he 2012 Se en h
In e na ional Con e ence on So wa e P ocess and P oduc Measu emen , IEEE, Assisi, I aly, 17–19 Oc obe 2012; pp. 126–131.
5.
Azzeh, M.; Nassi , A.B. Analogy-based e o es ima ion: A new me hod o disco e se o analogies om da ase cha ac e is ics.
IET So w. 2015,9, 39–50. [C ossRe ]
6.
Pu nam, L.H. A gene al empi ical solu ion o he mac o so wa e sizing and es ima ing p oblem. IEEE T ans. So w. Eng.
1978
,4,
345–361. [C ossRe ]
7. Boehm, B. So wa e Enginee ing Economics; P en ice-Hall: Englewood Cli s, NJ, USA, 1981.
8.
Alb ech , A.J. Measu ing applica ion de elopmen p oduc i i y. In P oceedings o he IBM Applica ions De elopmen Sympo-
sium, Mon e ey, CA, USA, 14–17 Oc obe 1979; p. 83.
9.
IFPUG. Func ion Poin Coun ing P ac ices; Manual, Release 4.3.1; In e na ional Func ion Poin Use s G oup: Wes e ille, OH, USA, 2010.
10.
ISO/IEC 19761:2011; So wa e Enginee ing—COSMIC: A Func ional Size Measu emen Me hod. In e na ional O ganiza ion o
S anda diza ion: Gene a, Swi ze land, 2011.
11.
ISO/IEC 29881:2010; In o ma ion Technology—Sys ems and So wa e Enginee ing—FiSMA 1.1 Func ional Size Measu emen
Me hod. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2010.
12.
ISO/IEC 20968:2002; So wa e Enginee ing—MK II Func ion Poin Analysis—Coun ing P ac ices Manual. In e na ional O ganiza-
ion o S anda diza ion: Gene a, Swi ze land, 2002.
13.
ISO/IEC 24570:2005; So wa e Enginee ing—NESMA Func ional Size Measu emen Me hod Ve sion 2.1—De ini ions and Coun ing
Guidelines o he Applica ion o Func ion Poin Analysis. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2005.
14.
Ki chenham, B.; Mendes, E. Why compa a i e e o p edic ion s udies may be in alid. In P oceedings o he 5 h In e na ional
Con e ence on P edic o Models in So wa e Enginee ing—PROMISE’09, Vancou e , BC, Canada, 18 May 2009; ACM P ess:
New Yo k, NY, USA, 2009; pp. 1–5.
15. Pe e , R.H. P ac ical So wa e P ojec Es ima ion; McG aw-Hill: New Yo k, NY, USA, 2011.
16.
Hai, V.V.; Nhung, H.L.T.K.; Hoc, H.T. A e iew o so wa e e o es ima ion by using unc ional poin s analysis. In Compu a ional
S a is ics and Ma hema ical Modeling Me hods in In elligen Sys ems; Silha y, R., Silha y, P., P okopo a, Z., Eds.; Sp inge : Cham,
Swi ze land, 2019; pp. 408–422.
17.
Al-Haj i, M.A.; Abdul Ghani, A.A.; Sulaiman, M.N.; Selama , M.H. Modi ica ion o s anda d Func ion Poin complexi y weigh s
sys em. J. Sys . So w. 2005,74, 195–206. [C ossRe ]
18.
Xia, W.; Cap e z, L.F.; Ho, D.; Ahmed, F. A new calib a ion o Func ion Poin complexi y weigh s. In . So w. Technol.
2008
,50,
670–683. [C ossRe ]
19.
Shukla, S.; Kuma , S. Applicabili y o neu al ne wo k based models o so wa e e o es ima ion. In P oceedings o he 2019
IEEE Wo ld Cong ess on Se ices (SERVICES), Milan, I aly, 8–13 July 2019; IEEE: Milan, I aly, 2019; pp. 339–342.
20.
Shukla, S.; Kuma , S.; Bal, P.R. Analyzing e ec o ensemble models on mul i-laye pe cep on ne wo k o so wa e e o
es ima ion. In P oceedings o he 2019 IEEE Wo ld Cong ess on Se ices (SERVICES), Milan, I aly, 8–13 July 2019; IEEE: Milan,
I aly, 2019; pp. 386–387.
21.
P iya, V.A.G.; Ani ha, K.; Va ada ajan, V. Es ima ing so wa e de elopmen e o s using a andom o es -based s acked ensemble
app oach. Elec onics 2021,10, 1195. [C ossRe ]
22.
Id i, A.; Hosni, M.; Ab an, A. Sys ema ic li e a u e e iew o ensemble e o es ima ion. J. Sys . So w.
2016
,118, 151–175.
[C ossRe ]
23.
Id i, A.; Hosni, M.; Ab an, A. Sys ema ic mapping s udy o ensemble e o es ima ion. In P oceedings o he 11 h In e na ional
Con e ence on E alua ion o No el So wa e App oaches o So wa e Enginee ing, Rome, I aly, 27–28 Ap il 2016; pp. 132–139.
24.
In e na ional So wa e Benchma king S anda ds G oupm. ISBSG Reposi o y Augus 2020 R1. A ailable online: h ps://www.isbsg.o g
(accessed on 20 Sep embe 2021).
25.
Silha y, P.; Silha y, R.; P okopo a, Z. Ca ego ical a iable segmen a ion model o so wa e de elopmen e o es ima ion. IEEE
Access 2019,7, 9618–9626. [C ossRe ]
26.
Pospieszny, P.; Cza nacka-Ch obo , B.; Kobylinski, A. An e ec i e app oach o so wa e p ojec e o and du a ion es ima ion
wi h machine lea ning algo i hms. J. Sys . So w. 2018,137, 184–196. [C ossRe ]
27.
Jayakuma , K.R.; Ab an, A. Es ima ion models o so wa e unc ional es e o . J. So w. Eng. Appl.
2017
,10, 338–353. [C ossRe ]
28.
Wei, K.T.; Selama , M.H.; Ghani, A.A.A.; Abdullah, R. Exponen ial E o Es ima ion Model Using Unadjus ed Func ion Poin s.
In P oceedings o he 5 h In e na ional Con e ence on New T ends in In o ma ion Science and Se ice Science, Macao, China,
24–26 Oc obe 2011; IEEE: Macao, China, 2011; pp. 111–115.
29.
Mis a, S.; Adewumi, A.; Fe nandez-Sanz, L.; Damase icius, R. A Sui e o Objec O ien ed Cogni i e Complexi y Me ics. IEEE
Access 2018,6, 8782–8796. [C ossRe ]
Compu e s 2022,11, 15 19 o 20
30.
Dewi, R.S.; Sub iadi, A.P.; Sholiq, S. A modi ica ion complexi y ac o in unc ion poin s me hod o so wa e cos es ima ion
owa ds public se ice applica ion. P ocedia Compu . Sci. 2017,124, 415–422. [C ossRe ]
31.
Leal, L.Q.; Fagundes, R.A.A.; de Souza, R.M.C.R.; Mou a, H.P.; Gusmao, C.M.G. Nea es -neighbo hood linea eg ession in an
applica ion wi h so wa e e o es ima ion. In P oceedings o he 2009 IEEE In e na ional Con e ence on Sys ems, Man and
Cybe ne ics, San An onio, TX, USA, 11–14 Oc obe 2009; pp. 5030–5034.
32.
Hamza, H.; Kamel, A.; Shams, K. So wa e e o es ima ion using a i icial neu al ne wo ks: A su ey o he cu en p ac ices.
In P oceedings o he 2013 10 h In e na ional Con e ence on In o ma ion Technology: New Gene a ions, Las Vegas, NV, USA,
15–17 Ap il 2013; IEEE: Las Vegas, NV, USA, 2013; pp. 731–733.
33.
Lena duzzi, V.; Mo asca, S.; Taibi, D. Es ima ing so wa e de elopmen e o based on phases. In P oceedings o he 2014 40 h
EUROMICRO Con e ence on So wa e Enginee ing and Ad anced Applica ions, Ve ona, I aly, 27–29 Augus 2014; IEEE: Ve ona,
I aly, 2014; pp. 305–308.
34.
P okopo a, Z.; Silha y, P.; Silha y, R. In luence analysis o selec ed ac o s in he unc ion poin wo k e o es ima ion. In
In elligen Sys ems in Cybe ne ics and Au oma ion Con ol Theo y; Silha y, R., Silha y, P., P okopo a, Z., Eds.; Sp inge : Cham,
Ge many, 2019; pp. 112–124.
35.
Hammad, M.; Alqaddoumi, A. Fea u es-le el so wa e e o es ima ion using machine lea ning algo i hms. In P oceedings o
he 2018 In e na ional Con e ence on Inno a ion and In elligence o In o ma ics, Compu ing, and Technologies (3ICT), Sakhie ,
Bah ain, 18–20 No embe 2018; IEEE: Sakhie , Bah ain, 2018; pp. 1–3.
36.
Abdella i , T.M. A compa ison s udy be ween so compu ing and s a is ical eg ession echniques o so wa e e o es ima ion.
In P oceedings o he 2018 IEEE Canadian Con e ence on Elec ical & Compu e Enginee ing (CCECE), Quebec, QC, Canada,
13–16 May 2018; IEEE: Quebec, QC, Canada, 2018; pp. 1–5.
37.
IFPUG. Announcing he New Business Applica ions Commi ee. A ailable online: h ps://www.i pug.o g (accessed on
20 Sep embe 2021).
38.
ISO/IEC 20926:2009; So wa e and Sys ems Enginee ing—So wa e Measu emen —IFPUG Func ional Size Measu emen Me hod.
In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2009.
39.
Azzeh, M.; Nassi , A.B. Analyzing he ela ionship be ween p ojec p oduc i i y and en i onmen ac o s in he use case poin s
me hod. J. So w. E ol. P ocess 2017,29, e1882. [C ossRe ]
40.
Azzeh, M.; Nassi , A.B.; Bani aan, S. Compa a i e analysis o so compu ing echniques o p edic ing so wa e e o based use
case poin s. IET So w. 2018,12, 19–29. [C ossRe ]
41. MacKay, D.J.C. Bayesian in e pola ion. Neu al Compu . 1992,4, 415–447. [C ossRe ]
42. Michael, E.T. Spa se Bayesian lea ning and he ele ance ec o machine. J. Mach. Lea n. Res. 2001,1, 211–244.
43.
Pa k, S.Y.; Be a, A.K. Maximum en opy au o eg essi e condi ional he e oskedas ici y model. J. Econom.
2009
,150, 219–230.
[C ossRe ]
44.
Kocaguneli, E.; Menzies, T.; Keung, J.W. On he alue o ensemble e o es ima ion. IEEE T ans. So w. Eng.
2012
,38, 1403–1416.
[C ossRe ]
45.
Kocaguneli, E.; Kul u , Y.; Bene , A. Combining mul iple lea ne s induced on mul iple da ase s o so wa e e o p edic ion. In .
Symp. So w. Reliab. Eng. 2009,17, 25–49.
46.
An, K.; Meng, J. Vo ing-a e aged combina ion me hod o eg esso ensemble. In Ad anced In elligen Compu ing Theo ies and
Applica ions; Huang, D.S., Zhao, Z., Be ilacqua, V., Figue oa, J.C., Eds.; Sp inge : Be lin/Heidelbe g, Ge many, 2010; pp. 540–546.
47.
Wi en, I.H.; F ank, E.; Hall, M.A.; Pal, C.J. Da a Mining: P ac ical Machine Lea ning Tools and Techniques; Mo gan Kau mann:
Ams e dam, The Ne he lands, 2017.
48.
Azzeh, M.; Nassi , A.B.; Minku, L.L. An empi ical e alua ion o ensemble adjus men me hods o analogy-based e o es ima ion.
J. Sys . So w. 2015,103, 36–52. [C ossRe ]
49. Lich enbe g, J.M.; ¸Sim¸sek, Ö. Simple eg ession models. P oc. Mach. Lea n. 2016,58, 13–25.
50. Up on, G.; Cook, I. Unde s anding S a is ics; Ox o d Uni e si y P ess: Ox o d, UK, 1996.
51.
Zwillinge , D.; Kokoska, S. CRC S anda d P obabili y and S a is ics Tables and Fo mulae; Chapman & Hall/CRC P ess: Boca Ra on,
FL, USA, 2000.
52.
Foss, T.; S ens ud, E.; Ki chenham, B.; My ei , I. A simula ion s udy o he model e alua ion c i e ion mm e. IEEE T ans. So w.
Eng. 2003,29, 985–995. [C ossRe ]
53.
Ki chenham, B.A.; Picka d, L.M.; MacDonell, S.G.; Sheppe d, M.J. Wha accu acy s a is ics eally measu e. IEE P oc. So w.
2001
,
148, 81–85. [C ossRe ]
54. Ha din, J.; Ha din, J.; Hilbe, J.; Hilbe, J. Gene alized Linea Models and Ex ensions; S a a P ess: College S a ion, TX, USA, 2007.
55.
Gup a, H.V.; Kling, H.; Yilmaz, K.K.; Ma inez, G.F. Decomposi ion o he mean squa ed e o and NSE pe o mance c i e ia:
Implica ions o imp o ing hyd ological modelling. J. Hyd ol. 2009,377, 80–91. [C ossRe ]
56.
Sheppe d, M.; MacDonell, S. E alua ing p edic ion sys ems in so wa e p ojec es ima ion. In . So w. Technol.
2012
,54, 820–827.
[C ossRe ]
57.
Kaise , M.; Ull ich, C. Es ima ion accu acy in la ge is p og ams insigh s om a desc ip i e case s udy. In P oceedings o he 22s
Eu opean Con e ence on In o ma ion Sys ems, Tel A i , Is ael, 9–11 June 2014; pp. 1–14.
58.
Ande son, D.R.; Sweeney, D.J.; William, T.A. S a is ics o Business and Economics, 14 h ed.; Thomson Sou h-Wes e n, Cengage
Lea ning: Bos on, MA, USA, 2009.
Compu e s 2022,11, 15 20 o 20
59.
Ross, A.; Willson, V.L.. Pai ed Samples -Tes . In Basic and Ad anced S a is ical Tes s; SensePublishe s: Ro e dam, The Ne he lands,
2017; pp. 17–19.
60.
Chai, T.; D axle , R.R. Roo mean squa e e o (RMSE) o mean absolu e e o (MAE)?—A gumen s agains a oiding RMSE in he
li e a u e. Geosci. Model De . 2014,7, 1247–1250. [C ossRe ]
61.
Tod os, K.; Tab ikian, J. On o de ela ions be ween lowe bounds on he MSE o unbiased es ima o s. In P oceedings o he
2010 IEEE In e na ional Symposium on In o ma ion Theo y, Aus in, TX, USA, 13–18 June 2010; IEEE: Aus in, TX, USA, 2010;
pp. 1663–1667.