A new approach to calibrating functional complexity weight in software development effort estimation

  Ci a ion: Hai, V.V.; Nhung, H.L.T.K.; P okopo a, Z.; Silha y, R.; Silha y, P. A New App oach o Calib a ing Func ional Complexi y Weigh in So wa e De elopmen E o Es ima ion. Compu e s 2022,11, 15. h ps://doi.o g/10.3390/ compu e s11020015 Academic Edi o : Robe as Damaše iˇcius Recei ed: 15 Decembe 2021 Accep ed: 19 Janua y 2022 Published: 22 Janua y 2022 Publishe ’s No e: MDPI s ays neu al wi h ega d o ju isdic ional claims in published maps and ins i u ional a il- ia ions. Copy igh : © 2022 by he au ho s. Licensee MDPI, Basel, Swi ze land. This a icle is an open access a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons A ibu ion (CC BY) license (h ps:// c ea i ecommons.o g/licenses/by/ 4.0/). compu e s A icle A New App oach o Calib a ing Func ional Complexi y Weigh in So wa e De elopmen E o Es ima ion Vo Van Hai * , Ho Le Thi Kim Nhung , Zdenka P okopo a, Radek Silha y and Pe Silha y Depa men o Compu e and Communica ion Sys ems, Tomas Ba a Uni e si y in Zlin, Nam. T.G.M. 5555, 76001 Zlin, Czech Republic; [email p o ec ed] (H.L.T.K.N.); p [email p o ec ed] (Z.P.); [email p o ec ed] (R.S.); [email p o ec ed] (P.S.) *Co espondence: [email p o ec ed] Abs ac : Func ion poin analysis is a widely used me ic in he so wa e indus y o de elopmen e o es ima ion. I was p oposed in he 1970s, and hen s anda dized by he In e na ional Func ion Poin Use s G oup, as accep ed by many o ganiza ions wo ldwide. While he so wa e indus y has g own apidly, he weigh alues speci ied o he s anda d unc ion poin coun ing ha e emained he same since i s incep ion. Ano he p oblem is ha so wa e de elopmen in di e en indus y sec o s is peculia , bu basic ules apply o all. These aise impo an ques ions abou he alidi y o weigh alues in p ac ical applica ions. In his s udy, we p opose an algo i hm o calib a ing he s anda dized unc ional complexi y weigh s, aiming o es ima e a mo e accu a e so wa e size ha i s speci ic so wa e applica ions, e lec s so wa e indus y ends, and imp o es he e o es ima ion o so wa e p ojec s. The esul s show ha he p oposed algo i hms imp o e e o es ima ion accu acy agains he baseline me hod. Keywo ds: so wa e de elopmen e o es ima ion; unc ion poin analysis; unc ional complexi y weigh 1. In oduc ion So wa e es ima ion has long been conside ed a co e issue ha di ec ly a ec s success o ailu e. Acco ding o he S andish G oup [ 1 ], he ailu e a e o a pa o a p ojec o o a whole p ojec is likely o be up o 83.9% (as o 2019). One o he easons o his ailu e is inaccu a e cos and e o es ima es. In ac , o ob ain so wa e p ojec s, companies pa icipa ing in ende s mus submi bids ha include cos , manpowe , and so wa e de elopmen ime. To be able o win he ende , he companies pa icipa ing need o gi e a easonable es ima e o he cos , manpowe , and ime equi ed o ca y ou he p ojec . Reasonabili y he e does no mean unde es ima ing he p ice, because in so doing he company will no gain (i no lose) when comple ing he p ojec . I is also no easonable o o e es ima e he p ice, because hen i is ce ain ha he company will no win he bid. The e o e, a p ojec es ima e is conside ed easonable only i i accu a ely e lec s he p ojec ’s ac ual alue. Th oughou he so wa e de elopmen p ocess, no ma e wha so wa e manage- men model a company uses, p ojec leade s o en ha e o plan he wo k o so wa e de elopmen miles ones, plan he nex miles one, and ecalcula e he wo k done in he p e ious miles one. All o hese asks equi e so wa e es ima ion skills. Many me hods ha e p e iously been p oposed o sol e he so wa e es ima ion p oblem. Due o he inc easing demand o mo e e icien and accu a e es ima ion me hods ha can wo k wi h mo e complex so wa e p ojec s, such es ima ion me hods need o be e ined. So wa e es ima ion me hods can be classi ied in o h ee main g oups: non-algo i hmic, algo i hmic, and machine lea ning app oaches [ 2 , 3 ]. In he non-algo i hmic ca ego y, he e a e wo ep esen a i e me hods: expe judgemen (EJ) [ 4 ] and analogy [ 5 ]; wi h hese me hods, expe s play he mos signi ican ole in judgemen . O cou se, p e ious samples (his o ical da ase ) also play ano he impo an ole. In he algo i hmic ca ego y, an algo i hm akes Compu e s 2022,11, 15. h ps://doi.o g/10.3390/compu e s11020015 h ps://www.mdpi.com/jou nal/compu e s Compu e s 2022,11, 15 2 o 20 he i s ole. so wa e li ecycle managemen (SLIM) [ 6 ], he Cons uc i e Cos Model (CO- COMO) [ 7 ], and unc ion poin analysis (FPA) and he In e na ional Func ion Poin Use G oup (IFPUG FPA) [ 8 , 9 ] a e ep esen a i e models. The IFPUG FPA me hod a ose as an al e na i e o o he solu ions. O he me hods based on IFPUG FPA —such as COSMIC [ 10 ], FiSMA [ 11 ], Ma kII [ 12 ], and NESMA [ 13 ]—we e p oposed o imp o e some aspec s o he o iginal me hod. In he las ca ego y, machine lea ning echniques ha e been used in ecen yea s o supplemen o eplace he o he wo echniques. Mos so wa e cos es ima ion echniques use s a is ical me hods, which canno p o ide s ong a ionales and esul s. Machine lea ning app oaches may be app op ia e in his a ea because hey can inc ease he accu acy o es ima es by aining he es ima ion ules and epea ing he unning cy- cles. Examples include uzzy logic modelling, eg ession ees, a i icial neu al ne wo ks (ANNs), and case-based easoning (CBR). These me hods ha e explo ed he applicabili y o machine lea ning echniques o so wa e e o es ima ion; hey ha e objec i e and analy i- cal o mulas ha a e no limi ed o a speci ic numbe o ac o s. Howe e , mos me hods only e alua e he limi a ions o modelling echniques on a pa icula da ase , educing he gene alisabili y o he obse ed esul s. Some esea che s also o e look s a is ical es ing o he esul s ob ained o e alua ion o he models agains he same da ase used o gene a e he models [14]. This pape is o ganised as ollows: Sec ion 1is he in oduc ion. The p oblem o mula- ion and he con ibu ions a e desc ibed in Sec ion 2. Sec ion 3illus a es he ela ed wo ks. Sec ion 4p esen s he backg ound, wi h a b ie o e iew o he FPA me hod, Bayesian idge eg ession model, and o ing eg esso model. The p oposed esea ch me hodology is exp essed in Sec ion 5, in which we in oduce he da ase and da a p ocessing, expe - imen al se up, and e alua ion c i e ia. Sec ion 6p esen s he expe imen al esul s and discussion. Sec ion 7desc ibes he h ea s o alidi y. We inalise ou pape in Sec ion 8 wi h he conclusion. 2. P oblem Fo mula ion FPA has been used o o e ou decades, and has p o en o be a dependable and con- sis en me hod [ 15 ] o sizing so wa e o p ojec es ima ion and p oduc i i y o e iciency compa isons. Al hough i has made signi ican con ibu ions in he so wa e indus y, i s ill has many p oblems. In an ea lie sys ema ic li e a u e e iew [ 16 ], we men ioned some limi a ions o FPA. The inadequacy o complexi y weigh is s ill a majo p oblem. In addi ion, he locali y o he da ase ha builds he FPA app oach (IBM p ojec s) does no e lec he en i e global so wa e indus y. Many p e ious s udies ha e sugges ed a new unc ional complexi y weigh [ 17 , 18 ] in di e en ways. In [ 18 ], Xia e al. p oposed a new unc ional complexi y able based on an IFPUG FPA calib a ion model called Neu o-Fuzzy Func ion Poin Calib a ion Model (NFFPCM), which in eg a es he powe o neu al ne - wo ks and uzzy logic. Ne e heless, he me hod needs o be changed in line wi h changes in he mode n so wa e indus y. Ano he issue ha needs o be men ioned is he speci ici y o each piece o so wa e. Di e ences in he pu pose o he so wa e being de eloped lead o di e en app oaches. Wi h FPA, a me hod ha applies o all so wa e es ima ion cases needs o be e isi ed; his was he mo i a ion o us o p opose a new, up- o-da e, nonlocal unc ional complexi y weigh ha e lec s he so wa e indus y. A e coun ing he unc ion poin s (FPs), we can calcula e he e o and hen w i e a epo [ 9 ] based on hese FPs. A his poin , he FPA coun ing p ocess is conside ed comple e. Howe e , one ques ion is whe he we can u he imp o e he e o a e he calcula ion o he coun ing p ocess. Recen s udies [ 19 – 23 ] show ha combining an ensemble model and o he app oaches p o ides be e esul s han using a single model. This s udy applies an ensemble model o he esul a e he coun ing and calcula ing e o o imp o e he accu acy gained om FP coun ing based on he p oposed unc ional complexi y weigh . Compu e s 2022,11, 15 3 o 20 This s udy aims o p opose an algo i hm called he calib a ion o unc ional complexi y weigh (CFCW) algo i hm. The p oposed algo i hm is based on he FPA me hod combined wi h eg ession me hods implemen ed in he In e na ional So wa e Benchma king S an- da ds G oup (ISBSG) da ase Release 2020 R1 [24]. Many companies a ound he wo ld con ibu e o he ISBSG da ase , so he locali y p oblem can be sol ed. In addi ion, he 2020 da abase upda e add esses he ou -o -da e is- sue. Many ecen s udies ( o example, [ 25 – 27 ]) used he indus y sec o (IS) as a ca ego ical a iable o da ase segmen ing in hei esea ch. Ou s udy es ed a second app oach— he calib a ion o unc ional complexi y weigh wi h op imisa ion based on an ensemble model called he o ing eg esso (CFCWO). Based on he abo e issues, we p opose h ee esea ch ques ions: RQ1:Is he accu acy o he p oposed CFCW algo i hm be e han ha o he IFPUG FPA o NFFPCM me hods? RQ2: Does he ad anced CFCWO algo i hm ou pe o m he CFCW algo i hm? RQ3: How accu a e is he es ima ion o each sec o compa ed o an ung ouped da ase ? To answe hese esea ch ques ions, we conduc ed an expe imen al s udy o e alua e he es ima ion accu acy o he p oposed app oaches. Con ibu ions The main con ibu ions o his esea ch a e as ollows: • In he i s phase, a new CFCW algo i hm o he calib a ion o unc ional complexi y weigh is p oposed; • In he second phase, he esul om he i s phase is op imised by using a o ing eg esso o es ima e he inal so wa e e o — he CFCWO algo i hm is p oposed; • The IFPUG FPA me hod is compa ed o he CFCW algo i hm o ung ouped da a and da a g ouped by IS; • The CFCW algo i hm is compa ed o he CFCWO algo i hm o ung ouped da a and da a g ouped by IS. 3. Rela ed Wo k FPA is a s anda dised me hod o de e mining he size o so wa e based on i s unc ional equi emen s; i is designed o be applicable ega dless o p og amming language o implemen a ion echnology. Alb ech [ 8 ] ecommends FPA o measu e he size o a sys em ha p ocesses da a om end-use s. Since i s in oduc ion, much esea ch has been ca ied ou o imp o e i s accu acy. Al-Haj i e al. [ 17 ] in oduced a modi ica ion weigh ing sys em o measu ing FP using an ANN model (backp opaga ion echnique). In hei s udy, a weigh ing sys em was buil based on ou s eps: (1) using he o iginal weigh ing sys em as a baseline o es ablish new weigh s; (2) using he DETs/RETs o he o iginal sys em’s FPA o calcula e he new alues, aining hese new alues wi h an ANN, and hen p edic ing he alues o he new weigh s; (3) applying he new weigh s and he o iginal weigh s in he FPA model; and (4) calcula ing he size o FPs as a unc ion o he o iginal weigh s and he new weigh s. Wei e al. [ 28 ] p oposed a di e en sizing app oach by in eg a ing he new calib a ed FP weigh p oposed in [ 18 ] in o a complexi y assessmen sys em o objec - o ien ed de elopmen e o es ima ion. Mis a e al. [ 29 ] p oposed a me ics sui e ha helps in de e mining he complexi y o objec -o ien ed p ojec s by e alua ion o message complexi y, a ibu e complexi y, weigh ed class complexi y, and code complexi y. Dewi e al. [ 30 ] p oduced a o mula o es ima ing he cos o so wa e de elopmen p ojec s, especially in he ield o public se ice applica ions; he au ho s modi ied he complexi y adjus men ac o o 16 ins ead o he 14 used in he s anda d FPA me hod and; as a esul , he accu acy was imp o ed by 7.19%. In he s udy o Leal e al. [ 31 ], he au ho s in es iga ed he use o nea es -neighbou s linea eg ession me hods o es ima ion in so wa e enginee ing. These me hods we e compa ed wi h mul ilaye pe cep on neu al ne wo ks, adial basis unc ion neu al ne - Compu e s 2022,11, 15 4 o 20 wo ks, suppo - ec o eg ession, and bagging p edic o s. The da ase used in he s udy was a NASA so wa e p ojec . Based on he ela i e e o and he es ima ion a e, he nea es -neighbou s linea eg ession me hods ou pe o med he o he s. In a su ey o applying ANNs o so wa e es ima ion, Hamza e al. [ 32 ] p o ided an o e iew o he use o ANN me hods o es ima e de elopmen e o o so wa e de el- opmen p ojec s; he au ho s o e ed ou main ANN models, including (1) eed o wa d neu al ne wo ks; (2) ecu en neu al ne wo ks; (3) adial basis unc ion ne wo ks; and (4) neu o- uzzy ne wo ks. The su ey also explains why hose me hods a e used and how accu a e hey a e. In he endea ou o es ima ing he e o needed o he nex phase o he emaining e o needed o inish a p ojec , Lena duzzi e al. [ 33 ] conduc ed an empi ical s udy on he es ima ion o so wa e de elopmen e o . The es ima ion was b oken down by phase so ha es ima ion could be used h oughou he so wa e de elopmen li ecycle. This means ha he e o needed o he nex phase a any gi en poin in he so wa e de elopmen li ecycle is es ima ed. Addi ionally, hey es ima ed he e o equi ed o he emaining pa o a so wa e de elopmen p ocess. The ISBSG da ase was used in he s udy. The esul s show s a is ically signi ican co ela ions be ween e o expended in one pe iod and e o expended in he ollowing pe iod, e o expended in one pe iod and o al e o expended, and accumula ed e o up o he p esen s age and emaining e o . The esul s also indica e ha hese es ima ion models ha e di e en deg ees o goodness o i . Fu he a ibu es, such as he unc ional size, do no signi ican ly imp o e es ima ion quali y. In [ 25 , 34 ], he au ho s p esen ed an in luence analysis o selec ed ac o s (FP coun app oach, business a ea, IS, and ela i e size) on he es ima ion o he wo k e o o which he FPA me hod is p ima ily used. They also s udied he ac o s ha in luence p oduc i i y and he p oduc i i y es ima ion capabili y in he FPA me hod. Based on hese selec ed ac o s and expe imen ally, he au ho s p o ed ha he selec ed ac o s ha e speci ic e ec s on wo k e o es ima ion accu acy. In [ 35 ], om so wa e ea u es, he au ho s used a ious machine lea ning algo i hms o build a so wa e e o es ima ion model. ANNs, suppo - ec o machines, K-s a , and linea eg ession machine lea ning algo i hms we e app aised on a PROMISE da ase (called Usp05- ) wi h ac ual so wa e e o s. The esul s e ealed ha he machine lea ning app oach could be applied o p edic so wa e e o . In he s udy, he esul s om he suppo - ec o machines we e he bes . In [ 36 ], he au ho s conduc ed a compa ison be ween so compu ing and s a is ical eg ession echniques in e ms o a so wa e de elopmen es ima ion eg ession p oblem. Suppo - ec o eg ession and ANNs we e used as so compu ing me hodologies, and s epwise mul iple linea eg ession and log-linea eg ession we e used as s a is ical eg es- sion me hods. Expe imen s we e pe o med using he NASA93 da ase om he PROMISE so wa e eposi o y, wi h mul iple da ase p e-p ocessing s eps pe o med. The au ho s elied on he holdou echnique associa ed wi h 25 andom epe i ions wi h con idence in e al calcula ion wi hin a 95% s a is ical con idence le el. The 30 p e-e alua ion c i e ia we e used o compa e he esul s. The esul s o he s udy show ha he suppo - ec o eg ession model has a signi ican impac on p ecision. 4. Backg ound 4.1. IFPUG FPA Alb ech [ 8 ] i s in oduced FPA in 1979, and p esen ed he FP me ic o measu e he unc ionali y o a p ojec . This was p oposed in esponse o a numbe o p oblems wi h o he sys em size measu es, such as lines o code. In 1986, he In e na ional Func ion Poin Use G oup (IFPUG) [ 37 ] p omo ed and popula ised e ec i e so wa e de elopmen and main enance managemen h ough FPA. The IFPUG is cu en ly he go e ning body o FPA, and is esponsible o imp o ing and de eloping coun ing ules and o he ela ed ma e s. Since he IFPUG was c ea ed, he o iginal FPA me hod has been known as he IFPUG’s FPA. In his s udy, he s anda d Compu e s 2022,11, 15 5 o 20 FPA me hod concep e e s o he IFPUG FPA me hod. FPA is cu en ly s anda dised by ISO/IEC 20926:2010 [ 38 ]; his s anda d speci ies a se o de ini ions, ules, and s eps o applica ion [ 9 ]. The e a e six phases in coun ing s anda ds; in his s udy, we a e only in e es ed in wo phases: (1) da a unc ion and ansac ional unc ion measu emen , and (2) unc ional size measu emen . The i s -phase esul s a e he unadjus ed unc ion poin s (UFPs) and he alue adjus - men ac o (VAF) alues. The UFP alue can be de e mined based on es ima ions o he numbe o ansac ional unc ions (ex e nal inpu (EI), ex e nal ou pu (EO), o ex e nal inqui y (EQ)) and da a unc ions (in e nal logic iles (ILFs), and ex e nal in e ace iles (EIFs)). These componen s a e called base unc ional componen s. Each o hese, in u n, is judged as low (L), a e age (A), o high (H), and assigned a weigh acco dingly. Table 1 shows he a ailable complexi y weigh o he componen s. Table 1. Da a and ansac ional unc ion complexi y. Componen EI EO EQ EIF ILF Complexi y weigh Low 3 4 3 5 7 A e age 4 5 4 7 10 High 6 7 6 10 15 The UFP o al se s he numbe o ypes in g oups, mul iplies hem by complexi y weigh s, and inds he sum o all ields, as in Equa ion (1): UFP = n ∑ i=1 m ∑ j=1 (Sij ×Wij)(1) whe e Sij ep esen s he o al o each unc ional componen , Wij ep esen s he complexi y weigh s, nis he numbe o ypes, and mis he numbe o complexi y g oups. The VAF coun is based on he a e o 14 gene al sys em cha ac e is ics (GSCs): da a communica ions, dis ibu ed da a p ocessing, pe o mance, hea ily used con igu a ion, ansac ion a e, online da a en y, end use e iciency, online upda e, complex p ocessing, eusabili y, ins alla ion ease, ope a ional ease, mul iple si es, and acili a e change. The e a e six in luence le els o GSC c i e ia, wi h he sys em being de e mined as a alue om 0 o 5 con ingen on he le el: 0—no in luence; 1—inciden al in luence; 2— mode a e in luence; 3—a e age in luence; 4—signi ican in luence; and 5—s ong in luence h oughou . The VAF coun is adjus ed as ollows: VAF =0.65 +0.01 × 14 ∑ i=1 (Fi× a ing)(2) The second-phase esul is he adjus ed unc ion poin s (AFPs) alue, which can be ob ained using Equa ion (3): AFP =UFP ×VAF (3) To es ima e he e o a e AFP coun ing, we should use ano he pa ame e . The p oduc i i y ac o (PF) was desc ibed as he ela ionship be ween one FP and he numbe o hou s needed o i s de elopmen by one pe son. P oduc i i y and PF we e s udied in [39,40]. The ollowing o mula can be used o calcula e he e o using he PF: E o =AFP ×PF (4) ISBSG uses he p oduc i i y deli e y a e (PDR) as a me ic o e iciency. The PDR is measu ed in pe son-hou s pe FP. F om he PDR, we can de i e he PF in FPs pe pe son-hou . We can see ha he PDR is he in e ed alue o he PF (and ice e sa) [9]. Compu e s 2022,11, 15 6 o 20 In ou s udy, he IFPUG FPA me hod is he base me hod o p oposing he new model; i is also used o he base compa ed wi h he p oposed model. Addi ionally, he e ms FPA and IFPUG FPA ha e he same meaning, and a e in e changeable. 4.2. Bayesian Ridge Reg ession Model The ull Bayesian eg ession in e ence uses he Ma ko chain Mon e Ca lo algo i hm o cons uc models [ 41 ]. The Bayesian modelling amewo k has been epu ed o i s abili y o deal wi h a hie a chical da a s uc u e. In Bayesian eg ession echniques, egula isa ion pa ame e s can be included in he es ima ion p ocedu e. A egula isa ion pa ame e is no ha d se , bu is uned o he da a a hand. This can be done by in oducing non- in o ma i e p io s o e he hype pa ame e s o he model. The l2 egula isa ion used in idge eg ession and classi ica ion is equi alen o inding a maximum a pos e io i es ima e unde a Gaussian p io o e he coe icien s wi h p ecision. Ins ead o se ing he lambda manually, his a iable can be andomly es ima ed om he a ailable da a [ 42 ]. To acqui e a ully p obabilis ic model, he ou pu yis assumed o be Gaussian dis ibu ed a ound Xω : p(y|X,ω,α)=ℵ(y|Xω,α)(5) whe e αis ea ed as a andom a iable ha is o be es ima ed om he da a. Bayesian idge eg ession (BRR) is a p obabilis ic me hod ha builds a eg ession model using Bayesian in e ence; i combines p io in o ma ion abou pa ame e s ( he coe icien o so wa e ea u es) wi h he obse ed aining da a in o de o acqui e he pa ame e s’ pos e io dis ibu ion [ 42 ]. The p io o he coe icien ω is speci ied by a sphe ical Gaussian: p(ω|λ)=ℵω  0, λ−1Ip(6) The p io s o e α and λ a e picked o be gamma dis ibu ions [ 43 ]— he conjuga e p io o he p ecision o he Gaussian dis ibu ion. In ou s udy, he BRR plays a signi ican ole in he calib a ion phase (see Figu e 1). Compu e s 2022, 11, 15 7 o 20 Figu e 1. Expe imen al p ocess. 4.3. Vo ing Reg esso Model The ensemble is a lea ning me hod ha uses a speci ic agg ega ion mechanism o c ea e a collec ion o p edic ion models, and hen uses a weigh ed o e o hei ini ial esul s o ob ain he inal solu ion [44–48]. The p incipal p emise is ha i echniques wo k oge he as a commi ee wi h eliable me hods, hey may be imp o ed and gene a e mo e signi ican esul s [44]. As a esul , his me hod is excellen o p edic ing so wa e e o , since each model has i s assump ions and se up pa ame e s, allowing he ensemble o pe o m excep ionally well wi h some desi able s a is ical quali ies [45]. Id i e al. [22,23] conduc ed a sys ema ic li e a u e e iew and mapping s udy, and disco e ed ha (1) ensemble e o es ima ion echniques a e mo e accu a e han solo me hods, (2) homogeneous ensembles a e he mos in es iga ed, (3) machine lea ning echniques a e he mos used solo echniques o cons uc ensembles, and (4) he e a e wo ypes o combine ules used o es ima e ensemble e o es ima ion: linea and nonlinea . A o ing eg esso [46] is based on he idea o in eg a ing a ious machine lea ning app oaches o e u n uni o m a e age p ojec ed alues. A o ing eg esso is a echnique ha i s each o he base eg esso s o he en i e da ase . A eg esso such as his can help a g oup o es ima o s wi h simila pe o mance le els o balance ou hei indi idual laws. When he p edic o s a e as independen as possible, ensemble app oaches pe o m bes . In gene al, each eg esso is ained using a dis inc echnique, in o de o make each p edic ion mo e independen o he o he s. This inc eases he likelihood ha hey will make a a ie y o blunde s, which will imp o e he ensemble’s pe o mance. A o ing eg esso can be applied o classi ica ion o eg ession. Each label’s p edic ions a e combined wi h ega d o classi ica ion, and he label wi h he mos o es is chosen. In he case o eg ession, his en ails compu ing he mean o he p edic ions om he models. Acco ding o Wi en e al. [47], a o ing ensemble is app op ia e when all applicable models should pe o m well on a p edic i e modelling ask; in o he wo ds, he models used in he ensemble mus mos ly ag ee. In ou s udy, he o ing eg esso was used in he CFCWO algo i hm in he op imiza ion phase (Figu e 1). Figu e 1. Expe imen al p ocess. Compu e s 2022,11, 15 7 o 20 4.3. Vo ing Reg esso Model The ensemble is a lea ning me hod ha uses a speci ic agg ega ion mechanism o c ea e a collec ion o p edic ion models, and hen uses a weigh ed o e o hei ini ial esul s o ob ain he inal solu ion [ 44 – 48 ]. The p incipal p emise is ha i echniques wo k oge he as a commi ee wi h eliable me hods, hey may be imp o ed and gene a e mo e signi ican esul s [ 44 ]. As a esul , his me hod is excellen o p edic ing so wa e e o , since each model has i s assump ions and se up pa ame e s, allowing he ensemble o pe o m excep ionally well wi h some desi able s a is ical quali ies [ 45 ]. Id i e al. [ 22 , 23 ] conduc ed a sys ema ic li e a u e e iew and mapping s udy, and disco e ed ha (1) ensemble e o es ima ion echniques a e mo e accu a e han solo me hods, (2) homogeneous ensembles a e he mos in es iga ed, (3) machine lea ning echniques a e he mos used solo echniques o cons uc ensembles, and (4) he e a e wo ypes o combine ules used o es ima e ensemble e o es ima ion: linea and nonlinea . A o ing eg esso [ 46 ] is based on he idea o in eg a ing a ious machine lea ning app oaches o e u n uni o m a e age p ojec ed alues. A o ing eg esso is a echnique ha i s each o he base eg esso s o he en i e da ase . A eg esso such as his can help a g oup o es ima o s wi h simila pe o mance le els o balance ou hei indi idual laws. When he p edic o s a e as independen as possible, ensemble app oaches pe o m bes . In gene al, each eg esso is ained using a dis inc echnique, in o de o make each p edic ion mo e independen o he o he s. This inc eases he likelihood ha hey will make a a ie y o blunde s, which will imp o e he ensemble’s pe o mance. A o ing eg esso can be applied o classi ica ion o eg ession. Each label’s p edic ions a e combined wi h ega d o classi ica ion, and he label wi h he mos o es is chosen. In he case o eg ession, his en ails compu ing he mean o he p edic ions om he models. Acco ding o Wi en e al. [ 47 ], a o ing ensemble is app op ia e when all applicable models should pe o m well on a p edic i e modelling ask; in o he wo ds, he models used in he ensemble mus mos ly ag ee. In ou s udy, he o ing eg esso was used in he CFCWO algo i hm in he op imiza- ion phase (Figu e 1). 5. Resea ch Me hodology In his sec ion, we p esen he esea ch me hodology. This includes desc ibing he da a o be used, along wi h he da a p ocessing and he expe imen al se up. In addi ion, e alua ion c i e ia a e in oduced he e. 5.1. Expe imen al Se up In his sec ion, we desc ibe he expe imen al p ocess, which is g aphically illus a ed in Figu e 1. In he da a p e-p ocessing phase (Figu e 1), da a il e ing and cleaning we e pe o med o c ea e he wo king da ase (see ollowing sec ion). This da ase was used o wo b anches o expe imen s: expe imen s on ung ouped da a (all sec o s), and expe imen s on g ouped da a, whe e IS ca ego ical a iables we e used o g ouping. The i e old c oss- alida ion was used o c ea e a aining/ es ing old. The da ase we used in ou expe imen s was he ISBSG eposi o y Augus 2020 R1 [ 24 ]. In ou s udy, he c i e ia o da a il e ing we e as ollows: 1. We selec ed eco ds wi h he IFPUG coun ing app oach (including IFPUG Old and IFPUG 4+); 2. Only he eco ds whe e he da a quali y a ing is A o B has been selec ed; 3. The de elopmen ype was new de elopmen ; 4. The ows wi h an emp y alue o base unc ional componen s we e elimina ed; 5. Rows wi h emp y alues in he indus y sec o column we e emo ed; 6. The ows wi h emp y alues in no malised p oduc i i y deli e y a e and summa y wo k e o (SWE) we e also e ased; 7. We illed he VAF blank cells wi h he alues ob ained om Equa ion (3). Compu e s 2022,11, 15 8 o 20 Acco ding o Lich enbe g and ¸Sim¸sek [ 49 ], he numbe o eco ds in a da ase is la ge enough o be eligible o a gi en aining se o a ain he mos sa is ac o y esul s. M. Hammad [ 35 ] also p o ed ha some algo i hms lea n pe ec ly as he size o he aining se inc eases. In ou case, a e many es s and e alua ions, he esul s om ISs wi h o e 30 eco ds ga e he bes esul s. Fo he ISs ha did no sa is y his condi ion ( he numbe o eco ds is less han 30), we ga he ed hem in o a g oup named “O he s”. Figu e 2shows a his og am o he da ase a e being p ocessed. Compu e s 2022, 11, 15 8 o 20 5. Resea ch Me hodology In his sec ion, we p esen he esea ch me hodology. This includes desc ibing he da a o be used, along wi h he da a p ocessing and he expe imen al se up. In addi ion, e alua ion c i e ia a e in oduced he e. 5.1. Expe imen al Se up In his sec ion, we desc ibe he expe imen al p ocess, which is g aphically illus a ed in Figu e 1. In he da a p e-p ocessing phase (Figu e 1), da a il e ing and cleaning we e pe o med o c ea e he wo king da ase (see ollowing sec ion). This da ase was used o wo b anches o expe imen s: expe imen s on ung ouped da a (all sec o s), and expe imen s on g ouped da a, whe e IS ca ego ical a iables we e used o g ouping. The i e old c oss- alida ion was used o c ea e a aining/ es ing old. The da ase we used in ou expe imen s was he ISBSG eposi o y Augus 2020 R1 [24]. In ou s udy, he c i e ia o da a il e ing we e as ollows: 1. We selec ed eco ds wi h he IFPUG coun ing app oach (including IFPUG Old and IFPUG 4+); 2. Only he eco ds whe e he da a quali y a ing is A o B has been selec ed; 3. The de elopmen ype was new de elopmen ; 4. The ows wi h an emp y alue o base unc ional componen s we e elimina ed; 5. Rows wi h emp y alues in he indus y sec o column we e emo ed; 6. The ows wi h emp y alues in no malised p oduc i i y deli e y a e and summa y wo k e o (SWE) we e also e ased; 7. We illed he VAF blank cells wi h he alues ob ained om Equa ion (3). Acco ding o Lich enbe g and Şimşek [49], he numbe o eco ds in a da ase is la ge enough o be eligible o a gi en aining se o a ain he mos sa is ac o y esul s. M. Hammad [35] also p o ed ha some algo i hms lea n pe ec ly as he size o he aining se inc eases. In ou case, a e many es s and e alua ions, he esul s om ISs wi h o e 30 eco ds ga e he bes esul s. Fo he ISs ha did no sa is y his condi ion ( he numbe o eco ds is less han 30), we ga he ed hem in o a g oup named “O he s”. Figu e 2 shows a his og am o he da ase a e being p ocessed. Figu e 2. His og am o he p ocessed da ase . We no ice ha he e is a possibili y ha he da a may be noisy because some SWE alues a e oo a om he mean g oup. In his s udy, we used he in e qua ile ange (IQR) me hod [50,51] on hese ea u es o de e mine and emo e ou lie s, wi h he lowe bounda y being 0.15 and he uppe bounda y being 0.85. Figu es 3 and 4 show he desc ip ion o he da ase be o e and a e he emo al o ou lie s, espec i ely. Figu e 2. His og am o he p ocessed da ase . We no ice ha he e is a possibili y ha he da a may be noisy because some SWE alues a e oo a om he mean g oup. In his s udy, we used he in e qua ile ange (IQR) me hod [ 50 , 51 ] on hese ea u es o de e mine and emo e ou lie s, wi h he lowe bounda y being 0.15 and he uppe bounda y being 0.85. Figu es 3and 4show he desc ip ion o he da ase be o e and a e he emo al o ou lie s, espec i ely. Compu e s 2022, 11, 15 9 o 20 Figu e 3. Boxplo o he da ase be o e emo ing ou lie s. Figu e 4. Boxplo o he da ase a e emo ing ou lie s. The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks as ollows: 1. Bayesian idge eg ession (Sec ion 4.2) is employed; Figu e 3. Boxplo o he da ase be o e emo ing ou lie s. Compu e s 2022,11, 15 9 o 20 Compu e s 2022, 11, 15 9 o 20 Figu e 3. Boxplo o he da ase be o e emo ing ou lie s. Figu e 4. Boxplo o he da ase a e emo ing ou lie s. The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks as ollows: 1. Bayesian idge eg ession (Sec ion 4.2) is employed; Figu e 4. Boxplo o he da ase a e emo ing ou lie s. The calib a ion phase (Figu e 1) ep esen s he CFCW algo i hm. The CFCW wo ks as ollows: 1. Bayesian idge eg ession (Sec ion 4.2) is employed; 2. CFCW elici s he complexi y weigh s o he EI, EO, EQ, EIF, and ELF a iables using Bayesian idge eg ession; 3. The UFP is calcula ed by using a newly es ima ed complexi y weigh o each o he a iables (EI, EO, EQ, EIF, and ELF); 4. Es ima ed e o is ob ained by mul iplying UFP by he VAF and, inally, by mul iply- ing by PF. The op imisa ion phase (Figu e 1) ep esen s he CFCWO algo i hm, which wo ks as ollows: 1. E o om CFCW (calib a ion phase) is used as inpu ; 2. The o ing eg esso (Sec ion 4.3) algo i hm is employed; 3. Vo ing eg esso is an ensemble model, consis ing o ou es ima o s (Table 2); 4. CFCWO op imises es ima ed e o by CFCW by minimising he e o o SWE (know e o om da ase ). Table 2. Base es ima o s o he o ing ensemble model. Algo i hm Implemen a ion Pa ame e s Random o es s sklea n.ensemble.RandomFo es Reg esso n_es ima o s = 200, andom_s a e = 0 Bayesian idge sklea n.linea _model. BayesianRidge n_i e = 300 ANN sklea n.neu al_ne wo k. MLPReg esso ol = 0.00001, max_i e =10000, momen um = 0.000001 Lasso sklea n.linea _model. Lasso alpha = 0.01, selec ion = ‘ andom’, andom_s a e = 63 Compu e s 2022,11, 15 16 o 20 (se ice indus y); he mean pe cen age di e ences o he indi idual sec o s compa ed o he ung ouped da ase a e MAE = 49.98%, MAPE = 49.03%, and RMSE = 38.25%. Pai ed-samples - es s we e used o e alua ing s a is ical signi icance compa isons [58,59] o see whe he he CFCWO me hod is signi ican ly di e en om he o he me hods, in o de o con i m he e alua ion conclusions (see Table 10). The no a ions  ,  , and ≈ e- lec he s a is ical supe io i y, in e io i y, and simila i y o he CFCWO app oach compa ed o each o he o he me hods (FPA and NFFPCM), espec i ely. We can conclude ha he di e ence in es ima ing accu acy be ween he CFCWO and each al e na i e app oach is signi ican when he p- alue is less han 0.05. Table 10. The s a is ical - es s based on he inal e alua ion esul s. Pai s o Me hods CFCW VS. FPA CFCWO VS. CFCW CFCWO VS. NFFPCM MAE esul s Mean MAE 280.77 s. 362.4 249.67 s. 280.77 10.06 s. 611.12 Mean p- alue 0.00787 0.00013 0.00156 S a is ical conclusion >> >> >> MAPE esul s Mean MAPE 8.42 s. 10.06 7.61 s. 8.42 636.52 s. 17.41 Mean p- alue 0.00475 0.00124 0.00017 S a is ical conclusion >> >> >> RMSE esul s Mean SE 459.87 s. 636.52 403.42 s. 459.87 403.42 s. 769.26 Mean p- alue 0.00215 0.00287 0.00445 S a is ical conclusion >> >> >> All used e alua ion c i e ia esul s in his s udy we e used as he sample es se o each me hod in his s udy. 7. Th ea s o Validi y In e nal alidi y in his s udy, which can a ec he alidi y o conclusions d awn om expe imen al esea ch, is an inco ec /inaccu a e e alua ion me hod o assess he p oposed me hod; speci ically, i e e s o he echnique o s a is ical sample alida ion. The h ea o he alidi y was con olled using he k- old c oss- alida ion me hod, which gua an ees ha he p oposed me hod is accu a ely assessed. Ano he in e nal h ea ha may a ec he alidi y o he ob ained esul s is he choice o pa ame e s in he machine lea ning echnique. In his s udy, we use he de aul pa ame e se ings o he Bayesian idge eg esso echnique o he p oposed algo i hm. Ex e nal alidi y in his s udy is conce ned wi h he ange o alidi y o he esul s ob ained, and whe he he esul s ob ained could be applied in a di e en con ex . The ISBSG eposi o y Augus 2020 R1 da ase was used o assess he p edic i e abili y o he p oposed me hod. This da ase con ains many so wa e p ojec s collec ed om di e en o ganisa ions wo ldwide ha di e in e ms o ea u es, ields, size, and numbe o ea u es. Unbiased e alua ion c i e ia a e used o e alua e he pe o mance accu acy o he p oposed me hod. This s udy used e alua ion c i e ia such as he MAE, MAPE, and RMSE, which a e unbiased e alua ion c i e ia acco ding o p e ious esea ch [ 60 , 61 ]. The e o e, we can conclude ha he expe imen al esul s o his s udy a e highly gene alizable. 8. Conclusions and Fu u e Wo k A s anda d IFPUG FPA me hod calib a ion algo i hm based on he Bayesian idge eg esso model o calib a ion (CFCW) and he o ing eg esso model o op imising e o es ima ion (CFCWO) wi h and wi hou da ase g ouping is p esen ed in his s udy. This pape aimed o answe h ee esea ch ques ions: In answe o RQ1, we can see a pe cen age accu acy imp o emen wi h he p oposed CFCW algo i hm compa ed o he IFPUG FPA me hod, depending on he e alua ion c i e ia and whe he a g ouped o ung ouped Compu e s 2022,11, 15 17 o 20 da ase was used. Fo he ung ouped da ase , he pe cen age accu acy imp o emen o MAE = 5.46%, MAPE = 4.10%, and RMSE = 10.39%. The mean pe cen age di e ence o he indi idual sec o s compa ed o he ung ouped da ase was MAE = 22.52%, MAPE = 16.26%, and RMSE = 27.75%, showing an e en g ea e imp o emen in he accu acy o he es ima es. This demons a es ha he IFPUG FPA me hod needs calib a ion, and can be calib a ed. When CFCW is compa ed o NFFCMP, MAE = 15.39%, MAPE = 28.41%, and RMSE = 19.9% o all sec o s. The second p oposed algo i hm, CFCWO, b ings u he imp o emen , and ou pe - o ms he CFCW algo i hm, answe ing RQ2. The pe cen age imp o emen a ies acco ding o he e alua ion c i e ia and da ase . Fo he ung ouped da ase , he pe cen age accu acy imp o emen is MAE = 11.89%, MAPE = 6.56%, and RMSE = 24.62%. The mean pe cen age di e ence o he indi idual sec o s compa ed o he ung ouped da ase is MAE = 11.08%, MAPE = 9.59%, and RMSE = 12.28%. The esul s also show ha i makes sense o wo k wi h da a belonging o a speci ic g oup. In ou case, we g ouped he da a acco ding o he IS. The answe o RQ3 is ha he es ima e’s accu acy in all indi idual sec o s o all e alua ion c i e ia is highe han o an ung ouped da ase . The unc ional complexi y weigh alues e lec he mode n so wa e indus y end o imp o ing wo k pe o mance hanks o he de elopmen o compu e echnology, p og am- ming languages, and CASE ools. This mani es s i sel in unc ional complexi y weigh alues ha a e smalle han he o iginal alue. In addi ion, he demand o sophis ica ion and complexi y o so wa e unc ions also inc eases o e ime in ce ain a eas, mani es - ing in calib a ed unc ional complexi y weigh alues ha a e mo e signi ican han he o iginal alues. IFPUG FPA is a calcula ion me hod ha es ima es he size, cos , and e o in he ield o so wa e de elopmen ; i plays a signi ican ole in oday’s so wa e indus y. Howe e , so wa e enginee ing is a apidly e ol ing ield; oday’s ac ual alues may no accu a ely e lec omo ow’s so wa e alues. The e o e, he weigh s p oposed in his pape need o be upda ed acco ding o he new end. The ISBSG da ase is an up- o-da e da abase o companies a ound he globe; i e lec s he mode n so wa e indus y ha is cons an ly upda ed. The e o e, in he u u e, when p ojec da a a e upda ed, he IFPUG FPA weigh ing alues should be ecalib a ed o e lec he la es so wa e indus y ends. Au ho Con ibu ions: Concep ualiza ion, V.V.H., P.S. and H.L.T.K.N.; me hodology, V.V.H., R.S. and Z.P.; so wa e, V.V.H. and H.L.T.K.N.; alida ion, V.V.H., P.S. and H.L.T.K.N.; in es iga ion, V.V.H., H.L.T.K.N., R.S. and Z.P.; esou ces, P.S., Z.P. and R.S.; da a cu a ion, V.V.H., P.S. and H.L.T.K.N.; w i ing—o iginal d a p epa a ion, V.V.H., R.S. and Z.P.; w i ing— e iew and edi ing, V.V.H. and R.S.; isualiza ion, V.V.H., P.S. and H.L.T.K.N.; supe ision, R.S. and Z.P.; p ojec adminis a ion, P.S., R.S. and Z.P.; unding acquisi ion, P.S., R.S. and Z.P. All au ho s ha e ead and ag eed o he published e sion o he manusc ip . Funding: This esea ch was unded by he Facul y o Applied In o ma ics, Tomas Ba a Uni e si y in Zlin, unde P ojec No.: RVO/FAI/2021/002. Ins i u ional Re iew Boa d S a emen : No applicable. In o med Consen S a emen : No applicable. Da a A ailabili y S a emen : The ISBSG da a used o suppo he indings o his s udy may be eleased upon applica ion o he ISBSG, which can be con ac ed a [email p o ec ed]g o h p://isbsg. o g/academic-subsidy (accessed on 20 Sep embe 2021). Con lic s o In e es : The au ho s decla e no con lic o in e es . Compu e s 2022,11, 15 18 o 20 Re e ences 1. S andish G oup Repo . A ailable online: h ps://www.s andishg oup.com (accessed on 20 Sep embe 2021). 2. Ve a, T.; Ochoa, S.F.; Pe o ich, D. Su ey o So wa e De elopmen E o Es ima ion Taxonomies; Technical Repo ; Compu e Science Depa men , Uni e si y o Chile: San iago, Chile, 2017. 3. Khan, B.; Khan, W.; A shad, M.; Jan, N. So wa e cos es ima ion: Algo i hmic and non-algo i hmic app oaches. In . J. Da a Sci. Ad . Anal. 2020,2, 1–5. 4. Fa ia, P.; Mi anda, E. Expe judgmen in so wa e es ima ion du ing he bid phase o a p ojec —An explo a o y su ey. In P oceedings o he 2012 Join Con e ence o he 22nd In e na ional Wo kshop on So wa e Measu emen and he 2012 Se en h In e na ional Con e ence on So wa e P ocess and P oduc Measu emen , IEEE, Assisi, I aly, 17–19 Oc obe 2012; pp. 126–131. 5. Azzeh, M.; Nassi , A.B. Analogy-based e o es ima ion: A new me hod o disco e se o analogies om da ase cha ac e is ics. IET So w. 2015,9, 39–50. [C ossRe ] 6. Pu nam, L.H. A gene al empi ical solu ion o he mac o so wa e sizing and es ima ing p oblem. IEEE T ans. So w. Eng. 1978 ,4, 345–361. [C ossRe ] 7. Boehm, B. So wa e Enginee ing Economics; P en ice-Hall: Englewood Cli s, NJ, USA, 1981. 8. Alb ech , A.J. Measu ing applica ion de elopmen p oduc i i y. In P oceedings o he IBM Applica ions De elopmen Sympo- sium, Mon e ey, CA, USA, 14–17 Oc obe 1979; p. 83. 9. IFPUG. Func ion Poin Coun ing P ac ices; Manual, Release 4.3.1; In e na ional Func ion Poin Use s G oup: Wes e ille, OH, USA, 2010. 10. ISO/IEC 19761:2011; So wa e Enginee ing—COSMIC: A Func ional Size Measu emen Me hod. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2011. 11. ISO/IEC 29881:2010; In o ma ion Technology—Sys ems and So wa e Enginee ing—FiSMA 1.1 Func ional Size Measu emen Me hod. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2010. 12. ISO/IEC 20968:2002; So wa e Enginee ing—MK II Func ion Poin Analysis—Coun ing P ac ices Manual. In e na ional O ganiza- ion o S anda diza ion: Gene a, Swi ze land, 2002. 13. ISO/IEC 24570:2005; So wa e Enginee ing—NESMA Func ional Size Measu emen Me hod Ve sion 2.1—De ini ions and Coun ing Guidelines o he Applica ion o Func ion Poin Analysis. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2005. 14. Ki chenham, B.; Mendes, E. Why compa a i e e o p edic ion s udies may be in alid. In P oceedings o he 5 h In e na ional Con e ence on P edic o Models in So wa e Enginee ing—PROMISE’09, Vancou e , BC, Canada, 18 May 2009; ACM P ess: New Yo k, NY, USA, 2009; pp. 1–5. 15. Pe e , R.H. P ac ical So wa e P ojec Es ima ion; McG aw-Hill: New Yo k, NY, USA, 2011. 16. Hai, V.V.; Nhung, H.L.T.K.; Hoc, H.T. A e iew o so wa e e o es ima ion by using unc ional poin s analysis. In Compu a ional S a is ics and Ma hema ical Modeling Me hods in In elligen Sys ems; Silha y, R., Silha y, P., P okopo a, Z., Eds.; Sp inge : Cham, Swi ze land, 2019; pp. 408–422. 17. Al-Haj i, M.A.; Abdul Ghani, A.A.; Sulaiman, M.N.; Selama , M.H. Modi ica ion o s anda d Func ion Poin complexi y weigh s sys em. J. Sys . So w. 2005,74, 195–206. [C ossRe ] 18. Xia, W.; Cap e z, L.F.; Ho, D.; Ahmed, F. A new calib a ion o Func ion Poin complexi y weigh s. In . So w. Technol. 2008 ,50, 670–683. [C ossRe ] 19. Shukla, S.; Kuma , S. Applicabili y o neu al ne wo k based models o so wa e e o es ima ion. In P oceedings o he 2019 IEEE Wo ld Cong ess on Se ices (SERVICES), Milan, I aly, 8–13 July 2019; IEEE: Milan, I aly, 2019; pp. 339–342. 20. Shukla, S.; Kuma , S.; Bal, P.R. Analyzing e ec o ensemble models on mul i-laye pe cep on ne wo k o so wa e e o es ima ion. In P oceedings o he 2019 IEEE Wo ld Cong ess on Se ices (SERVICES), Milan, I aly, 8–13 July 2019; IEEE: Milan, I aly, 2019; pp. 386–387. 21. P iya, V.A.G.; Ani ha, K.; Va ada ajan, V. Es ima ing so wa e de elopmen e o s using a andom o es -based s acked ensemble app oach. Elec onics 2021,10, 1195. [C ossRe ] 22. Id i, A.; Hosni, M.; Ab an, A. Sys ema ic li e a u e e iew o ensemble e o es ima ion. J. Sys . So w. 2016 ,118, 151–175. [C ossRe ] 23. Id i, A.; Hosni, M.; Ab an, A. Sys ema ic mapping s udy o ensemble e o es ima ion. In P oceedings o he 11 h In e na ional Con e ence on E alua ion o No el So wa e App oaches o So wa e Enginee ing, Rome, I aly, 27–28 Ap il 2016; pp. 132–139. 24. In e na ional So wa e Benchma king S anda ds G oupm. ISBSG Reposi o y Augus 2020 R1. A ailable online: h ps://www.isbsg.o g (accessed on 20 Sep embe 2021). 25. Silha y, P.; Silha y, R.; P okopo a, Z. Ca ego ical a iable segmen a ion model o so wa e de elopmen e o es ima ion. IEEE Access 2019,7, 9618–9626. [C ossRe ] 26. Pospieszny, P.; Cza nacka-Ch obo , B.; Kobylinski, A. An e ec i e app oach o so wa e p ojec e o and du a ion es ima ion wi h machine lea ning algo i hms. J. Sys . So w. 2018,137, 184–196. [C ossRe ] 27. Jayakuma , K.R.; Ab an, A. Es ima ion models o so wa e unc ional es e o . J. So w. Eng. Appl. 2017 ,10, 338–353. [C ossRe ] 28. Wei, K.T.; Selama , M.H.; Ghani, A.A.A.; Abdullah, R. Exponen ial E o Es ima ion Model Using Unadjus ed Func ion Poin s. In P oceedings o he 5 h In e na ional Con e ence on New T ends in In o ma ion Science and Se ice Science, Macao, China, 24–26 Oc obe 2011; IEEE: Macao, China, 2011; pp. 111–115. 29. Mis a, S.; Adewumi, A.; Fe nandez-Sanz, L.; Damase icius, R. A Sui e o Objec O ien ed Cogni i e Complexi y Me ics. IEEE Access 2018,6, 8782–8796. [C ossRe ] Compu e s 2022,11, 15 19 o 20 30. Dewi, R.S.; Sub iadi, A.P.; Sholiq, S. A modi ica ion complexi y ac o in unc ion poin s me hod o so wa e cos es ima ion owa ds public se ice applica ion. P ocedia Compu . Sci. 2017,124, 415–422. [C ossRe ] 31. Leal, L.Q.; Fagundes, R.A.A.; de Souza, R.M.C.R.; Mou a, H.P.; Gusmao, C.M.G. Nea es -neighbo hood linea eg ession in an applica ion wi h so wa e e o es ima ion. In P oceedings o he 2009 IEEE In e na ional Con e ence on Sys ems, Man and Cybe ne ics, San An onio, TX, USA, 11–14 Oc obe 2009; pp. 5030–5034. 32. Hamza, H.; Kamel, A.; Shams, K. So wa e e o es ima ion using a i icial neu al ne wo ks: A su ey o he cu en p ac ices. In P oceedings o he 2013 10 h In e na ional Con e ence on In o ma ion Technology: New Gene a ions, Las Vegas, NV, USA, 15–17 Ap il 2013; IEEE: Las Vegas, NV, USA, 2013; pp. 731–733. 33. Lena duzzi, V.; Mo asca, S.; Taibi, D. Es ima ing so wa e de elopmen e o based on phases. In P oceedings o he 2014 40 h EUROMICRO Con e ence on So wa e Enginee ing and Ad anced Applica ions, Ve ona, I aly, 27–29 Augus 2014; IEEE: Ve ona, I aly, 2014; pp. 305–308. 34. P okopo a, Z.; Silha y, P.; Silha y, R. In luence analysis o selec ed ac o s in he unc ion poin wo k e o es ima ion. In In elligen Sys ems in Cybe ne ics and Au oma ion Con ol Theo y; Silha y, R., Silha y, P., P okopo a, Z., Eds.; Sp inge : Cham, Ge many, 2019; pp. 112–124. 35. Hammad, M.; Alqaddoumi, A. Fea u es-le el so wa e e o es ima ion using machine lea ning algo i hms. In P oceedings o he 2018 In e na ional Con e ence on Inno a ion and In elligence o In o ma ics, Compu ing, and Technologies (3ICT), Sakhie , Bah ain, 18–20 No embe 2018; IEEE: Sakhie , Bah ain, 2018; pp. 1–3. 36. Abdella i , T.M. A compa ison s udy be ween so compu ing and s a is ical eg ession echniques o so wa e e o es ima ion. In P oceedings o he 2018 IEEE Canadian Con e ence on Elec ical & Compu e Enginee ing (CCECE), Quebec, QC, Canada, 13–16 May 2018; IEEE: Quebec, QC, Canada, 2018; pp. 1–5. 37. IFPUG. Announcing he New Business Applica ions Commi ee. A ailable online: h ps://www.i pug.o g (accessed on 20 Sep embe 2021). 38. ISO/IEC 20926:2009; So wa e and Sys ems Enginee ing—So wa e Measu emen —IFPUG Func ional Size Measu emen Me hod. In e na ional O ganiza ion o S anda diza ion: Gene a, Swi ze land, 2009. 39. Azzeh, M.; Nassi , A.B. Analyzing he ela ionship be ween p ojec p oduc i i y and en i onmen ac o s in he use case poin s me hod. J. So w. E ol. P ocess 2017,29, e1882. [C ossRe ] 40. Azzeh, M.; Nassi , A.B.; Bani aan, S. Compa a i e analysis o so compu ing echniques o p edic ing so wa e e o based use case poin s. IET So w. 2018,12, 19–29. [C ossRe ] 41. MacKay, D.J.C. Bayesian in e pola ion. Neu al Compu . 1992,4, 415–447. [C ossRe ] 42. Michael, E.T. Spa se Bayesian lea ning and he ele ance ec o machine. J. Mach. Lea n. Res. 2001,1, 211–244. 43. Pa k, S.Y.; Be a, A.K. Maximum en opy au o eg essi e condi ional he e oskedas ici y model. J. Econom. 2009 ,150, 219–230. [C ossRe ] 44. Kocaguneli, E.; Menzies, T.; Keung, J.W. On he alue o ensemble e o es ima ion. IEEE T ans. So w. Eng. 2012 ,38, 1403–1416. [C ossRe ] 45. Kocaguneli, E.; Kul u , Y.; Bene , A. Combining mul iple lea ne s induced on mul iple da ase s o so wa e e o p edic ion. In . Symp. So w. Reliab. Eng. 2009,17, 25–49. 46. An, K.; Meng, J. Vo ing-a e aged combina ion me hod o eg esso ensemble. In Ad anced In elligen Compu ing Theo ies and Applica ions; Huang, D.S., Zhao, Z., Be ilacqua, V., Figue oa, J.C., Eds.; Sp inge : Be lin/Heidelbe g, Ge many, 2010; pp. 540–546. 47. Wi en, I.H.; F ank, E.; Hall, M.A.; Pal, C.J. Da a Mining: P ac ical Machine Lea ning Tools and Techniques; Mo gan Kau mann: Ams e dam, The Ne he lands, 2017. 48. Azzeh, M.; Nassi , A.B.; Minku, L.L. An empi ical e alua ion o ensemble adjus men me hods o analogy-based e o es ima ion. J. Sys . So w. 2015,103, 36–52. [C ossRe ] 49. Lich enbe g, J.M.; ¸Sim¸sek, Ö. Simple eg ession models. P oc. Mach. Lea n. 2016,58, 13–25. 50. Up on, G.; Cook, I. Unde s anding S a is ics; Ox o d Uni e si y P ess: Ox o d, UK, 1996. 51. Zwillinge , D.; Kokoska, S. CRC S anda d P obabili y and S a is ics Tables and Fo mulae; Chapman & Hall/CRC P ess: Boca Ra on, FL, USA, 2000. 52. Foss, T.; S ens ud, E.; Ki chenham, B.; My ei , I. A simula ion s udy o he model e alua ion c i e ion mm e. IEEE T ans. So w. Eng. 2003,29, 985–995. [C ossRe ] 53. Ki chenham, B.A.; Picka d, L.M.; MacDonell, S.G.; Sheppe d, M.J. Wha accu acy s a is ics eally measu e. IEE P oc. So w. 2001 , 148, 81–85. [C ossRe ] 54. Ha din, J.; Ha din, J.; Hilbe, J.; Hilbe, J. Gene alized Linea Models and Ex ensions; S a a P ess: College S a ion, TX, USA, 2007. 55. Gup a, H.V.; Kling, H.; Yilmaz, K.K.; Ma inez, G.F. Decomposi ion o he mean squa ed e o and NSE pe o mance c i e ia: Implica ions o imp o ing hyd ological modelling. J. Hyd ol. 2009,377, 80–91. [C ossRe ] 56. Sheppe d, M.; MacDonell, S. E alua ing p edic ion sys ems in so wa e p ojec es ima ion. In . So w. Technol. 2012 ,54, 820–827. [C ossRe ] 57. Kaise , M.; Ull ich, C. Es ima ion accu acy in la ge is p og ams insigh s om a desc ip i e case s udy. In P oceedings o he 22s Eu opean Con e ence on In o ma ion Sys ems, Tel A i , Is ael, 9–11 June 2014; pp. 1–14. 58. Ande son, D.R.; Sweeney, D.J.; William, T.A. S a is ics o Business and Economics, 14 h ed.; Thomson Sou h-Wes e n, Cengage Lea ning: Bos on, MA, USA, 2009. Compu e s 2022,11, 15 20 o 20 59. Ross, A.; Willson, V.L.. Pai ed Samples -Tes . In Basic and Ad anced S a is ical Tes s; SensePublishe s: Ro e dam, The Ne he lands, 2017; pp. 17–19. 60. Chai, T.; D axle , R.R. Roo mean squa e e o (RMSE) o mean absolu e e o (MAE)?—A gumen s agains a oiding RMSE in he li e a u e. Geosci. Model De . 2014,7, 1247–1250. [C ossRe ] 61. Tod os, K.; Tab ikian, J. On o de ela ions be ween lowe bounds on he MSE o unbiased es ima o s. In P oceedings o he 2010 IEEE In e na ional Symposium on In o ma ion Theo y, Aus in, TX, USA, 13–18 June 2010; IEEE: Aus in, TX, USA, 2010; pp. 1663–1667.