205
APPENDIX
D
PDSoS - Sys emic Mapping P o ocols
Ini ial Rema ks
Ou mapping was conduc ed in he con ex o so wa e de elopmen p ocesses o SoS. This
mapping was ca ied ou om Ap il/2015 o Oc obe /2015 and upda ed om Sep embe /2019
o Decembe /2018 and in ol ed six esea che s, among hem ou expe s. We used he guide-
lines es ablished by Ki chenham and Cha e s (2007) ha is di ided in h ee phases: planning,
conduc ing, and epo ing In he i s phase, he esea ch objec i es and he sys ema ic e iew
p o ocol a e de ined. In he second phase, he p ima y s udies a e iden i ying, selec ing, and
e alua ing second o he inclusion and exclusion c i e ia de ined in he planning phase. Fo each
selec ed s udies, da a a e ex ac ed and syn hesized. In he hi d phase, a inal epo is o ganized,
p esen ed and discussed. The nex sec ions a e p esen ed p esen hese h ee phases in de ails.
Phase I - Planning
In his phase, he SM p o ocol was es ablished, including he esea ch ques ions, sea ch s a egy,
selec ion c i e ia, quali y assessmen c i e ia and da a ex ac ion and syn hesis me hod. The SM
p o ocol was elabo a ed by he au ho and alida ed by he expe s.
Resea ch Ques ion
In o de o s uc u e and simpli y he esea ch ques ions de elopmen , we used he PICO model
(KITCHENHAM; CHARTERS,2007), which includes he conside a ion o ou elemen s:
P
opula ion,
I
n e en ion,
C
ompa ison and
O
u comes. In he Popula ion is desc ibed he pop-
ula ion g oup ha is obse ed by he in e en ion; In he In e en ion iden i ies wha will be
in es iga ed his SM; Compa ison is iden i ied wha will be used o compa e in he con ex o
he SM; and Ou comes a e he expec ed e ec s o he p oposed in e en ion. To his SM he
206 APPENDIX D. PDSoS - Sys emic Mapping P o ocols
Popula ion comp ises he de elope s and esea che s o SoS and sys ems enginee ing communi y.
In e en ion o ou SM is ela ed o he con ex o SoS de elopmen p ocesses. Compa ison i
no applicable. Las ly, in Ou comes cha ac e ize all p ima y s udies iden i ied in he li e a u e ha
epo SoS de elopmen p ocesses. Fu he mo e, we hope o iden i y SoS cha ac e is ics, ype
o empi ical e alua ion, echnical ac o s, ac i i ies and/o a i ac s add essed o men ioned by
he p ocesses, ac o s ha can in luence he de elopmen o he SoS, and ends and challenges
discussed by he s udies. The esea ch ques ions a e:
RQ-01
Ha e SoS been cons uc ed based on de elopmen p ocesses? Which a e hose p ocesses?
This RQ aims a iden i ying p ocesses, bu no excluding o he ini ia i es (e.g., me hods,
amewo ks, and models) p oposed o building SoS.
RQ-02 Which SoS cha ac e is ics ha e been conside ed in SoS de elopmen p ocesses?
We aim a in es iga e he main cha ac e is ics men ioned by he au ho s based on he i e
Maie ’s cha ac e is ics (MAIER,1998).
RQ-03
Which echnical ac o s and managemen ac i i ies we e add essed o men ioned in he
SoS de elopmen p ocesses?
This RQ, he objec i e is o iden i y he echnical ac o s (e.g., communica ion, cos depen-
dence, isk managemen ) ha we e used o in luenced o build o he SoS de elopmen
p ocesses.
RQ-04
Which main challenges and ends ha e been discussed by he au ho s o he p ima y
s udies?
This RQ aims a iden i ying he main challenges and ends o SoS de elopmen p ocesses
discussed by he au ho s o he p ima y s udies.
RQ-05 How ha e been es ablished he SoS de elopmen p ocesses?
The objec i e is o iden i y app oaches used o building SoS de elopmen p ocesses and
o e i y which echnologies help in he de elopmen hose p ocesses (e.g, SOA, MDA,
MDD).
RQ-06 Wha do ha e in luenced he de elopmen o he SoS?
This RQ aims a iden i y he ac o s ha in luence (e.g., cons i uen s a chi ec u e, se e al
li e cycle) he in e ope abili y in SoS de elopmen p ocesses.
Sea ch S a egy
In o de o de e mine he sea ch s a egy o selec ion o he p ima y s udies we choose ou
sea ch sou ces: publica ion da abase, ela ed wo ks, snowballing and expe opinion. The ou
sea ch sou ces a e p esen ed as ollows:
207
∙Publica ion da abase:
Fo iden i ying p ima y s udies is necessa y o de ine which publica ion da abase will be
adop ed and he sea ch s ing ha will be used on hese publica ion da abases. A hese
SM, we adop ed he publica ion da abase ACM Digi al Lib a y, IEEE Xplo e, Science
Di ec , Scopus, and ISI Web o Science. They we e chosen o be ecognized and widely
used in So wa e Enginee ing, as example: (PERTENSEN e al.,2015) (KITCHENHAM
e al.,2010) (MADEYSKI e al.,2014) (MUNIR e al.,2014) (SILVA e al.,2015c)
(MACHADO e al.,2013) (NARCISO e al.,2014) (GUESSI e al.,2015).
To de ine he sea ch s ing, i e s eps we e aken: (i) sea ch by synonyms o “de elopmen
p ocess” in sys ema ic li e a u e e iews o adi ional so wa e de elopmen p ocesses;
(ii) sea ch in dic iona ies; (iii) consul ing expe s; (i ) pilo sea ch; and ( ) use o con ol
s udies (CALINESCU; KWIATKOWSKA,2008a) (CARBON e al.,2008a) (FARROHA;
FARROHA,2011a) (RAMOS e al.,2012a).
The pilo esea ch showed ha he e m “de elopmen echnique” and he plu al o he
synonyms o “de elopmen p ocess” p esen in he sea ch s ing no in luenced he o al
numbe o p ima y s udies e u ned by he engine sea ch. Fu he mo e, i was no ed ha
50% o con ol s udies we e no e u ning a hese sea ch, which could be a ailu e o he
sea ch s ing. An analysis and modi ica ions we e done oge he wi h expe s achie ing a
highe e u n han 231%
1
o p ima y s udies when he sea ch s ing e u ned all con ol
s udies.
The sea ch s ing was designed o co e a ia ions and synonyms o e ms ela ed o
“de elopmen p ocess” and “sys ems-o -sys ems” and adap ed o mee ing pa icula i ies
o each engine sea ch. The sea ch s ing used in ou SM is p esen ed in Table 23. The i s
column ep esen s he esea ch a eas and he second column he co esponding e ms.
Table 23 – Sea ch s ing
A ea Te ms
De elopmen P ocess
(“De elopmen Me hod” OR “De elopmen App oach”
OR “De elopmen Design” OR “Design Flow” OR
“De elopmen P ocedu e” OR “De elopmen Li e Cy-
cle” OR “Agile De elopmen ” OR “So wa e P oduc
Line”)
AND
Sys ems-o -Sys ems
(“Sys em o Sys ems” OR “Sys em-o -Sys ems” OR
“Sys ems-o -Sys ems” OR “Sys ems o Sys ems”)
∙ ela ed wo ks:
1
he sea ch s ing o he pilo esea ch e u ned 183 p ima y s udies, a e he modi ica ions he sea ch
s ing e u ned he o al o 423 ones.
208 APPENDIX D. PDSoS - Sys emic Mapping P o ocols
I was also conside ed he s udies ci ed as ela ed wo ks in he ele an p ima y s udies
ound in he engine sea ch;
∙snowballing:
To inc ease he accu acy o ou SM was used he snowballing. The snowballing e e s o
using he e e ence lis o a pape o he ci a ions o he pape o iden i y addi ional pape s
(WOHLIN,2014a). I using he e e ence is called o backwa d snowballing, on he o he
hand, using he ci a ion is known as o wa d snowballing. The s eps ollowing in ou SM
o snowballing p ocedu e is desc ibed in (WOHLIN,2014a).
∙expe opinion:
S udies sugges ed by expe s in SoS domain also was conside ed. Al hough he indica ion
o s udies by expe can be conside ed as bias, we ha e adop ed his sou ce aiming o no
lose any impo an e idence.
Selec ion c i e ia
In o de o acco dingly selec he s udies o answe ou esea ch ques ions, we es ablished he
Inclusion (IC) and Exclusion C i e ia (EC), which i was used in he i s and second selec ion.
In he i s selec ion, he applica ion o he c i e ia we e limi ed o p ima y s udies me ada a
(i.e., i le, abs ac , and keywo ds). A he second selec ion, he c i e ia we e also conside ed in
eading ull s udy. In he case o any disco dance in he i s selec ion, he s udies we e included
and analyzing in he second selec ion. In he second selec ion, in case o disco dance, he s udy
was discussed oge he wi h expe s un il eaching an ag eemen . The IC and EC a e:
Inclusion C i e ia
IC.1: The s udy discusses SoS de elopmen p ocess.
IC.2:
The s udy add esses SoS cha ac e is ics, p oblems o ac i i ies ela ed o SoS de elopmen
p ocesses.
Exclusion c i e ia
EC.1: The s udy is no ela ed o SoS.
EC.2: The s udy does no discuss any SoS de elopmen p ocess.
EC.3: De elopmen p ocess is no he main ocus o he s udy.
EC.4: The s udy is an edi o ial, keyno e, opinion, u o ial, pos e o panel.
EC.5:
The s udy is duplica ed o he e is newe o a mo e comple e one abou he same esea ch.
EC.6: The s udy compiles esul s o o he s udies.
EC.7: The s udy is no w i en in English.
EC.8: The ull ex o he s udy is no a ailable.
209
Quali y Assessmen C i e ia
In o de o e alua e he p ima y s udies we e adap ed he c i e ia desc ibed in (KHAN e al.,
2001) (KITCHENHAM; CHARTERS,2007) (KITCHENHAM e al.,2010). The quali y c i e ia
and he speci ica ion sco es o he SM a e shown in Table 24. The possible sco e depends o
each quali y c i e ia. When he p ima y s udy ully ma ches a quali y c i e ion, i had 2 as sco e
and 0 o he wise. In some cases, he quali y c i e ion is pa ially me by he p ima y s udies and
in hose case, in e media e sco es we e de ined, which can ange om 0.5 o 1.5.
The inal sco e o he quali y assessmen (QA) o each p ima y s udy was calcula ed
using he ollowing o mula:
QA =
i=1
∑
8CQ_i
maximum sco e *100
Table 24 – Quali y assessmen c i e ia
Sou ce: Adap ed om Ki chenham e al. (2010).
Quali y C i e ia (QC) Sco e
QC1: Do he au ho s clea ly s a e he aims o he esea ch?
0 - No
1 - Pa ially
2 - To ally clea
QC2: Do he au ho s discuss he limi a ions o hei s udy?
0 - No
1 - Pa ially
2 - Clea ly discussed
QC3: Do he au ho s s a e he indings clea ly ?
0 - No
1 - Clea ly pa ially
2 - To ally clea
QC4: Is he e e idence ha he indings o he s udy can be used by
o he esea che s/p ac i ione s?
0 - No
1 - Pa ially eusable
2 - To ally eusable
QC5: Does he s udy body ully mee issues p o ided in i s abs ac ?
0 - No
1 - A end pa ially
2 - A end comple ely
QC6: Do he desc ibed me hods we e empi ically e alua ed?
0 - No
1 - Pa ially e alua ed
2 - To ally e alua ed
QC7: How he empi ical e alua ion conduc ed?
0 - No
0.5 - Illus a i e example
1 - Case s udy
1.5 - Resea che s/Expe s
2 - Use in indus y
QC8: A e he e any ideas o u he in es iga ion p esen ed? 0 - No
2 - Yes
Da a Ex ac ion and Syn hesis Me hod
The da a ex ac ion is designed o ga he he in o ma ion necessa y o answe he esea ch
ques ions and e alua e he quali y o he s udy (MAGDALENO e al.,2012). Thus, o each
210 APPENDIX D. PDSoS - Sys emic Mapping P o ocols
p ima y s udy included, da a we e ex ac ed and discussed wi h he expe s. To ex ac ion o he
da a we used he S a Tool
2
(FABBRI e al.,2012), Mic oso o ice Excel 2013 and conside ed
h ee ca ego ies o da a: (i) cha ac e iza ion o s udies (e.g., au ho , yea , and publica ion ehicle);
(ii) in as uc u e used in de elopmen o app oaches (e.g., echnologies, ools, and languages);
and (iii) gene al ea u es o he s udies (e.g., SoS cha ac e is ics, echnical ac o s, and p ocesses)
ha enable us eco d de ails o he s udies, o ganizing he syn hesis in able, g aphic o g aphs.
To sha e he ac i i ies o he esea ch p o ocol and he s eps ollowed by he e iewe s, i
becomes possible o audi s and acili a es he p ocess o a new conduc ion, add essing issues
ela ed.
2<h p://lapes.dc.u sca .b / ools/s a ool>
