Ci a ion: Alonso, J.; O ue-Eche a ia,
L.; Hua e, M. CloudOps: Towa ds
he Ope a ionaliza ion o he Cloud
Con inuum: Concep s, Challenges
and a Re e ence F amewo k. Appl.
Sci. 2022,12, 4347. h ps://
doi.o g/10.3390/app12094347
Academic Edi o : Eui-Nam Huh
Recei ed: 25 Feb ua y 2022
Accep ed: 23 Ap il 2022
Published: 25 Ap il 2022
Publishe ’s No e: MDPI s ays neu al
wi h ega d o ju isdic ional claims in
published maps and ins i u ional a il-
ia ions.
Copy igh : © 2022 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license (h ps://
c ea i ecommons.o g/licenses/by/
4.0/).
applied
sciences
A icle
CloudOps: Towa ds he Ope a ionaliza ion o he Cloud
Con inuum: Concep s, Challenges and a Re e ence F amewo k
Juncal Alonso 1,* , Lei e O ue-Eche a ia 1and Maide Hua e 2
1TECNALIA, Basque Resea ch and Technology Alliance (BRTA), Pa que Cien í ico y Tecnológico de Bizkaia,
48160 De io, Spain; lei e.o [email p o ec ed]
2Depa men o Communica ions Enginee ing, Facul y o Enginee ing, Uni e si y o he Basque Coun y
UPV/EHU, Alda. U quijo S/N, 48013 Bilbao, Spain; maide [email p o ec ed]
*Co espondence: [email p o ec ed]
Abs ac :
The cu en end o de eloping highly dis ibu ed, con ex awa e, he e ogeneous com-
pu ing in ense and da a-sensi i e applica ions is changing he bounda ies o cloud compu ing.
Encou aged by he g owing IoT pa adigm and wi h lexible edge de ices a ailable, an ecosys em
o a combina ion o esou ces, anging om high densi y compu e and s o age o e y ligh weigh
embedded compu e s unning on ba e ies o sola powe , is a ailable o De Ops eams om wha is
known as he Cloud Con inuum. In his dynamic con ex , manageabili y is key, as well as con olled
ope a ions and esou ces moni o ing o handling anomalies. Un o una ely, he ope a ion and
managemen o such he e ogeneous compu ing en i onmen s (including edge, cloud and ne wo k
se ices) is complex and ope a o s ace challenges such as he con inuous op imiza ion and au-
onomous ( e-)deploymen o con ex -awa e s a eless and s a e ul applica ions whe e, howe e , hey
mus ensu e se ice con inui y while an icipa ing po en ial ailu es in he unde lying in as uc u e.
In his pape , we p opose a no el CloudOps wo k low (ex ending he adi ional De Ops pipeline),
p oposing echniques and me hods o applica ions’ ope a o s o ully emb ace he possibili ies o he
Cloud Con inuum. Ou app oach will suppo De Ops eams in he ope a ionaliza ion o he Cloud
Con inuum. Secondly, we p o ide an ex ensi e explana ion o he scope, possibili ies and u u e o
he CloudOps.
Keywo ds: De Ops; cloud con inuum; deploymen ; mul i-cloud; hyb id-cloud; sel -healing
1. In oduc ion
Cloud Compu ing e olu ion in he las decade and i s ans o ma ion in o a se ice
u ili y has p omo ed a wide adop ion by he indus y o applica ions in gene al o s o e
and p ocess da a. Wi h he expansion o he IoT pa adigm [
1
], he need o compu a ional
and s o age se ices is expec ed o g ow in he nex ew yea s, as well as he amoun o
da a gene a ed a he edge o he ne wo k. While cloud compu ing has been an e ec i e
way o acqui ing compu a ion and s o age as a se ice o many applica ions, i may no be
sui able o handle he endless da a om IoT de ices and ul il he la gely he e ogeneous
applica ion equi emen s [
2
]. To his ex en , some o he limi a ions o he adi ional Cloud
Pa adigm pa icula ly applies o applica ions ha need a eal ime esponse, low la encies
o hose gi ing suppo o c i ical in as uc u es. The cen alized na u e o adi ional
cloud se ices poses some limi a ions implying communica ion and da a ans e s o
a e se mul iple hops, which in oduces delays and consumes ne wo k bandwid h o
edge and co e ne wo ks [
3
]. Compu ing capaci y a he edge inc eased wi h he ha dwa e
e olu ion o pe sonal de ices and, as a esul , p oposed he u iliza ion o edge de ices o
un applica ions and s o e da a, b inging a new playe : Edge Compu ing.
As a esul , new app oaches ha e ec i ely le e age dis ibu ed compu a ional and
s o age in as uc u e and se ices a e necessa y. These app oaches mus seamlessly
Appl. Sci. 2022,12, 4347. h ps://doi.o g/10.3390/app12094347 h ps://www.mdpi.com/jou nal/applsci
Appl. Sci. 2022,12, 4347 2 o 24
combine esou ces and se ices a he edge (edge compu ing), in he co e (cloud compu ing)
and along he da a pa h ( og compu ing) as needed, h ough he Cloud Con inuum.
Thus, o applica ion de elope s and ope a o s o ully emb ace his new pa adigm,
speci ic and in-dep h knowledge on he ple ho a o unde lying echniques and echnologies
is needed. Fu he mo e, he ope a ion o such he e ogeneous compu ing en i onmen s
poses complex asks o he ope a o s o he applica ions as hey mus con igu e, plan,
p epa e and execu e hem, acing new challenges in all s ages o he ope a ion phase o
he applica ion:
•Op imiza ion in esou ce alloca ion
, which becomes mo e challenging as he numbe
o a iables (e.g., he e ogeneous esou ces) inc ease as well as hei dynamics (e.g.,
hese a iables change mo e o en o e ime). The composi ion o he in as uc-
u al se ices in he Cloud Con inuum b ings new a iables as he he e ogenei y o
se ices and applica ions’ componen s each g ound-b eaking le els. This dynamic
na u e o he sys em, along wi h high le els o he e ogenei y, c ea es a demand o
dynamic, mul i-c i e ia esou ce alloca ion s a egies ha can cope wi h he cons an ly
changing en i onmen ;
•Mic ose ices managemen
h oughou he Cloud Con inuum s ack p esen s chal-
lenges associa ed o he mo emen o se ices among he di e en le els (senso s, edge
and cloud compu ing). The au oma ic adap a ion o he execu ion o mic ose ices
mus conside deploymen loca ion and con ex , bu should also no neglec esou ce
cons ain s ha may exis a each le el o he Cloud Con inuum. To achie e his
au oma ic and anspa en adap a ion, se ices’ econ igu a ion ha conside s quali y
o se ice equi emen s should be conside ed;
•The he e ogenei y o ne wo ks
ac oss he Cloud Con inuum ecosys em in ega d
o mic ose ices’ deploymen and econ igu a ion is also challenging. S andalone
se ices can ha e ne wo k equi emen s o he da a sou ces, which can be achie ed
h ough ne wo k echnologies such as ne wo k i ualiza ion and so wa e de ined
ne wo ks (SDN). In his case, he need o a econ igu a ion o se ices includes a
econ igu a ion o he ne wo k in o de o ensu e equi emen s’ consis ency. On
he o he hand, he composi ion o se ices wi h di e en equi emen s can also be
enac ed e ically in he hie a chy ( om he ne wo k o he cloud h ough he edge),
whe e a econ igu a ion o se ices (and ne wo k, i necessa y) is e en mo e complex
due o he he e ogenei y o se ices in e ms o compu ing needs and equi emen s
(e.g., la ency). Ne wo k opology is expec ed o cons an ly change along wi h de ice
mobili y and a iable applica ion equi emen s, in oducing a mo e dynamic beha io
in he sys em;
•Da a managemen and po abili y
a un ime in ol es he design and deploymen
o policies, a chi ec u es and p ocedu es, allowing he accu a e managemen o he
ull da a li ecycle, including s a egies o placemen and accessing i (e.g., measu ing
and quan i ying he ade-o be ween placing da a and se ices a he cloud o edge
le el, e c.) as well as pe o ming he app op ia e ac ions when da a need o be po ed
o assu e se ice con inui y. E en i con aine s-based echnologies can suppo and
ease he po abili y o s a eless componen s, complexi y ises when add essing s a e ul
componen s, whe e in eg i y needs o be main ained du ing he po ing p ocess while,
a he same ime, ensu ing business con inui y;
•
Applying
ede a ion concep s in he Cloud and in he Edge
: The pe asi eness o
he echnologies in oduces a need o manage sha ed esou ces in a mo e in elligen ,
comp ehensi e manne ha is less ad hoc. This has gi en ise o he concep o
ede a ion [
4
]. The Cloud Con inuum will ul ima ely need some ype o ede a ion
o manage how di e en se s o da a p oduce s (e.g., senso s, edge nodes) and da a
consume s (e.g., applica ion componen s unning on adi ional cloud esou ces) can
collabo a e and sha e da a. This will also imply he ede a ion o he esou ces whe e
hese di e en elemen s a e unning;
Appl. Sci. 2022,12, 4347 3 o 24
•Deploymen and go e nance
p ope ies will ake on addi ional dimensions when
conside ing he e ogeneous dis ibu ed en i onmen s. While senso s and IoT de ices
will ypically no ha e any ex a capaci y o hos ing he ede a ion p ope ies, edge
nodes could hos hese unc ions, depending on hei ac ual capaci y and he numbe
o senso s associa ed wi h hem. Scalabili y is an issue as an edge node will ha e a
ini e capaci y ha will de ine how many IoT de ices i can manage, and how i can
manage he sha ing o da a wi h ex e nal da a consume s;
•Secu i y Assu ance in Cloud/Edge/IoT:
Secu i y Teams and Ope a o s in he Cloud
Con inuum a e challenged oday wi h he ask o p o iding and consuming se ices
in a secu e way, bu o en lack he p ope ools o quickly iden i y, choose and com-
pose he mos sui able se o echnologies and pla o ms ha o e he ideal ade-o
be ween unc ionali y and secu i y o p i acy gua an ees. Addi ionally, hey a e in
a scena io whe e da a o se ices migh be mig a ed be ween IoT, Edge and Cloud
when secu i y implica ions (secu i y, p i acy, physical ampe ing o legal policies) a e
ele an and need o be add essed. Due o he peculia ea u es o edge compu ing
a chi ec u es (i.e., he e ogenei y, dis ibu ed a chi ec u e, massi e da a p ocessing,
loca ion-awa eness and ola ili y), he adi ional da a secu i y and p i acy-p ese ing
mechanisms in cloud compu ing a e no longe sui able. In pa icula , da a secu i y
and p i acy-p ese ing in edge compu ing ha e o be ligh weigh , ine-g ained and
dis ibu ed [5].
Key Takeaway:
Applica ion de elope s and ope a o s in he Cloud Con inuum (CloudOps)
ace oday he challenge o emb acing he new pa adigm bu o en lack he p ope ools and mech-
anisms o con igu e, plan, p epa e and execu e hese he e ogeneous compu a ional en i onmen s.
Mo eo e , hey a e aced wi h he asks o con inuous op imiza ion and au onomous ( e-)deploymen
o complex con ex -awa e s a eless and s a e ul applica ions and da a in a ede a ed en i onmen
(including edge, cloud and ne wo k se ices) assu ing se ice con inui y and an icipa ing po en ial
ailu es in he unde lying in as uc u e, especially in c i ical sys ems ha mus be esilien and
whose esponse ime becomes i al.
The es o he pape is s uc u ed in ou main sec ions. Sec ion 2is dedica ed o p e-
sen ing he equi emen s and challenges in he ope a ionaliza ion o complex applica ions
in he Cloud Con inuum h ough he analysis o he ela ed wo k and, in pa icula , in
he ields ha a e mos ele an o he a icle: benchma king o in as uc u al esou ces
and applica ion classi ica ion and p o iling; deploymen o ches a ion and op imiza ion;
sel -lea ning h ough moni o ing in hyb id en i onmen s and sel -healing mechanisms
o co ec i e ac ions and da a po abili y s a egies. Sec ion 3desc ibes he CloudOps
concep and wo k low and p esen s he CloudOps e e ence amewo k. The main compo-
nen s o he p oposed solu ion a e de ined, namely, applica ion componen s classi ica ion
and in as uc u al equi emen s speci ica ion, esou ce disco e y, op imized deploymen
con igu a ion, deploymen and sel -healing, con inuous moni o ing and sel -lea ning. An
o e iew o how each o hem wo ks is p o ided. Sec ion 4desc ibes he applicabili y o
he solu ion and ela es he ad an ages b ough o speci ic applica ion domains. Finally,
Sec ion 5p o ides he conclusions, a gene al o e iew o he esea ch and u u e wo k.
2. Requi emen s and Challenges in he Ope a ionaliza ion o he Cloud Con inuum
Deli e ing inno a i e so wa e as e gi es companies a compe i i e ad an age. So -
wa e deli e y will con inue inc easing in he upcoming yea s, led by he gian s such as Ama-
zon who allegedly deploy new code e e y 11.7 s [
6
]. Mic ose ices applica ions, con aine s,
cloud in as uc u e and edge compu ing, De Ops (Con inuous In eg a ion/Con inuous
De elopmen ), a i icial in elligence d i en so wa e de elopmen and cloud agnos icism
and cybe secu i y ha e been iden i ied among he ends in he cu en yea s [
7
–
10
]. Achie -
ing excellence in con inuous deli e y and con inuous deploymen is one o he main easons
o so wa e companies when pu ing in place a De Ops s a egy. De Ops equi es agile
wo k p ocesses and au oma ed wo k lows which can only be achie ed h ough he assu -
Appl. Sci. 2022,12, 4347 4 o 24
ance o eadily a ailable IT in as uc u e which is needed o con inuously un and deploy
he de eloped code. This can only happen wi hin an au oma ed wo k low.
Among he challenges (Figu e 1) epo ed when deploying edge applica ions, he
ollowing s and ou [
11
,
12
]: (1) managemen o a la ge numbe o endpoin s, ha can be
sol ed wi h an au oma ed app oach a ope a ion ime; (2) lack o skills o knowledge o
pe o m said au oma ion; (3) deploymen s a egy: deciding which componen s go on he
edge and which ones go in he cen alized node, usually, he cloud, o ha e an op imized
applica ion in e ms o he non- unc ional equi emen s (e.g., la ency, pe o mance) and
how he deploymen is ac ually made, which needs o be simple ; (4) eco e y when he e is
a ailu e should also be easie and mo e s aigh o wa d; (5) moni o ing esou ce u iliza ion
ac oss all nodes simul aneously o unde s and he o e all Quali y o Se ice (QoS) o
he applica ion; (6) o ches a ion ools ha manage and coo dina e many edge si es and
wo kloads, e en ually de eloping in o a sel -managed edge applica ion; (7) ools o manage
cloud and edge applica ion li e cycles, including: “ he de ini ion o ad anced placemen
cons ain s in o de o cope wi h la ency equi emen s o applica ion componen s, he
p o isioning/scheduling o applica ions in o de o sa is y placemen equi emen s (ini ial
placemen ), and da a and wo kload eloca ions acco ding o in e nal/ex e nal e en s
(mobili y use-cases, ailu es, pe o mance conside a ions, and so o h)”.
Appl. Sci. 2022, 12, x FOR PEER REVIEW 4 o 24
De Ops equi es agile wo k p ocesses and au oma ed wo k lows which can only be
achie ed h ough he assu ance o eadily a ailable IT in as uc u e which is needed o
con inuously un and deploy he de eloped code. This can only happen wi hin an au o-
ma ed wo k low.
Among he challenges (Figu e 1) epo ed when deploying edge applica ions, he ol-
lowing s and ou [11,12]: (1) managemen o a la ge numbe o endpoin s, ha can be
sol ed wi h an au oma ed app oach a ope a ion ime; (2) lack o skills o knowledge o
pe o m said au oma ion; (3) deploymen s a egy: deciding which componen s go on he
edge and which ones go in he cen alized node, usually, he cloud, o ha e an op imized
applica ion in e ms o he non- unc ional equi emen s (e.g., la ency, pe o mance) and
how he deploymen is ac ually made, which needs o be simple ; (4) eco e y when he e
is a ailu e should also be easie and mo e s aigh o wa d; (5) moni o ing esou ce u ili-
za ion ac oss all nodes simul aneously o unde s and he o e all Quali y o Se ice (QoS)
o he applica ion; (6) o ches a ion ools ha manage and coo dina e many edge si es and
wo kloads, e en ually de eloping in o a sel -managed edge applica ion; (7) ools o man-
age cloud and edge applica ion li e cycles, including: “ he de ini ion o ad anced place-
men cons ain s in o de o cope wi h la ency equi emen s o applica ion componen s,
he p o isioning/scheduling o applica ions in o de o sa is y placemen equi emen s
(ini ial placemen ), and da a and wo kload eloca ions acco ding o in e nal/ex e nal
e en s (mobili y use-cases, ailu es, pe o mance conside a ions, and so o h)”.
Figu e 1. Challenges in he Ope a ionaliza ion (Ops) o applica ions in he Cloud Con inuum.
The analysis o he ela ed wo k in he opics ele an o he main challenges (Table
1) p esen ed is de ailed nex .
Table 1. Cu en unsol ed challenges in he Ope a ionaliza ion o applica ions in he Cloud Con-
inuum.
Topic Cu en Challenges
(De )Ops
• Pa ial au oma ion o he De Ops cycle especially in he e ogenous
en i onmen s
• Con ex (cloud, edge, ne wo k) speci ic ools
• Speci ic skills needed, especially o “combined” en i onmen s,
wi h cloud se ices, edge nodes and ne wo k elemen s
Moni o ing
• Ine iciencies mainly due o lack o dynamici y o he dis ibu ed
p oposed models
Figu e 1. Challenges in he Ope a ionaliza ion (Ops) o applica ions in he Cloud Con inuum.
The analysis o he ela ed wo k in he opics ele an o he main challenges (Table 1)
p esen ed is de ailed nex .
2.1. Benchma k o Resou ces and Applica ion Classi ica ion
Gi en he he e ogenei y o he cloud esou ces and he ypes and complexi y o he
applica ions i is no i ial o selec he op imal esou ces (o combina ion o esou ces)
o each applica ion (o applica ion componen ) in o de o maximize i s pe o mance [
13
],
o o choose he bes pe o mance/cos ade-o [
14
]. Tha is why benchma king is a
widely used echnique in cloud compu ing. Mo eo e , when shi ing om he cloud o
he edge compu ing, benchma king echniques a e seen as e en mo e aluable because
o he high di e si y and a iabili y o edge nodes in e ms o ha dwa e, so wa e s acks,
ene gy consump ion o connec i i y capabili ies. A common p ac ice used in he cloud [
15
]
is o ely on specialized benchma king ools [
16
–
18
] o simula e how an applica ion would
beha e when deployed on di e en cloud op ions. The ex ension o hese echniques o
he edge, howe e , s ill poses some new challenges [
19
] such as he limi ed capabili ies o
edge nodes needing mo e ligh weigh benchma king ools, speci ic wo kloads unning
in he edge (e.g., i ual eali y, image p ocessing) being di e en om he wo kloads in
he cloud (e.g., da a analysis, sea ching, ideo s eaming), and benchma king composi ion
Appl. Sci. 2022,12, 4347 5 o 24
(cloud + edge nodes)
in o de o p o ide in o ma ion on complex deploymen op ions.
These challenges a e cu en ly being ackled by di e en ini ia i es h ough di e en ongo-
ing wo ks: cha ac e iza ion o he mos common edge use cases and hei wo kloads [
20
],
selec ion o me ics o de ine hei pe o mance [
21
] and he implemen a ion o new sui es
o benchma king ools designed o un in he edge o di e en use cases such as speech
decoding and ace ecogni ion [
22
], su eillance came as and au onomous ehicles [
23
], e c.
A big challenge oday in exis ing se ice ca alogues is ha , in pa icula , he in o ma ion
abou secu i y a ibu es, such as ce i ica es o compliance o s anda ds, is no a ailable.
In [
24
], a solu ion is p oposed ha allows he disco e y and moni o ing o secu i y- ela ed
aspec s on wha i is called he Legal Le el. This Legal Le el assesses he egula ion and
legisla ion suppo ed by he di e en Cloud Se ices h ough he e i ica ion agains e i-
dences included in CSPs con ac o ms. Ne e heless, new me ics and aspec s ela ed o
p i acy and secu i y and o he in as uc u al elemen s (i.e., edge nodes, IoT agen s) need
o be add essed.
Table 1.
Cu en unsol ed challenges in he Ope a ionaliza ion o applica ions in he Cloud Con inuum.
Topic Cu en Challenges
(De )Ops
•Pa ial au oma ion o he De Ops cycle especially in he e ogenous
en i onmen s
•Con ex (cloud, edge, ne wo k) speci ic ools
•Speci ic skills needed, especially o “combined” en i onmen s,
wi h cloud se ices, edge nodes and ne wo k elemen s
Moni o ing
•Ine iciencies mainly due o lack o dynamici y o he dis ibu ed
p oposed models
•Lack o holis ic app oaches co e ing no only he moni o ing o
adi ional cloud esou ces bu also he in eg a ed moni o ing o
edge nodes and ne wo k communica ions
•Communica ion channels be ween applica ion elemen s (e.g.,
mic ose ices) a e usually conside ed a de ice le el bu no a
so wa e componen s le el (e en inside he same de ice)
•Lacks he p ecision o speed needed o high pe o mance
a chi ec u es as exis ing solu ions o o ecas ing me hods a e based
on basic o line algo i hms such as linea eg ession o nea es
neighbo s adap ed o online en i onmen s
Benchma king
•Benchma king o edge esou ces is incipien and exis ing solu ions
a e pla o m o scena io speci ic
•Exis ing ools usually in ol e ine icien o ine ec i e es s as he
comple e benchma king p ocess is no au oma ed
•Cu en app oaches o applica ion classi ica ion a e usually
ocused on speci ic applica ions ypes (e.g., da a s eaming
applica ions) and no sui able o o he ypes
•The cha ac e iza ion and benchma king o ne wo king and
communica ion ha e been ba ely add essed
Sel -lea ning,
sel -healing
•Few a ailable solu ions ackle he au oma ic sel -healing o
applica ions based on ailu e p edic ion
•Exis ing sel -healing wo ks do no conside s a e ul componen s’
needs (e.g., da a po abili y issues)
•Cu en app oaches ocus on ce ain aspec s o he sel -healing
p ocess, usually he e-deploymen o he applica ion, bu do no
o e in eg a ed amewo ks co e ing p e ious o subsequen asks
such as ailu e p edic ion, new esou ces con ac ing o “old”
de ices and se ices shu down
Appl. Sci. 2022,12, 4347 6 o 24
Table 1. Con .
Topic Cu en Challenges
Cloud Con inuum •
Few o no a ailable app oaches suppo ing he Cloud Con inuum a
p e-deploymen and ope a ion ime
Secu i y Assu ance
in Cloud/Edge/IoT
•Se ices and ools o implemen SecDe Ops as a whole in a Cloud
Con inuum a e missing
•Secu i y and p i acy assu ance in Cloud/Edge/IoT
•Ac i e Da a P o ec ion in he Cloud Con inuum
Besides, applica ion and applica ion componen s need o be classi ied (e.g., s a e ul,
s a eless) and p o iled (e.g., compu a ion, da a, se e less) in an agnos ic manne , wi hou
equi ing a p io i knowledge abou he applica ion in e nals so ha he deploymen needs
can be in e ed and deduced. Howe e , mos o he a ailable applica ions p o iles a e
pla o m- and scena io-dependen and ha dly eusable in o he con ex s. The NAMB p ojec
has s a ed o wo k in he di ec ion o a gene ic app oach ha is con ex and pla o m
agnos ic [
25
,
26
]. Mo eo e , benchma king o he whole s ack, including he ne wo k
esou ces, in pa icula he pa hs ha in e connec he physical o i ual machines hos ing
applica ion componen s, has no ye been deeply add essed. In ha espec , he p o iling o
he pa hs wi hin he cloud and he edge will p o ide ele an in o ma ion which can impac
on he decision on he selec ion o he bes combina ion o esou ces o he deploymen .
The unde s anding o hese pa hs is no i ial because public cloud se ice p o ide s such
as Amazon Web Se ices and Mic oso Azu e p o ide links ha in e connec da a cen e s
wi h high pe o mance [
27
] opposi e o ypical bes -e o In e ne links [
28
]. Howe e ,
hose links p esen complex a ic enginee ing echniques, which equi e ca e ul design and
uning o moni o ing ools in o de o a oid biased measu emen s. Addi ionally, he pa hs
ha ex end om he cloud up o he edge expec edly o e less p edic able pe o mance
han cloud links [29].
Osmo ic compu ing [
30
] is ano he app oach which “aims o decompose applica ions
in o mic ose ices wi hou deg ada ion o QoS and pe o m dynamic ailo ing o mic ose -
ices in sma en i onmen s exploi ing esou ces in edge and cloud in as uc u es”.
Un o una ely, decomposi ion o applica ions in o mic ose ices is no always pos-
sible. The e o e, s a eless and s a e ul componen s will coexis in he Cloud Con inuum.
The benchma k o s a e ul applica ions is based on la ency, scalabili y and elas ici y, and
de elope s on he cloud y o inco po a e hese pa ame e s in o hei designs [
31
]. On he
o he hand, s a eless applica ions a e assessed in e ms ela ed o he modula i y o he
applica ion, such as con ol and lexibili y.
2.2. Deploymen O ches a ion and Op imiza ion
The Cloud Con inuum concep encompasses he idea o p ocessing each applica ion
and/o applica ion componen wi h he mos app op ia e esou ce, om cloud compu ing
sys ems o IoT de ices, depending on di e en ac o s: he need o esou ces, p oximi y,
e c. In his complex scena io, he selec ion o he op imized deploymen con igu a ion is s ill
a challenge o be sol ed. The selec ion o he bes combina ion o esou ces can be ea ed
as an op imiza ion p oblem which can be sol ed h ough di e en ypes o echniques
om he h ee main ca ego ies: exac me hods [
32
]; heu is ics [
33
] and me aheu is ics [
34
],
he las ones enjoying g ea e popula i y because o hei adap abili y and e iciency [
35
].
The li e a u e is sca ce in s udies cen e ed on he op imal deploymen o mic ose ice o e
he e ogeneous cloud se ices. In [
36
], a NSGA-II algo i hm was used o ma ch he speci ic
equi emen s o a ce ain Big Da a applica ion o he capabili ies p o ided by an IaaS
in as uc u e and he Big Da a pla o m deployed he ein agains h ee design aspec s o
he in as uc u e o he Big Da a applica ion: cos , eliabili y and ne compu ing capaci y.
Simila ly, in [
37
], wo main objec i es we e conside ed o op imiza ion: (1) ul ilmen
Appl. Sci. 2022,12, 4347 7 o 24
o he mic ose ices’ Non-Func ional-Requi emen s (NFRs) (loca ion, a ailabili y, cos ,
pe o mance and legal le el), and (2) mee ing o he cha ac e is ics se by he de elope s
o hese mic ose ices (classi ica ion, public IP, disk space, RAM and numbe o co es).
The op imiza ion p ocess desc ibed abo e can be en iched by ha ing in o ma ion abou
he beha io o he execu ion en i onmen and he unde lying esou ces a un- ime, so
ha he op imiza ion unc ion can also conside his in o ma ion as ano he dimension o
incoming da a.
The deploymen o applica ions in gene al has bene i ed in he las decade om
he In as uc u e-as-Code (IaC) [
7
] concep which aims o au oma e he p o isioning,
con igu a ion and deploymen o in as uc u e esou ces ollowing a machine- eadable ile
app oach. Tools o IaC p ocess con igu a ion iles de eloped by Ops eams o di e en
pu poses include: P o isioning; Con igu a ion; Deploymen and O ches a ion o he
esou ces. Howe e , he e a e se e al open issues ha s ill need o be sol ed, especially
when alking abou au oma ic deploymen and o ches a ion in he Cloud Con inuum:
•
Tools [
38
–
41
] o i ual and physical esou ces p o isioning a e pla o m dependen
lacking in e ope abili y and po abili y and mainly ocused on cloud esou ces and do
no conside edge nodes;
•
Some Con inuous Deli e y and Con inuous Deploymen suppo ing ools a e speci ic
o deploying con aine s-based applica ions (Ranche [
42
]) while o he s a e mo e
gene ic (e.g., Apache B ooklyn, Spinnake [
43
], Alien4Cloud [
44
], Cloudi y [
45
]) and
can deploy bo h adi ional and con aine -based applica ions;
•
Kube ne es [
46
], he leading, mos used and ad anced o ches a ion pla o m oday
has been used o he o ches a ion o adi ional cloud esou ces. When i comes
o he edge hough, new ea u es such as low ne wo k la ency [
47
] and sel -healing
need o be conside ed due o he edge’s highe p oneness o ailu es ha impac on
main enance cos s. Eme ging Kube ne es on he edge open sou ce solu ions such
as K3S, Mic oK8S and KubeEdge a e ge ing huge in e es by he Kube ne es’ edge
communi y and aim o le e age on he exis ing ecosys em h ough o icial Cloud
Na i e Compu ing Founda ion ce i ica ion, while a he same ime a e ligh enough
o un on low-cos boa ds;
•
The managemen o IoT de ices, especially hose wi h high cons ain s (i.e., no
always connec ed, lack o con aine iza ion suppo ), needs o be add essed so ha
he deploymen can be done suppo ing key edge cha ac e is ics, such as mobili y,
he e ogenei y and ola ili y [48].
2.3. Cloud Resou ces Moni o ing and Sel Lea ning in Hyb id En i onmen s
Cloud esou ces’ moni o ing is a challenging ask when applied o hyb id en i on-
men s. Cloud p o ide s claim o suppo ex ensi e moni o ing mechanisms o aid in
con olling applica ion pe o mance and wo king condi ions bu hey ba ely o e ans-
pa en access o ac ual law le el me ics (e.g., Mean Time To Reco e (MTTR), Mean Time
Be ween Failu es (MTBF), eal ime CPU consump ion, e c.) no p o ide common o s an-
da dized me ics ha can be compa ed among di e en cloud esou ces ypes o p o ide s.
In [
49
], he au ho s iden i ied he QoS and Se ice Le el Ag eemen (SLA) assessmen in
complex hyb id scena ios as one o he ou clea gaps o he cloud compu ing managemen ,
due o he lack o mechanisms o add ess he pa icula i ies o la ge-scale cloud se ups wi h
mo e complex en i onmen s in e ms o esou ce he e ogenei y and dis ibu ion, such as
hyb id and mul i-cloud scena ios. In [
50
], he au ho s p oposed an app oach o adap he
planning in Complex Se ice-Based Sys ems, should an SLA iola ion occu . Howe e , in
his disse a ion he au ho s assumed ha a iola ion occu s bu do no suppo he ac ual
moni o ing no go in dep h in o how he NFRs a e moni o ed o calcula ed so ha he
iola ion can be iden i ied.
In [
37
] moni o ing o cloud esou ces was iden i ied as one o he missing unc ionali-
ies in cu en solu ions and app oaches. This conclusion was ex ac ed om an analysis
o di e en mul i-cloud abs ac ion solu ions, as cloud b oke s, cloud in e media o s o
Appl. Sci. 2022,12, 4347 8 o 24
cloud ma ke places which au oma e he assembly o complex applica ions, i s deploymen
and ope a ion on one o mul iple cloud in as uc u es. This app oach p oposes using a
combina ion o push and pull moni o ing [
51
] o ele an NFRs such as pe o mance, cos ,
a ailabili y o loca ion and suppo ing he ISO 19086.
In [
52
] a cloud moni o ing s ack is p oposed o di e en Cloud Se ices ( i ual
machines, da abases as a se ice, e c.) om di e en p o ide s (Amazon, Azu e, ARSYS).
Fu he mo e, i ocuses he moni o ing o he esou ces on he lowe le el, conside ing
Fowle ’s [
53
] de ini ion. Ne e heless, i doesn’ ackle he communica ion laye , which
in some speci ic scena ios can pose ele an challenges [
54
]. The app oach p oposed in
his pape ies o ex end i o suppo he moni o ing o dis ibu ed and he e ogeneous
(di e en na u e) in as uc u al elemen s. These me ics will be p o ided bo h o he
sel -lea ning and sel -healing algo i hms as ex a inpu da a o enhance hei esul s.
Moni o ing and p e en ing QoS iola ions a e a c i ical aspec o he design and
planning o in as uc u al se ices, hence o he ope a ionaliza ion o applica ions, in
which p edic i e models play an essen ial ole. Ini ial pos -mo em analysis echniques [
55
]
applied o he managemen o hese iola ions ha e made way o new app oaches ocused
on he an icipa ion o he ailu es, i s as an o line p ocess [
56
] and mo e ecen ly a un
ime [
57
,
58
]. Mo e ecen ly, and due o he na u e o he e ogeneous dis ibu ed se ices
especially in highly demanding en i onmen s, he In e ne o Things (IoT) has become
one o he main applica ions o s eam lea ning [
59
], since i is composed o senso s and
ac ua o s connec ed by ne wo ks o compu ing sys ems, which manage he heal h and
ac ions o connec ed objec s o machines in eal- ime. One o he main p oblems o he
Cloud Con inuum pa adigm e e ing o he p edic ion o QoS ailu es is ha hey a e
subjec o con igu a ion d i s, unau ho ized changes, missing dependencies o any in isible
o no moni o able en i onmen a ia ions ha ha e a isen in he se ices. Howe e ,
echniques such as concep d i [60,61] o anomalies de ec ion [62,63], ha could p o ide
ele an bene i s, ha e no a ac ed much a en ion ye in he esea ch communi y.
2.4. Sel -Healing and Da a Po abili y
The applica ion o sel -healing, e e s o he au onomous and esponsible beha io
o he applica ions o changes in he execu ion en i onmen , and has been a challenge
since he cloud and dis ibu ed compu ing pa adigms appea ed on he so wa e ope a ion
gameboa d. Se e al solu ions ha e been p oposed o speci ic scena ios such as IoT [
64
,
65
]
o adi ional cloud en i onmen s [
66
]. O he solu ions a e ocused on speci ic s eps in
he sel -healing, sel -con igu a ion p ocess [
67
], o in he esolu ion o speci ic p oblems
such as scalabili y [
68
] o us en o cemen [
69
]. The c oss/mul i-laye and he ne wo king
aspec s a e challenges ha ha e no ye been add essed, no has he ace o he p oblem in a
gene ic way, co e ing he whole sel -healing p ocess om he disco e y and con igu a ion
o he esou ces o he ne wo k p epa a ion and deploymen o all so wa e laye s.
Sel -healing mechanisms can include se e al ac ions, going om he s a ing up o he
ailing esou ce wi h a new one, o he e-con igu a ion o he ne wo k o he o ches a ion
o po abili y wo k lows so ha business con inui y is ensu ed. Se e al app oaches ha e
been applied o au oma ically c ea e compu a ional esou ces (usually i ual machines)
bu ew ini ia i es can be ound when add essing he e ogeneous en i onmen s o di e en
na u es (edge and cloud) and ne wo king se ups. In such a complex en i onmen , sel -
healing ac ions may need dynamic o ches a ion o he se ices used o e-deploymen
ollowing he p ope wo k low, as well as ne wo k esou ces’ e-alloca ion and se ing up.
This p ocess becomes e en mo e complex when add essing no only he po abili y o
he compu a ional componen s (s a eless componen s) bu also he po abili y o da a, o
s a e ul componen s. One o he enable s o such po abili y can be adop ed om he appli-
ca ion poin o iew, h ough he adop ion o con aine s-based echnologies, such as Docke .
Howe e , con aine iza ion pe se does no sol e he po abili y p oblem. When po ing
componen s be ween wo cloud p o ide s, da a need o be mo ed and kep synch onized
(a h ee di e en le els—blocks, iles o ansac ions) and mos con aine -based pla o ms
Appl. Sci. 2022,12, 4347 9 o 24
don’ handle his. Some con aine s’ o ches a ion sys ems, such as Kube ne es, p o ide
con enien abs ac ions o applica ion esou ce equi emen s and endpoin s, namely pe -
sis en olumes, ing ess ules and ing ess con olle s [
70
]. Howe e , manual con igu a ion
is s ill needed, and s a eless componen s a e decoupled om he da a ha ha e o be
s o ed in a da abase o o he ype o s o age. Ano he impo an concep o be conside ed
when po ing da a componen s om one esou ce o ano he is he “da a g a i y”, a e m
coined by Da e McC o y [
71
], who explained ha , because o ne wo k bandwid h limi s,
la ency, cos s and o he conside a ions, da a “wan ” o be nea he applica ions analyzing,
ans o ming o o he wise wo king on i . Consequen ly, da a po abili y in oduces new
equi emen s added o he al eady ime consuming, e o p one and mainly manual ac i i y
o po ing applica ion componen s o e di e en in as uc u al elemen s [
72
], such as:
(1) Es ablishing he igh ne wo king condi ions, so ha da a can be accessed om he
equi ed mic ose ice wi h he equi ed (ne wo k) condi ions; (2) Handling pe sis en da a
s o age, du ing a edeploymen ; (3) Da a Base au oma ic con igu a ion so ha i can be
e-deployed wi hou manual in e en ion. Secu i y is also o be add essed when selec ing
a po ing s a egy. Mechanisms o au oma ic execu ion o secu e da a po abili y in he
Cloud Con inuum, p ese ing da a in eg i y and assu ing ha he da a a e no being
co up ed need o be conside ed.
3. CloudOps: Concep s and a Re e ence F amewo k
3.1. CloudOps Concep
The cu en IT ma ke is mo e and mo e domina ed by he “Cloud Con inuum”. In-
c easing ubiqui y and he pe asi eness o compu e capabili ies and da a a ailabili y ha e
esul ed in he p oli e a ion o complex applica ions which e ec i ely p ocess da a om
he e ogenous digi al sou ces in a imely manne [
73
]. As he ypes and olumes o a ailable
da a g ow, new applica ions a e implemen ed ha seamlessly combine eal- ime da a wi h
complex models and da a analy ics o moni o and manage sys ems o in e es . The da a
ha e o be p ocessed p omp ly o ex ac insigh s ha can d i e decision making. T adi-
ional app oaches ha ely on mo ing da a o emo e da a cen e s o p ocessing a e no
longe easible. Ins ead, new app oaches ha e ec i ely le e age dis ibu ed compu a ional
in as uc u e and se ices a e necessa y. Speci ically, hese app oaches mus seamlessly
combine esou ces and se ices a he edge (edge compu ing), in he co e (cloud compu ing)
and along he da a pa h ( og compu ing) as needed, h ough he Cloud Con inuum. A
un ime, applica ions can choose o execu e pa s o hei logic on di e en in as uc u es
ha cons i u e he con inuum, wi h he goal o minimizing la ency, ene gy consump ion
and maximizing a ailabili y o pe o mance [
74
]. This equi es no el echniques and me h-
ods o ede a ing in as uc u e, p og amming applica ions and se ices and composing
dynamic wo k lows, which a e capable o eac ing in eal- ime o unp edic able da a sizes,
a ailabili y, loca ions and a es [75].
The he e ogenei y o he “compu ing con inuum” is b oad and mul is age. In he
“ adi ional” cloud, compu ing esou ces a e ypically p o ided h ough i ualiza ion and
con aine iza ion [
74
], wi h “in ini e” esou ce a ailabili y hanks o ho izon al scaling. In
con as , in edge compu ing, compu a ional esou ces a e sca ce and mus be managed
e y e icien ly due o ba e y cons ain s o o he limi a ions [76].
The combina ion o such echnologies implies ha he De Ops eams need o ha e
a deep knowledge o he unde lying echniques and echnologies and need o be able o
wo k wi h se e al o hem, seamlessly in eg a ed. These kinds o he e ogeneous com-
pu ing en i onmen s pose complex asks o he ope a ionaliza ion o he applica ions
by he De Ops eams as hey mus con igu e, plan, p epa e and execu e hem. When
applying i o dynamic, changeable execu ion en i onmen s whe e he applica ions need
o be econ igu ed unde eal ime igh equi emen s, his p ocess needs o be sho , e-
pea able and (semi-)au oma ic. Applica ion ope a o s need o: (1) ha e eal ime upda ed
in o ma ion on he ne wo k condi ions, compu ing esou ces a ailable and da a in as uc-
u es and se ices equi emen s and (2) be able o seamlessly and anspa en ly adap he
Appl. Sci. 2022,12, 4347 16 o 24
in pa icula o applica ions ha need eal ime, low la ency o hose gi ing suppo o
c i ical in as uc u es.
In pa icula , he p esen ed CloudOps app oach can be o special ele ance when
ackling c i ical in as uc u es o business-c i ical applica ions whe e he esponse ime
becomes i al and he sys em mus be esilien and able o ope a e in di e en scena ios,
including lack o connec i i y ha can be eplaced by al e na i e communica ion ech-
nologies (e.g., new edge nodes). C i ical in as uc u e is he body o sys ems, ne wo ks
and asse s ha a e so essen ial ha hei con inued ope a ion is equi ed o ensu e he
secu i y o a gi en ci y o egion, i s economy and he public’s heal h and/o sa e y. Those
c i ical in as uc u es mus ha e a e y high a ailabili y a e, e en highe han i e nines.
Following on om ha , how he ope a ion o c i ical in as uc u es-based sys ems can
bene i om he app oach and he e e ence amewo k p oposed in his a icle is explained
h ough wo speci ic examples.
4.1. C i ical In as uc u es o Sa e y and Secu i y
The ad en o sma ci ies is os e ing he deploymen o di e en ypes o in as uc-
u es o complemen and p o ide public secu i y se ices wi h da a o de ec h ea ening
si ua ions. Sys ems such as sma s ee ligh ing in as uc u es suppo ing he secu i y
and sa e y o ci izens and goods in public spaces a e gaining momen um. Howe e , hese
sys ems need o manage he la ge quan i y o da a gene a ed, e.g., by su eillance came as,
he e o e, AI-based sys ems a e equi ed o au oma ically analyze and il e such da a o
epo only on po en ially h ea ening si ua ions. The la ge a ie y o possible si ua ions
and he la ge quan i y o da a gene a ed implies an obliga ion o he sys ems o be able o
adap o hese con inuously changing si ua ions, which may also be comple ely di e en
depending on he ci y neighbo hood, seasons o he yea o speci ic e en s., Machine
lea ning echnologies will be equi ed o ake ad an age o algo i hms ha may need o
be adap ed and edeployed depending on he de ec ed si ua ions. In he case o sma
ligh ing in as uc u es, he CloudOps e e ence amewo k can p o ide suppo o he
op imiza ion o he deploymen con igu a ion in an en i onmen wi h con inuously chang-
ing equi emen s and a limi ed p ocessing capaci y. Addi ionally, he algo i hms ha
analyze he da a may suddenly need high compu a ional capaci y in speci ic loca ions and
imes, and pe manen ly p o iding such compu a ional capaci y up o he edge would be
nei he a o dable no sus ainable, so he e is also he need o lexibili y o ake ad an age
o he compu a ional powe o o he edge nodes o he cloud. To achie e his, un ime
moni o ing and sel -healing mechanisms can suppo he ope a o s in he ope a ion o
such complex en i onmen s, p o iding hem wi h he lexibili y o execu e code a any
ime in he mos con enien compu ing esou ce, he capaci y o upda e and o e-deploy
algo i hms whene e equi ed and keeping in es men and ope a ional cos s a o dable.
4.2. C i ical In as uc u es o Eme gency Managemen
The managemen o public e en s and c owd con ol is a ypical example o mission
c i ical se ices; a se ice down ime could unde mine ci izen secu i y and c ea e p oblems
in he se ice con inui y leading o e y se ious issues. This ype o sys em collec s da a
om people using in-place nodes, e.g., Wi-Fi/Blue oo h. Wi h his in o ma ion he sys em
is able o es ima e he numbe o people in a speci ic a ea and, aking ad an age o he
bandwid h and low la ency ea u es o e ed by 5G connec i i y, send hese da a o he
edge nodes so ha hey a e able o de e mine i a c i ical condi ion is going o happen, o
example, i he low o people is cohe en wi h he da a collec ed on he buses o in o he
public anspo a ion sys ems. In his si ua ion, i an edge node goes down, aul - ole ance
mechanisms need o be pu in place. Howe e , while in many o he cases i is possible o
use aul - ole ance ea u es o many o he mos common o ches a o s (such as Kube ne es)
his is no possible o s a e ul se ices. In his espec , he CloudOps sel -healing app oach,
including au oma ic da a po abili y echniques, can p o ide suppo o he Ope a o s
o he sys em. Simila ly, ins ead o managing he se ice con inui y by ha ing wo edge
Appl. Sci. 2022,12, 4347 17 o 24
nodes, i could be possible o mo e compu a ion ac oss he con inuum mo ing om he
edge o he cloud. In bo h cases, using s a e ul objec s, he sel -healing mechanisms will
ensu e ha in he e en o a aul , ope a ions con inue wi hou in e up ion by p o iding
da a on he mo emen o he c owd.
4.3. Business C i ical Applica ions: Ini ial Resul s
The p oposed CloudOps wo k low and amewo k ha e been pa ially alida ed
h ough an implemen a ion o an MVP (Minimum Viable P oduc ) in an e-heal h scena io.
In his case, an ini ial e sion o he p oposed CloudOps amewo k was used o deploy
and ope a e a mul i-cloud e-heal h .NET applica ion. The solu ion implemen ed included
all he desc ibed unc ional componen s om Figu e 3, wi h limi ed ea u es (i.e., only
cloud in as uc u al elemen s we e conside ed, a limi ed se o NFRs we e moni o ed
and s a eless componen s we e no suppo ed). The di e en componen s om Figu e 3
ha e been inse ed in o he ac ual amewo k and he main sub-componen s o each o
he unc ional blocks a e ep esen ed in Figu e 7. Figu e 7depic s he a chi ec u e o he
ini ial MVP o he p oposed CloudOps amewo k alida ed in he e-heal h scena io. I
also shows how he addi ional ac i i ies p oposed in he CloudOps concep in Figu e 2(i.e.,
p e-deploymen and sel -healing) a e suppo ed by speci ic ools in his es case. To his
end, he p e-deploymen phase is ma e ialized in wo new ool suppo ed ac i i ies:
Appl. Sci. 2022, 12, x FOR PEER REVIEW 17 o 24
mechanisms need o be pu in place. Howe e , while in many o he cases i is possible o
use aul - ole ance ea u es o many o he mos common o ches a o s (such as Kube -
ne es) his is no possible o s a e ul se ices. In his espec , he CloudOps sel -healing
app oach, including au oma ic da a po abili y echniques, can p o ide suppo o he
Ope a o s o he sys em. Simila ly, ins ead o managing he se ice con inui y by ha ing
wo edge nodes, i could be possible o mo e compu a ion ac oss he con inuum mo ing
om he edge o he cloud. In bo h cases, using s a e ul objec s, he sel -healing mecha-
nisms will ensu e ha in he e en o a aul , ope a ions con inue wi hou in e up ion by
p o iding da a on he mo emen o he c owd.
4.3. Business C i ical Applica ions: Ini ial Resul s
The p oposed CloudOps wo k low and amewo k ha e been pa ially alida ed
h ough an implemen a ion o an MVP (Minimum Viable P oduc ) in an e-heal h scena io.
In his case, an ini ial e sion o he p oposed CloudOps amewo k was used o deploy
and ope a e a mul i-cloud e-heal h .NET applica ion. The solu ion implemen ed included
all he desc ibed unc ional componen s om Figu e 3, wi h limi ed ea u es (i.e., only
cloud in as uc u al elemen s we e conside ed, a limi ed se o NFRs we e moni o ed and
s a eless componen s we e no suppo ed). The di e en componen s om Figu e 3 ha e
been inse ed in o he ac ual amewo k and he main sub-componen s o each o he
unc ional blocks a e ep esen ed in Figu e 7. Figu e 7 depic s he a chi ec u e o he ini ial
MVP o he p oposed CloudOps amewo k alida ed in he e-heal h scena io. I also
shows how he addi ional ac i i ies p oposed in he CloudOps concep in Figu e 2 (i.e.,
p e-deploymen and sel -healing) a e suppo ed by speci ic ools in his es case. To his
end, he p e-deploymen phase is ma e ialized in wo new ool suppo ed ac i i ies:
Figu e 7. A chi ec u e o he MVP o he p oposed CloudOps amewo k alida ed in an e-heal h
scena io.
1. A ailable cloud esou ces disco e y h ough he “In as uc u al elemen s ca a-
logue” (Resou ce disco e y unc ional block in Figu e 3): This ca alogue p o ides a
unique egis y o all he in as uc u al elemen s a ailable along wi h hei ele an
cha ac e is ics. In his conc e e es case, cloud se ices om Amazon, Azu e and
A sys we e included. The ope a o s o he es ed e-heal h applica ion we e able o
disco e and il e he di e en esou ces o hei own con enience. As epo ed by
he ope a o s, his supposed a g ea ad an age and ele an ime sa ings as hey did
no need o dig in o he web pages o each o he cloud p o ide s looking o he bes
Figu e 7.
A chi ec u e o he MVP o he p oposed CloudOps amewo k alida ed in an e-heal h scena io.
1.
A ailable cloud esou ces disco e y h ough he “In as uc u al elemen s ca alogue”
(Resou ce disco e y unc ional block in Figu e 3): This ca alogue p o ides a unique
egis y o all he in as uc u al elemen s a ailable along wi h hei ele an cha ac-
e is ics. In his conc e e es case, cloud se ices om Amazon, Azu e and A sys we e
included. The ope a o s o he es ed e-heal h applica ion we e able o disco e and
il e he di e en esou ces o hei own con enience. As epo ed by he ope a o s,
his supposed a g ea ad an age and ele an ime sa ings as hey did no need o dig
in o he web pages o each o he cloud p o ide s looking o he bes combina ion o
cloud esou ces. The ca alogue has been de eloped wi h he JHips e [
77
] on end
and backend gene a o ;
2.
“Op imized deploymen con igu a ion” p o isioning ( he Op imized Deploymen
Con igu a ion building block in Figu e 3): This componen ga he s he in o ma-
ion om he applica ion (a ailable in he applica ion desc ip ion ile shown in
Figu es 8and 9)
and collec s he equi ed in o ma ion om he In as uc u al el-
emen s’ ca alogue ( h ough a REST API). F om his, i c ea es he bes combina ion o
Appl. Sci. 2022,12, 4347 18 o 24
a ailable in as uc u al elemen s ul illing he applica ion componen s ypes’ needs
(i.e., compu ing capabili ies, s o age capabili ies, e c.) and he SLOs (i.e., loca ion,
pe o mance and a ailabili y). I uses he NSGA-II Gene ic Algo i hm [
78
] imple-
men ed in MOEA F amewo k [
79
], an open sou ce Ja a lib a y. These componen s
p o ide he applica ion ope a o eam wi h a ool o pe o m se e al es s be o e he
applica ion is deployed. Fu he mo e, i eases he selec ion o he se ices agains he
selec ed NFRs, in a mul i-objec i e op imiza ion p oblem.
Appl. Sci. 2022, 12, x FOR PEER REVIEW 18 o 24
combina ion o cloud esou ces. The ca alogue has been de eloped wi h he JHips e
[77] on end and backend gene a o ;
2. “Op imized deploymen con igu a ion” p o isioning ( he Op imized Deploymen
Con igu a ion building block in Figu e 3): This componen ga he s he in o ma ion
om he applica ion (a ailable in he applica ion desc ip ion ile shown in Figu es 8
and 9) and collec s he equi ed in o ma ion om he In as uc u al elemen s’ ca a-
logue ( h ough a REST API). F om his, i c ea es he bes combina ion o a ailable
in as uc u al elemen s ul illing he applica ion componen s ypes’ needs (i.e., com-
pu ing capabili ies, s o age capabili ies, e c.) and he SLOs (i.e., loca ion, pe o mance
and a ailabili y). I uses he NSGA-II Gene ic Algo i hm [78] implemen ed in MOEA
F amewo k [79], an open sou ce Ja a lib a y. These componen s p o ide he appli-
ca ion ope a o eam wi h a ool o pe o m se e al es s be o e he applica ion is
deployed. Fu he mo e, i eases he selec ion o he se ices agains he selec ed
NFRs, in a mul i-objec i e op imiza ion p oblem.
Figu e 8. Snippe o he applica ion desc ip ion JSON ile, whe e he applica ion’s NFRs a e de-
sc ibed.
Figu e 8.
Snippe o he applica ion desc ip ion JSON ile, whe e he applica ion’s NFRs a e desc ibed.
Appl. Sci. 2022, 12, x FOR PEER REVIEW 19 o 24
Figu e 9. Snippe o he applica ion desc ip ion JSON ile, whe e he bes combina ion o in as uc-
u al elemen s a e sugges ed in h ough he “schema” elemen .
As explained, he mic ose ices-based e-heal h applica ion’s main cha ac e is ics
we e main ained in he “Applica ion Desc ip ion” JSON ile which was used by he
di e en componen s o ga he and exchange ele an in o ma ion. Consequen ly,
he in as uc u al equi emen s and applica ion componen s’ desc ip ion (Figu e 3)
a e collec ed in he applica ion desc ip ion ile as p esen ed in Figu e 7.
The ope a ion phase is also en iched wi h a se o new ool-suppo ed ac i i ies spe-
cially o he sel -healing;
3. Au oma ic deploymen o he applica ion occu s h ough he “In as uc u e gene -
a ion” and he “Applica ion Deploymen ” sub-modules (Deploymen and Sel -Heal-
ing in Figu e 3). They c ea e he needed in as uc u al esou ces and deploy he ap-
plica ion h ough Te a o m and Ansible empla es. Besides he deploymen o he
applica ion, hey also deploy a se o moni o ing agen s (Teleg a -based [80]) in he
in as uc u e o ga he he co esponding un ime me ics aligned wi h he selec ed
NFRs;
4. “Au oma ic un- ime moni o ing” o all he in as uc u al elemen s (Figu e 3): I
ga he s in a single place all he moni o ed me ics om he di e en p o ide s’ e-
sou ces. I comp ises a ull moni o ing s ack, based on Teleg a agen s, he In luxDB
[81] ime-se ies da abase as he pe sis ence laye and G a ana dashboa ds [82] o he
g aphical in e ace. In his componen , he “Moni o ing con olle ” is also in cha ge
o de ec ing SLO iola ions in he in as uc u al elemen s and in o ming bo h he
Ope a o o he applica ion ( h ough an email) and he “Deploymen and Sel -Heal-
ing” module ( h ough a REST API) abou hese iola ions;
5. Finally, he “Po ing s a egies selec o ” sub-module (as pa o he Sel -lea ning
building block in Figu e 3) ecei es he eques s om he “Moni o ing con olle ”
once a iola ion occu s and e-deploys he applica ion, i needed. I his is he case,
he “Op imized deploymen con igu a ion” is called and he cycle s a s again wi h
he op imiza ion o he deploymen con igu a ion. I is o be no ed ha he “In a-
s uc u al Elemen s ca alogue” is also upda ed wi h he in o ma ion o he iola ions
ha ha e occu ed on he used in as uc u al elemen s. As a esul , when he nex
se o a ailable esou ces a e eques ed he in o ma ion o he iola ions is also con-
side ed so ha he opiniona ed selec ion o in as uc u al elemen s can be p o ided
by he “Op imized deploymen con igu a ion”.
As an icipa ed, he es case only co e ed some o he main unc ionali ies o he
CloudOps amewo k. Ne e heless, i showed posi i e impac s and imp o emen s o he
Ope a ionaliza ion o he e e enced applica ion bo h quali a i e, p esen ed in he de-
sc ip ion o he Figu e 7 componen s, and quan i a i e, p esen ed in Table 3. Table 3
shows he quan i a i e imp o emen s in he o m o compa a i e cos s (measu ed in e -
o , PM-Pe son Mon h and PH-Pe son Hou ) be ween using he CloudOps amewo k
and no using i . These igu es ha e been epo ed by he ope a o s o he e-heal h
Figu e 9.
Snippe o he applica ion desc ip ion JSON ile, whe e he bes combina ion o in as uc-
u al elemen s a e sugges ed in h ough he “schema” elemen .
Appl. Sci. 2022,12, 4347 19 o 24
As explained, he mic ose ices-based e-heal h applica ion’s main cha ac e is ics
we e main ained in he “Applica ion Desc ip ion” JSON ile which was used by he
di e en componen s o ga he and exchange ele an in o ma ion. Consequen ly, he
in as uc u al equi emen s and applica ion componen s’ desc ip ion (Figu e 3) a e
collec ed in he applica ion desc ip ion ile as p esen ed in Figu e 7.
The
ope a ion phase
is also en iched wi h a se o new ool-suppo ed ac i i ies
specially o he sel -healing;
3.
Au oma ic deploymen o he applica ion occu s h ough he “In as uc u e gen-
e a ion” and he “Applica ion Deploymen ” sub-modules (Deploymen and Sel -
Healing in Figu e 3). They c ea e he needed in as uc u al esou ces and deploy
he applica ion h ough Te a o m and Ansible empla es. Besides he deploymen
o he applica ion, hey also deploy a se o moni o ing agen s (Teleg a -based [
80
])
in he in as uc u e o ga he he co esponding un ime me ics aligned wi h he
selec ed NFRs;
4.
“Au oma ic un- ime moni o ing” o all he in as uc u al elemen s (Figu e 3): I ga h-
e s in a single place all he moni o ed me ics om he di e en p o ide s’ esou ces.
I comp ises a ull moni o ing s ack, based on Teleg a agen s, he In luxDB [
81
] ime-
se ies da abase as he pe sis ence laye and G a ana dashboa ds [
82
] o he g aphical
in e ace. In his componen , he “Moni o ing con olle ” is also in cha ge o de ec ing
SLO iola ions in he in as uc u al elemen s and in o ming bo h he Ope a o o
he applica ion ( h ough an email) and he “Deploymen and Sel -Healing” module
( h ough a REST API) abou hese iola ions;
5.
Finally, he “Po ing s a egies selec o ” sub-module (as pa o he Sel -lea ning
building block in Figu e 3) ecei es he eques s om he “Moni o ing con olle ”
once a iola ion occu s and e-deploys he applica ion, i needed. I his is he case, he
“Op imized deploymen con igu a ion” is called and he cycle s a s again wi h he
op imiza ion o he deploymen con igu a ion. I is o be no ed ha he “In as uc u al
Elemen s ca alogue” is also upda ed wi h he in o ma ion o he iola ions ha ha e
occu ed on he used in as uc u al elemen s. As a esul , when he nex se o
a ailable esou ces a e eques ed he in o ma ion o he iola ions is also conside ed
so ha he opiniona ed selec ion o in as uc u al elemen s can be p o ided by he
“Op imized deploymen con igu a ion”.
As an icipa ed, he es case only co e ed some o he main unc ionali ies o he
CloudOps amewo k. Ne e heless, i showed posi i e impac s and imp o emen s o
he Ope a ionaliza ion o he e e enced applica ion bo h quali a i e, p esen ed in he
desc ip ion o he Figu e 7componen s, and quan i a i e, p esen ed in Table 3. Table 3
shows he quan i a i e imp o emen s in he o m o compa a i e cos s (measu ed in e o ,
PM-Pe son Mon h and PH-Pe son Hou ) be ween using he CloudOps amewo k and no
using i . These igu es ha e been epo ed by he ope a o s o he e-heal h applica ion. The
di e en phases o he CloudOps wo k low we e implemen ed by he ope a o s manually
when using he CloudOps amewo k.
Assuming a cos o 40 eu os/h (calcula ed by he e-heal h applica ion ope a o s, based
on he cos s o a ypical p ojec in hei company) a o al o 7280
€
eu os was sa ed. I
we assume ou heal h p ojec s o a simila scale pe yea , hen we an icipa e a sa ing o
app oxima ely 29,000 Eu os, o a ound 2400 Eu os pe mon h (app ox.
1
2
a Pe son Mon h.)
The ope a o s o he e-heal h applica ion also epo ed quali a i e imp o emen s. The
mos ele an ones we e as ollows:
1.
The
p e-deploymen phase
and he ela ed ools (“In as uc u al elemen s ca a-
logue”, “Op imized deploymen con igu a ion”) p o ide e iciency o e alua ing a
use ’s po en ial NFR op ions o each o he se ice classi ica ions and p o ide an
ins an esponse o illus a e ma ching based on he use ’s speci ied alues o each
o hei NFRs o help e alua e he mos e ec i e op ions when selec ing a se ice
solu ion. This e iciency exis s, as wi hou his ool he use would ha e o sea ch each
Appl. Sci. 2022,12, 4347 20 o 24
CSP indi idually o iden i y ma ches and each a decision abou he bes solu ion on
he in o ma ion ha was disco e ed manually;
2.
The
ope a ion and sel -healing
phase enhances he ul ilmen o he applica ion’s
Se ice Le el Ag eemen (SLA). As he SLA, and he unde lying SLAs o he unning
cloud se ices, can con inuously be moni o ed and he iola ions a e au oma ically
de ec ed, so he applica ion’s SLA is ha dly b oken
Based on hese ini ial esul s we can en ision g ea e impac s in u u e implemen a-
ions o he p oposed CloudOps amewo k, in he wo p oposed es cases in Sec ions 4.1
and 4.2. Especially when we will inco po a e in o he alida ion p ocess new in as uc u al
elemen s (such as edge o IoT agen s) and he comple e unc ionali y desc ibed.
Table 3. Es ima ion o he sa ings p o ided by he Cloud Ops amewo k in he e-heal h scena io.
CloudOps
Phase
E o
Manual
Implemen a ion
E o
CloudOps
F amewo k
Sa ings
Resou ces disco e y and op imiza ion 0.7 PM 0 PM 0.7 PM
Deploymen 0.1 PM 025 PH 0.1 PM
Moni o ing (annual p ojec ed) 0.25 PM 0.25 PH 0.25 PM
Re-adap a ion and sel -healing 0.25 PM 0.01 PM 0.24 PM
TOTAL 1.3 PM 0.01 PM 1.3 PM
5. Conclusions
Ecosys ems o digi aliza ion equi e he de elopmen o applica ions (SaaS) o en
unning on he e ogeneous esou ces encompassing in as uc u e elemen s (e.g., senso s),
cloud and edge esou ces and ne wo k cha ac e is ics. This a icle has p esen ed esea ch
ha s udies he ope a ionaliza ion o applica ions in he Cloud Con inuum and he co -
esponding equi emen s and needs. We ha e ou lined he main challenges h ough
an analysis o he ela ed wo ks, which has se ed as he basis o he de ini ion o he
CloudOps wo k low. We ha e in oduced he CloudOps concep and he suppo ing
e e ence amewo k which cus omizes he adi ional Ops cycle wi h ac i i ies and sup-
po ing echniques add essing he speci ic needs and challenges o he applica ions in he
Cloud Con inuum.
The no el concep o he CloudOps wo k low and e e ence amewo k a e key
indings o he wo k. They ocus on acili a ing and speeding up deploymen and ope a ion
o such ecosys ems when hey exploi he e ogeneous cloud esou ces. To his end, ou
p oposal p o ides Ops eams wi h applica ions unde he Cloud Con inuum pa adigm wi h
a CloudOps e e ence amewo k ha allows hem o deploy in an op imized con igu a ion
en i onmen while simpli ying i s ope a ion. We also inco po a ed he concep o sel -
healing and da a po abili y in he Cloud Con inuum wi h special ocus on he po abili y
o s a e ul componen s, which is an unsol ed issue up o da e. We ha e also explained
he main bene i s o he solu ion o he a ge use s and especially o he ope a o s o
c i ical in as uc u es-based sys ems. We included he ini ial esul s cap u ed wi h a i s
implemen a ion o he MVP (minimum iable p oduc ) o he p esen ed amewo k. This
will be ex ended, including a comple e s uc u al and beha io al a chi ec u al desc ip ion
o he CloudOps’ componen s desc ibed in he pape , and also wi h he de elopmen o he
co esponding comple e POCs (p oo o concep s) ha will se e o exhaus i ely alida e
he p esen ed app oach.
The nex s eps will also include he ull alida ion and e i ica ion o he applicabili y
o his no el app oach in ele an indus ial use cases. As we ha e al eady ad anced,
en isioned candida es o es ing he solu ion a e mainly applica ions o manage c i ical
in as uc u es (e.g., sa e y and secu i y, eme gency managemen ).
Appl. Sci. 2022,12, 4347 21 o 24
Au ho Con ibu ions:
Concep ualiza ion, in es iga ion, me hodology, J.A. and L.O.-E.; w i ing—
o iginal d a , J.A. and L.O.-E.; w i ing— e iew and edi ing, J.A., L.O.-E. and M.H. All au ho s ha e
ead and ag eed o he published e sion o he manusc ip .
Funding:
This esea ch was unded by he Eu opean p ojec PIACERE (Ho izon 2020 Resea ch and
Inno a ion P og amme, unde g an ag eemen No. 101000162).
Ins i u ional Re iew Boa d S a emen : No applicable.
In o med Consen S a emen : No applicable.
Da a A ailabili y S a emen : No applicable.
Con lic s o In e es : The au ho s decla e no con lic o in e es .
Re e ences
1.
Gubbi, J.; Buyya, R.; Ma usic, S.; Palaniswami, M. In e ne o Things (IoT): A ision, a chi ec u al elemen s, and u u e di ec ions.
Fu u e Gene . Compu . Sys . 2013,29, 1645–1660. [C ossRe ]
2.
Bi encou , L.; Immich, R.; Sakella iou, R.; Fonseca, N.; Madei a, E.; Cu ado, M.; Villas, L.; DaSil a, L.; Lee, C.; Rana, O. The
In e ne o Things, Fog and Cloud con inuum: In eg a ion and challenges. In e ne Things 2018,3–4, 134–155. [C ossRe ]
3.
Bi encou , L.F.; Diaz-Mon es, J.; Buyya, R.; Rana, O.F.; Pa asha , M. Mobili y-Awa e Applica ion Scheduling in Fog Compu ing.
IEEE Cloud Compu . 2017,4, 26–35. [C ossRe ]
4.
Lee, C.A. Cloud Fede a ion Managemen and Beyond: Requi emen s, Rele an S anda ds, and Gaps. IEEE Cloud Compu .
2016
,
3, 42–49. [C ossRe ]
5.
Ome o , A.; Molua, O.L.; Koma o , M.; Nu mi, J. A Su ey o Secu i y in Cloud, Edge, and Fog Compu ing. Senso s
2022
,22, 927.
[C ossRe ]
6.
Download In as uc u e as Code: Fueling he Fi e o Fas e Applica ion Deli e y—Fo es e TLP om O icial Mic oso
Download Cen e . A ailable online: h ps://www.mic oso .com/en-us/download/de ails.aspx?id=46403 (accessed on 11
Feb ua y 2022).
7.
In as uc u e as Code (IaC): The Comple e Beginne ’s Guide—BMC So wa e Blogs. A ailable online: h ps://www.bmc.com/
blogs/in as uc u e-as-code/ (accessed on 11 Feb ua y 2022).
8.
Su ey: De SecOps P og ess Remains Elusi e—De Ops.com. A ailable online: h ps://de ops.com/su ey-de secops-
p og ess- emains-elusi e/ (accessed on 11 Feb ua y 2022).
9.
9 Top So wa e De elopmen T ends o 2021. A ailable online: h ps://www.s anda d i ms.com/ op-so wa e-de elopmen -
ends/ (accessed on 11 Feb ua y 2022).
10.
Zhou, X.; Peng, X.; Xie, T.; Sun, J.; Ji, C.; Li, W.; Ding, D. Faul Analysis and Debugging o Mic ose ice Sys ems: Indus ial Su ey,
Benchma k Sys em, and Empi ical S udy. IEEE T ans. So w. Eng. 2018,47, 243–260. [C ossRe ]
11.
Shackle , M. How o Implemen Edge Compu ing—TechRepublic. A ailable online: h ps://www. ech epublic.com/a icle/
how- o-implemen -edge-compu ing/ (accessed on 11 Feb ua y 2022).
12.
Cloud Edge Compu ing: Beyond he Da a Cen e —OpenS ack is Open Sou ce So wa e o C ea ing P i a e And Public Clouds.
A ailable online: h ps://www.opens ack.o g/use-cases/edge-compu ing/cloud-edge-compu ing-beyond- he-da a-cen e /
(accessed on 11 Feb ua y 2022).
13.
Va ghese, B.; Akgun, O.; Miguel, I.; Thai, L.; Ba ke , A. Cloud Benchma king o Maximising Pe o mance o Scien i ic Applica-
ions. IEEE T ans. Cloud Compu . 2016,7, 170–182. [C ossRe ]
14.
Kousiou is, G.; Aisopos, F.; Psychas, A.; Va a igou, T.; Domaschka, J.; Bau , D.; G iesinge , F.; Nikolo , V.;
Lybe opoulos, G.
;
Theodo opoulou, E.; e al. A Toolki Based A chi ec u e o Op imizing Cloud Managemen , Pe o mance E alua ion and
P o ide Selec ion P ocesses. In P oceedings o he 2017 In e na ional Con e ence on High Pe o mance Compu ing & Simula ion
(HPCS), Genoa, I aly, 17–21 July 2017; pp. 224–232.
15.
Pali , T.; Shen, Y.; Fe dman, M. Demys i ying cloud benchma king. In P oceedings o he 2016 IEEE In e na ional Symposium on
Pe o mance Analysis o Sys ems and So wa e (ISPASS), Uppsala, Sweden, 17–19 Ap il 2016; pp. 122–132.
16.
CloudSui e—A Benchma k Sui e o Cloud Se ices. A ailable online: h ps://www.cloudsui e.ch/ (accessed on 16 Feb ua y 2022).
17.
Round 20 Resul s—TechEmpowe F amewo k Benchma ks. A ailable online: h ps://www. echempowe .com/benchma ks/
(accessed on 16 Feb ua y 2022).
18.
Home—Filebench/Filebench Wiki—Gi Hub. A ailable online: h ps://gi hub.com/ ilebench/ ilebench/wiki (accessed on 16
Feb ua y 2022).
19.
Hong, C.-H.; Va ghese, B. Resou ce Managemen in Fog/Edge Compu ing: A Su ey on A chi ec u es, In as uc u e, and
Algo i hms. ACM Compu . Su . 2020,52, 97. [C ossRe ]
20.
Tocze, K.; Schmi , N.; B andic, I.; A al, A.; Nadjm-Teh ani, S. Towa ds Edge Benchma king: A Me hodology o Cha ac e izing
Edge Wo kloads. In P oceedings o he 2019 IEEE 4 h In e na ional Wo kshops on Founda ions and Applica ions o Sel * Sys ems
(FAS*W), Umea, Sweden, 16–20 June 2019; pp. 70–71.
Appl. Sci. 2022,12, 4347 22 o 24
21.
Toczé, K.; Nadjm-Teh ani, S. A Taxonomy o Managemen and Op imiza ion o Mul iple Resou ces in Edge Compu ing. Wi el.
Commun. Mob. Compu . 2018,2018, 7476201. [C ossRe ]
22.
Das, A.; Pa e son, S.; Wi ie, M. EdgeBench: Benchma king Edge Compu ing Pla o ms. In P oceedings o he 2018 IEEE/ACM
In e na ional Con e ence on U ili y and Cloud Compu ing Companion (UCC Companion), Zu ich, Swi ze land, 17–20
Decembe 2018; pp. 175–180.
23.
Zheng, C. Bench 2021: 2021 BenchCouncil In e na ional Symposium on Benchma king, Measu ing and Op imizing. A ailable
online: h p://wikic p.com/c p/se le /e en .showc p?e en id=132576©owne id=168997 (accessed on 16 Feb ua y 2022).
24.
Alonso, J.; O ue-Eche a ia A ie a, L.; Escalan e, M.; Bengu ia, G.; Eche a ia, G. Fede a ed Cloud Se ice B oke (FCSB): An
Ad anced Cloud Se ice In e media o o Public Adminis a ions. In P oceedings o he 7 h In e na ional Con e ence on Cloud
Compu ing and Se ices Science, Po o, Po ugal, 24–26 Ap il 2017; p. 391.
25.
Gi Hub—ale93p/namb: No Only a Mic o-Benchma k. A ailable online: h ps://gi hub.com/ale93p/namb (accessed on 16
Feb ua y 2022).
26.
Paglia i, A.; Hue , F.; U oy-Kelle , G. Towa ds a High-Le el Desc ip ion o Gene a ing S eam P ocessing Benchma k Ap-
plica ions. In P oceedings o he 2019 IEEE In e na ional Con e ence on Big Da a (Big Da a), Los Angeles, CA, USA, 9–12
Decembe 2019; pp. 3711–3716.
27.
Haq, O.; Raja, M.; Doga , F.R. Measu ing and Imp o ing he Reliabili y o Wide-A ea Cloud Pa hs. In P oceedings o he 26 h
In e na ional Con e ence on Wo ld Wide Web, Pe h, Aus alia, 3–7 Ap il 2017; In e na ional Wo ld Wide Web Con e ences
S ee ing Commi ee: Gene a, Swi ze land, 2017; pp. 253–262.
28.
Yeganeh, B.; Du ai ajan, R.; Rejaie, R.; Willinge , W. A Fi s Compa a i e Cha ac e iza ion o Mul i-cloud Connec i i y in Today’s
In e ne . In Passi e and Ac i e Measu emen ; Spe o o, A., Daino i, A., S ille , B., Eds.; Lec u e No es in Compu e Science; Sp inge
In e na ional Publishing: Cham, Swi ze land, 2020; Volume 12048, pp. 193–210. ISBN 978-3-030-44080-0.
29.
Jin, Y.; Rengana han, S.; Anan hana ayanan, G.; Jiang, J.; Padmanabhan, V.N.; Sch ode , M.; Calde , M.; K ishnamu hy, A.
Zooming in on wide-a ea la encies o a global cloud p o ide . In P oceedings o he ACM Special In e es G oup on Da a
Communica ion, Beijing, China, 19–23 Augus 2019; pp. 104–116.
30.
Villa i, M.; Fazio, M.; Dus da , S.; Rana, O.; Ranjan, R. Osmo ic Compu ing: A New Pa adigm o Edge/Cloud In eg a ion. IEEE
Cloud Compu . 2016,3, 76–83. [C ossRe ]
31.
Va ghese, B.; Buyya, R. Nex gene a ion cloud compu ing: New ends and esea ch di ec ions. Fu u e Gene . Compu . Sys .
2018
,
79, 849–861. [C ossRe ]
32.
Jou dan, L.; Basseu , M.; Talbi, E.-G. Hyb idizing exac me hods and me aheu is ics: A axonomy. Eu . J. Ope . Res.
2009
,199, 620–629.
[C ossRe ]
33. Mülle -Me bach, H. Heu is ics and hei design: A su ey. Eu . J. Ope . Res. 1981,8, 1–23. [C ossRe ]
34.
Gend eau, M.; Po in, J.-Y. (Eds.) Handbook o Me aheu is ics; In e na ional Se ies in Ope a ions Resea ch & Managemen Science;
Sp inge : Bos on, MA, USA, 2010; Volume 146, ISBN 978-1-4419-1663-1.
35.
Osaba, E.; Ca balledo, R.; Diaz, F.; Onie a, E.; Masegosa, A.D.; Pe allos, A. Good p ac ice p oposal o he implemen a ion,
p esen a ion, and compa ison o me aheu is ics o sol ing ou ing p oblems. Neu ocompu ing 2018,271, 2–8. [C ossRe ]
36.
A os egi, M.; To e-Bas ida, A.; Bilbao, M.N.; Del Se , J. A heu is ic app oach o he mul ic i e ia design o IaaS cloud in as uc-
u es o Big Da a applica ions. Expe Sys . 2018,35, e12259. [C ossRe ]
37.
Alonso, J.; S e anidis, K.; O ue-Eche a ia, L.; Blasi, L.; Walke , M.; Escalan e, M.; López, M.J.; Du kowski, S. DECIDE: An
Ex ended De Ops F amewo k o Mul i-cloud Applica ions. In P oceedings o he 2019 3 d In e na ional Con e ence on Cloud
and Big Da a Compu ing, Ox o d, UK, 28–30 Augus 2019; Associa ion o Compu ing Machine y: New Yo k, NY, USA, 2019;
pp. 43–48.
38. Te a o m by HashiCo p. A ailable online: h ps://www. e a o m.io/ (accessed on 17 Feb ua y 2022).
39. Hea —OpenS ack. A ailable online: h ps://wiki.opens ack.o g/wiki/Hea (accessed on 17 Feb ua y 2022).
40.
Templa e S uc u e and Syn ax—Azu e Resou ce Manage —Mic oso Docs. A ailable online: h ps://docs.mic oso .com/en-
us/azu e/azu e- esou ce-manage / empla es/syn ax (accessed on 17 Feb ua y 2022).
41.
Google Cloud Deploymen Manage Documen a ion—Cloud Deploymen Manage Documen a ion. A ailable online: h ps:
//cloud.google.com/deploymen -manage /docs/ (accessed on 17 Feb ua y 2022).
42. En e p ise Kube ne es Managemen —Ranche . A ailable online: h ps:// anche .com/ (accessed on 17 Feb ua y 2022).
43. Spinnake . A ailable online: h ps://spinnake .io/ (accessed on 17 Feb ua y 2022).
44. ALIEN 4 Cloud. A ailable online: h ps://alien4cloud.gi hub.io/ (accessed on 17 Feb ua y 2022).
45.
Cloudi y O ches a ion Pla o m—Mul i Cloud, Cloud Na i e & Edge. A ailable online: h ps://cloudi y.co/ (accessed on 17
Feb ua y 2022).
46. Kube ne es. A ailable online: h ps://kube ne es.io/ (accessed on 17 Feb ua y 2022).
47.
Haja, D.; Szalay, M.; Sonkoly, B.; Pong acz, G.; Toka, L. Sha pening Kube ne es o he Edge. In P oceedings o he ACM
SIGCOMM 2019 Con e ence Pos e s and Demos on—SIGCOMM Pos e s and Demos’ 19, Beijing, China, 19–23 Augus 2019;
ACM P ess: New Yo k, NY, USA; pp. 136–137.
48.
Masip-B uin, X.; Ma ín-To de a, E.; Sánchez-López, S.; Ga cia, J.; Jukan, A.; Juan Fe e , A.; Que al , A.; Salis, A.; Ba oli, A.;
Canka , M.; e al. Managing he Cloud Con inuum: Lessons Lea n om a Real Fog- o-Cloud Deploymen . Senso s
2021
,21, 2974.
[C ossRe ]
Appl. Sci. 2022,12, 4347 23 o 24
49.
Gonzalez, N.M.; de Ca alho, T.C.M.B.; Mie s, C.C. Cloud esou ce managemen : Towa ds e icien execu ion o la ge-scale
scien i ic applica ions and wo k lows on complex in as uc u es. J. Cloud Compu . Ad . Sys . Appl. 2017,6, 13. [C ossRe ]
50.
Ismail, A.; Ca dellini, V. Towa ds Sel -adap a ion Planning o Complex Se ice-Based Sys ems. In Se ice-O ien ed Compu ing—
ICSOC 2013 Wo kshops; Lomuscio, A.R., Nepal, S., Pa izi, F., Bena allah, B., B andi´c, I., Eds.; Lec u e No es in Compu e Science;
Sp inge In e na ional Publishing: Cham, Swi ze land, 2014; Volume 8377, pp. 432–444. ISBN 978-3-319-06858-9.
51.
Push s. Pull Moni o ing Con igs. How Do Moni o ing Agen s Know—by S e e Mushe o—Medium. A ailable online:
h ps://s e e-mushe o.medium.com/push- s-pull-con igs- o -moni o ing-c541ea 9e927 (accessed on 11 Feb ua y 2022).
52.
Alonso, J.; O ue-Eche a ia, L.; Escalan e, M. Con ibu ion o he up ake o Cloud Compu ing solu ions: Design o a cloud se ices
in e media o o os e an ecosys em o us ed, in e ope able and legal complian cloud se ices. Applica ion o Mul i-Cloud
Awa e So wa e. In P oceedings o he 15 h In e na ional Con e ence on Web In o ma ion Sys ems and Technologies (WEBIST),
Vienna, Aus ia, 18–20 Sep embe 2019.
53.
Fowle , M. In as uc u e as Code. A ailable online: h ps://ma in owle .com/bliki/In as uc u eAsCode.h ml (accessed on 29
Ma ch 2021).
54.
Na u, M.; Ghosh, R.K.; Shyamsunda , R.K.; Ranjan, R. Holis ic Pe o mance Moni o ing o Hyb id Clouds: Complexi ies and
Fu u e Di ec ions. IEEE Cloud Compu . 2016,3, 72–81. [C ossRe ]
55. Dus da , S.; Sch eine , W. A su ey on web se ices composi ion. In . J. Web G id Se . 2005,1, 1. [C ossRe ]
56.
Lei ne , P.; Michlmay , A.; Rosenbe g, F.; Dus da , S. Moni o ing, P edic ion and P e en ion o SLA Viola ions in Composi e
Se ices. In P oceedings o he 2010 IEEE In e na ional Con e ence on Web Se ices, Miami, FL, USA, 5–10 July 2010; pp. 369–376.
57.
I ano i´c, D.; Ca o, M.; He menegildo, M. Cons ain -Based Run ime P edic ion o SLA Viola ions in Se ice O ches a ions. In
Se ice-O ien ed Compu ing; Kappel, G., Maama , Z., Mo aha i-Nezhad, H.R., Eds.; Lec u e No es in Compu e Science; Sp inge :
Be lin/Heidelbe g, Ge many, 2011; Volume 7084, pp. 62–76. ISBN 978-3-642-25534-2.
58.
Lei ne , P.; We zs ein, B.; Rosenbe g, F.; Michlmay , A.; Dus da , S.; Leymann, F. Run ime P edic ion o Se ice Le el Ag eemen Vi-
ola ions o Composi e Se ices. In Se ice-O ien ed Compu ing—ICSOC 2007; K äme , B.J., Lin, K.-J., Na asimhan, P., Eds.; Lec u e
No es in Compu e Science; Sp inge : Be lin/Heidelbe g, Ge many, 2010; Volume 4749, pp. 176–186. ISBN 978-3-540-74973-8.
59.
De F ancisci Mo ales, G.; Bi e , A.; Khan, L.; Gama, J.; Fan, W. IoT Big Da a S eam Mining. In P oceedings o he 22nd ACM
SIGKDD In e na ional Con e ence on Knowledge Disco e y and Da a Mining, San F ancisco, CA, USA, 13–17 Augus 2016;
pp. 2119–2120.
60. Ba os, R.S.M.; San os, S.G.T.C. A la ge-scale compa ison o concep d i de ec o s. In . Sci. 2018,451–452, 348–370. [C ossRe ]
61.
Lu, J.; Liu, A.; Song, Y.; Zhang, G. Da a-d i en decision suppo unde concep d i in s eamed big da a. Complex In ell. Sys .
2020,6, 157–163. [C ossRe ]
62.
Rahman, A.; Elde , S.; Shezan, F.H.; F os , V.; S allings, J.; Williams, L. Ca ego izing De ec s in In as uc u e as Code. a Xi
2018
,
a Xi :1809.07937.
63.
Xu, X.; Zhu, L.; Webe , I.; Bass, L.; Sun, D. POD-Diagnosis: E o Diagnosis o Spo adic Ope a ions on Cloud Applica ions. In
P oceedings o he 2014 44 h Annual IEEE/IFIP In e na ional Con e ence on Dependable Sys ems and Ne wo ks, A lan a, GA,
USA, 23–26 June 2014; pp. 252–263.
64.
Anga i a, R.; Manou ie , M.; Rukoz, M. An Agen A chi ec u e o Enable Sel -healing and Con ex -awa e Web o Things
Applica ions. In P oceedings o he In e na ional Con e ence on In e ne o Things and Big Da a, Rome, I aly, 23–25 Ap il 2016;
Science and and Technology Publica ions: Se úbal, Po ugal, 2016; pp. 82–87.
65.
Adel Se hani, M.; El-Kassabi, H.T.; Shuaib, K.; Na az, A.N.; Bena allah, B.; Behesh i, A. Sel -adap ing cloud se ices o ches a ion
o ul illing in ensi e senso y da a-d i en IoT wo k lows. Fu u e Gene . Compu . Sys . 2020,108, 583–597. [C ossRe ]
66.
Gill, S.S.; Chana, I.; Singh, M.; Buyya, R. RADAR: Sel -con igu ing and sel -healing in esou ce managemen o enhancing quali y
o cloud se ices. Concu . Compu . P ac . Exp. 2019,31, e4834. [C ossRe ]
67.
Gao, H.; Huang, W.; Yang, X.; Duan, Y.; Yin, Y. Towa d se ice selec ion o wo k low econ igu a ion:An in e ace-based
compu ing solu ion. Fu u e Gene . Compu . Sys . 2018,87, 298–311. [C ossRe ]
68.
To e i, G.; B unne , S.; Blöchlinge , M.; Spillne , J.; Bohne , T.M. Sel -managing cloud-na i e applica ions: Design, implemen a-
ion, and expe ience. Fu u e Gene . Compu . Sys . 2017,72, 165–179. [C ossRe ]
69.
El-Kassabi, H.T.; Adel Se hani, M.; Dssouli, R.; Na az, A.N. T us en o cemen h ough sel -adap ing cloud wo k low o ches a ion.
Fu u e Gene . Compu . Sys . 2019,97, 462–481. [C ossRe ]
70.
Achie ing Applica ion Po abili y wi h Kube ne es & Cloud Na i e. A ailable online: h ps://kubl .com/blog/applica ion-
po abili y-wi h-kube ne es-and-cloud-na i e/ (accessed on 16 Feb ua y 2022).
71.
Da a G a i y—In he Clouds—Da a G a i as. A ailable online: h ps://da ag a i as.com/2010/12/07/da a-g a i y-in- he-
clouds/ (accessed on 16 Feb ua y 2022).
72.
S a e ul Se ices—The Black Sheep o he Con aine Wo ld|D2iQ. A ailable online: h ps://d2iq.com/blog/s a e ul-se ices-
black-sheep-con aine -wo ld (accessed on 16 Feb ua y 2022).
73.
Balouek-Thome , D.; Rena , E.G.; Zamani, A.R.; Simone , A.; Pa asha , M. Towa ds a compu ing con inuum: Enabling
edge- o-cloud in eg a ion o da a-d i en wo k lows. In . J. High Pe o m. Compu . Appl. 2019,33, 1159–1174. [C ossRe ]
74.
Ba esi, L.; Mendonça, D.F.; Ga iga, M.; Guinea, S.; Qua occhi, G. A Uni ied Model o he Mobile-Edge-Cloud Con inuum.
ACM T ans. In e ne Technol. 2019,19, 29. [C ossRe ]
Appl. Sci. 2022,12, 4347 24 o 24
75.
Eu opean Commission. Wo k P og amme 2018–2020. A ailable online: h ps://ec.eu opa.eu/ esea ch/pa icipan s/da a/ e /h2
020/wp/2018-2020/main/h2020-wp1820-lei -ic _en.pd (accessed on 12 Feb ua y 2022).
76.
Shi, W.; Cao, J.; Zhang, Q.; Li, Y.; Xu, L. Edge Compu ing: Vision and Challenges. IEEE In e ne Things J.
2016
,3, 637–646.
[C ossRe ]
77.
JHips e —Full S ack Pla o m o he Mode n De elope . A ailable online: h ps://www.jhips e . ech/ (accessed on 11
Ma ch 2022).
78.
Deb, K.; P a ap, A.; Aga wal, S.; Meya i an, T. A as and eli is mul iobjec i e gene ic algo i hm: NSGA-II. IEEE T ans. EVol.
Compu . 2002,6, 182–197. [C ossRe ]
79.
MOEA F amewo k—A Ja a Lib a y o Mul iobjec i e E olu iona y Algo i hms. A ailable online: h p://moea amewo k.o g/
(accessed on 11 Ap il 2022).
80.
Teleg a Open Sou ce Se e Agen —In luxDB. A ailable online: h ps://www.in luxda a.com/ ime-se ies-pla o m/ eleg a /
(accessed on 11 Ma ch 2022).
81. In luxDB: Open Sou ce Time Se ies Da abase. A ailable online: h ps://www.in luxda a.com/ (accessed on 11 Ap il 2022).
82.
G a ana Cloud—G a ana Labs. A ailable online: h ps://go2.g a ana.com/g a ana-cloud.h ml?s c=ggl-s&mdm=cpc&camp=b-
g a ana-exac-emea&cn =118483912276& m=g a ana&de ice=c&gclid=CjwKCAjw6dmSBhBkEiwA_W-EoMb226-I3u B 952
aGsX0ocblLsqRjX8W7hia68oxUNGmdBgIdVgBBoCK8cQA D_BwE (accessed on 13 Ap il 2022).
