Deep Reinforcement Learning for the Management of Software-Defined Networks and Network Function Virtualization in an Edge-IoT Architecture

sus ainabili y A icle Deep Rein o cemen Lea ning o he Managemen o So wa e-De ined Ne wo ks and Ne wo k Func ion Vi ualiza ion in an Edge-IoT A chi ec u e Rica do S. Alonso1,* , Inés Si ón-Candanedo 1, Robe o Casado-Va a 1, Ja ie P ie o 1,2 and Juan M. Co chado 1,2,3,4 1BISITE Resea ch G oup, Uni e si y o Salamanca, Edi icio Mul iusos I+D+i, Calle Espejo 2, 37007 Salamanca, Spain; [email p o ec ed] (I.S.-C.); [email p o ec ed] (R.C.-V.); [email p o ec ed] (J.P.); [email p o ec ed] (J.M.C.) 2AIR Ins i u e, Edi icio Pa que Cien í ico, Módulo 305, Paseo de Belén 11, Campus Miguel Delibes, 47011 Valladolid, Spain 3Depa men o Elec onics, In o ma ion and Communica ion, Facul y o Enginee ing, Osaka Ins i u e o Technology, 5-16-1 Omiya, Asahi-ku, Osaka 535-8585, Japan 4Pusa Kompu e an dan In o ma ik, Uni e si i Malaysia Kelan an, Bachok 16300, Malaysia *Co espondence: [email p o ec ed] Recei ed: 31 May 2020; Accep ed: 10 July 2020; Published: 15 July 2020   Abs ac : The In e ne o Things (IoT) pa adigm allows he in e connec ion o millions o senso de ices ga he ing in o ma ion and o wa ding o he Cloud, whe e da a is s o ed and p ocessed o in e knowledge and pe o m analysis and p edic ions. Cloud se ice p o ide s cha ge use s based on he compu ing and s o age esou ces used in he Cloud. In his ega d, Edge Compu ing can be used o educe hese cos s. In Edge Compu ing scena ios, da a is p e-p ocessed and il e ed in ne wo k edge be o e being sen o he Cloud, esul ing in sho e esponse imes and p o iding a ce ain se ice le el e en i he link be ween IoT de ices and Cloud is in e up ed. Mo eo e , he e is a g owing end o sha e physical ne wo k esou ces and cos s h ough Ne wo k Func ion Vi ualiza ion (NFV) a chi ec u es. In his sense, and ela ed o NFV, So wa e-De ined Ne wo ks (SDNs) a e used o econ igu e he ne wo k dynamically acco ding o he necessi ies du ing ime. Fo his pu pose, Machine Lea ning mechanisms, such as Deep Rein o cemen Lea ning echniques, can be employed o manage i ual da a lows in ne wo ks. In his wo k, we p opose he e olu ion o an exis ing Edge-IoT a chi ec u e o a new imp o ed e sion in which SDN/NFV a e used o e he Edge-IoT capabili ies. The p oposed new a chi ec u e con empla es he use o Deep Rein o cemen Lea ning echniques o he implemen a ion o he SDN con olle . Keywo ds: indus ial in e ne o hings; edge compu ing; so wa e de ined ne wo ks; ne wo k unc ion i ualiza ion; deep ein o cemen lea ning 1. In oduc ion Technologies such as In e ne o Things (IoT),Indus ial In e ne o Things (especially obus and aul ole an IoT de ices) and Cybe -Physical Sys ems allow millions o senso and ac ua o de ices ha in e ac wi h he con ex o use s [ 1 ]. These da a is usually managed by he Cloud, whe e digi al wins ep esen physical en i ies. Da a is s o ed in Big Da a eposi o ies and p ocessed o ex ac knowledge and o ecas he s a e o en i ies and con ex in he u u e. Fo his pu pose, di e en a i icial in elligence echniques a e applied in he Cloud, including mul i-agen sys ems [ 2 ], o Machine Lea ning/Deep Lea ning echniques [ 3 ]. IoT, IIoT and CPS pa adigms a e used in many di e en applica ions, such as heal hca e, sma ene gy, sma a ming, o indus y 4.0, among many o he s [ 4 ]. Sus ainabili y 2020,12, 5706; doi:10.3390/su12145706 www.mdpi.com/jou nal/sus ainabili y Sus ainabili y 2020,12, 5706 2 o 23 The e a e scena ios whe e a solu ion consis ing o only one IoT laye and one Cloud laye may ha e ce ain d awbacks. The dependency on he Cloud as a da a s o e and applica ion p o ide implies ha an in e up ion in he link be ween he IoT laye and he Cloud laye also in e up s he se ice [ 5 ]. Mo eo e , Cloud se ice p o ide s o e p icing plans based on he amoun o esou ces ha cus ome s use du ing ime . In his sense, he Edge Compu ing pa adigm a ises o educe he cos associa ed wi h he ans e , s o age and p ocessing o da a in he Cloud. In his way, da a is il e ed and p e-p ocessed in he edge o he ne wo k be o e being sen o he Cloud, allowing he applica ion o machine lea ning echniques in he same edge, ob aining sho e esponse imes and main aining sys em unc ionali ies e en du ing communica ion b eaks be ween he IoT laye and he Cloud laye [6]. On he o he hand, he e is a g owing end o sha e physical ne wo k esou ces and cos s by di e en use en i ies h ough Ne wo k Func ion Vi ualiza ion (NFV) [ 7 ]. To educe he main enance cos s o hese physical esou ces and make he con igu a ion o he ne wo k mo e lexible o e ime, So wa e-De ined Ne wo ks (SDN) echniques a e also used, which a e closely ela ed o NFV [ 8 ]. Thanks o SDN, i is possible o use gene al pu pose COTS (Comme cial O -The-Sel ) elemen s ha can be econ igu ed o e ime acco ding o he changing needs o he ne wo k, ins ead o deploying speci ic elemen s (e.g., swi ches, b idges o ou e s) ha ha e o be eplaced when he ne wo k g ows o changes [9]. In his sense, in elligen mechanisms a e needed o e icien ly op imize i ual da a lows in ne wo ks acco ding o he Quali y o se ice (QoS) equi ed by each applica ion. Deep Rein o cemen Lea ning is one o he mos p omising ends in ecen yea s in he ield o machine lea ning aimed a con ol [ 10 ]. Among i s applica ions a e sel -d i ing ca s In his ype o applica ion, we will call as “agen ” a sel -d i ing ca , chess i ual playe o an Edge node in an IoT ne wo k edi ec ing ne wo k a ic and de ending i sel om mul iple sou ces o cybe a acks. None heless, he agen canno s o e and con empla e all he possible scena ios in which i can be ound due o he eno mous numbe o possibili ies. Thus, he agen explo es i s en i onmen knowing he possible s a es, he p obabili y o changing be ween hose s a es, he se o ac ions i can pe o m in each o hose s a es and he ewa d o penal y i will ge in each case. In his wo k, we p opose he e olu ion o an exis ing Edge-IoT a chi ec u e o a new imp o ed e sion in which SDN/NFV a e used o e he Edge-IoT capabili ies. Fo his pu pose, he Global Edge Compu ing A chi ec u e (GECA) has been aken as a basis, o ien ed o he implemen a ion o Edge-IoT solu ions and p esen ed by Si ón-Candanedo e al. [ 11 ]. The Global Edge Compu ing A chi ec u e has al eady been applied, in ac , in Sma Ene gy [ 12 ] and Sma Fa ming [ 6 ] scena ios whe e i was necessa y o educe da a ans e be ween he IoT and he Cloud and, he e o e, he cos s o compu ing and s o age in he Cloud. Howe e , in i s p e ious e sion i did no ha e SDN/NFV capabili ies, which ha e been explo ed by Alonso e al. [ 13 ] o i s cu en inco po a ion o he a chi ec u e. The new p oposed a chi ec u e con empla es he use o Deep Rein o cemen Lea ning echniques o he implemen a ion o he SDN con olle . In his sense, i is p oposed he applica ion o a Deep Q-Lea ning model [ 14 ] o he managemen o i ual da a lows in SDN/NFV in an Edge-IoT a chi ec u e acco ding o he equi ed quali y o se ice. The es o his wo k is s uc u ed as ollows. The nex sec ion (Sec ion 2) desc ibes he p oblem o be sol ed along wi h he s a e o he a in he ield o In e ne o Things and Edge Compu ing solu ions, he di e en app oaches when using Ne wo k Func ion Vi ualiza ion and So wa e-De ined Ne wo ks o Edge Compu ing scena ios, as well as he applica ion o Deep Rein o cemen Lea ning o da a low managemen in ne wo ks. A e ha , Sec ion 3p esen s he Global Edge Compu ing A chi ec u e and he new mechanism p oposed o manage da a lows in So wa e De ined Ne wo ks based on GECA. Then, Sec ion 4desc ibes he expe imen a ion and he ob ained esul s a e implemen ing he GECA 2.0 SDN mechanism based on Deep Q-Ne wo ks. Finally, conclusions and u u e wo k a e p esen ed in Sec ion 5. Sus ainabili y 2020,12, 5706 3 o 23 2. P oblem Desc ip ion and Rela ed Wo k In his sec ion we will p og essi ely desc ibe he p oblem o be sol ed. In o de o do his, we will make an analysis o he ela ed wo k exis ing in he ield o he In e ne o Things and Indus ial In e ne o Things in Sec ion 2.1. A e ha , in Sec ion 2.2 we will desc ibe wha ad an ages he Edge Compu ing pa adigm p o ides o e solu ions based only on IoT and Cloud laye s. On he o he hand, Sec ion 2.3 will desc ibe how So wa e-De ined Ne wo king and Ne wo k Func ion Vi ualiza ion can help in Edge-IoT scena ios. Finally, Sec ion 2.4 will p esen Rein o cemen Lea ning and Deep Rein o cemen Lea ning echniques o manage ne wo k i ualiza ion in Edge-IoT app oaches. 2.1. Challenges in In e ne o Things and Indus ial In e ne o Things Scena ios The In e ne o Things (IoT) is, in pa , an e olu ion o o he concep s such as Cybe -Physical Sys ems, Senso Ne wo ks and Wi eless Senso Ne wo ks [ 15 ], among o he s. Wi eless Senso Ne wo ks allow us o collec in o ma ion abou di e en physical measu emen s o he use s’ en i onmen and, some imes, o he use s hemsel es [ 16 ]. WSNs ha e been applied in mul iple scena ios such as heal hca e and eleca e [ 17 ], sma buildings [ 18 ] o sma a ming [ 6 ]. As can be seen, o e ime he e ha e been and a e mul iple wi eless echnologies o implemen WSNs. In ac , we o en ind scena ios whe e se e al echnologies a e used a he same ime because he e a e se e al elemen s o be moni o ed (e.g., people, objec s, he en i onmen , e c.), so we alk abou He e ogeneous Wi eless Senso Ne wo ks [ 17 ]. To accommoda e hese necessa y he e ogenei y, di e en app oaches ha e been employed in e ms o amewo ks and pla o ms o cohabi a ion and in o ma ion usion [ 15 ]. None heless, he need o connec an inc easing numbe o senso s o di e en ypes and moni o and s o e in o ma ion in he Cloud, leads us inexo ably o he In e ne o Things. Howe e , i was no un il 1999 ha he e m In e ne o Things was i s men ioned by Ke in Ash on [ 19 ]. In e es in IoT g ew as la ge companies and go e nmen s saw i as a key echnology. In his sense, Google began o s o e da a ela ed o he Wi-Fi ne wo ks o he use s o i s se ices in 2010. Tha same yea , he Chinese go e nmen es ablished he IoT as a p io i y opic wi hin i s i e-yea plan [ 20 ]. O e he yea s he e minology ook on i s own o mal de ini ions, such as ha o Ke ha eswa an and Ram [21] , which de ined he In e ne o Things as he connec ion o objec s (buildings, ehicles) h ough a ne wo k in as uc u e wi h elec onic elemen s (senso s, ac ua o s, adio equency iden i ica ion ags, e c.) o collec and exchange da a. The mos impo an IoT applica ion a eas include heal hca e [ 17 ], anspo and logis ics [ 22 ], sma homes [ 23 ], sma ene gy [ 4 ], sma ci ies [ 24 ] o Indus y 4.0 [ 25 ], among many o he s. On he o he hand, he Indus ial In e ne o Things (IIoT) [ 26 ] is one o he undamen al aspec s o he new Indus y 4.0 [ 27 ]. The IIoT eme ges as a new miles one on he In e ne so ha billions o machines in he indus y a e equipped wi h a ious ypes o senso s, connec ed o he In e ne h ough he e ogeneous ne wo ks [ 28 ]. Possible applica ions o he IIoT include abno mal pa e n de ec ion [ 29 ] o p edic i e main enance [ 25 ], among many o he s. Al hough he implemen a ion o IoT inges ion laye s allows o deal wi h he p oblem o he e ogenei y, wo o he p oblems emain o be sol ed. One o hem is he la ge amoun o da a ha housands o de ices can send o an IoT pla o m. To educe he da a a ic be ween he IoT laye and he cloud, solu ions such as he Edge Compu ing pa adigm a ise. Edge Compu ing allows educing conges ion by he demand o compu ing esou ces, ne wo k o s o age in he cloud. Wi h his s a egy, compu a ional and se ice in as uc u es app oach he end use by mig a ing da a il e ing, p ocessing o s o age om he cloud o he edge o he ne wo k [ 30 , 31 ]. The second challenge o be sol ed is ha , ega dless o he ansmission echnology used, he e may be mul iple di e en use o applica ions using a common ansmission in as uc u e om he IoT o he Cloud. Sha ing ne wo k esou ces and in as uc u e by a ious o ganiza ions, ope a o s and use g oups can be done h ough Ne wo k Func ion Vi ualiza ion and So wa e-De ined Ne wo ks [13]. Sus ainabili y 2020,12, 5706 4 o 23 2.2. Edge Compu ing and Edge-IoT Pla o ms The Edge Compu ing pa adigm allows us o ake pa o he compu a ional load om he Cloud o he Edge nodes [ 30 ]. Shi e al. [ 32 ] de ine Edge compu ing as compu e and ne wo k esou ces loca ed be ween da a sou ces, such as IoT de ices, and cloud da a cen e s. In hese edge nodes i is possible o il e he in o ma ion coming om he IoT laye o he Cloud, hus educing he cos o compu ing and s o age se ices in he Cloud. Edge nodes can p e-p ocess he in o ma ion collec ed om he IoT nodes, hus educing he in o ma ion o be p ocessed and s o ed in he Cloud [ 11 ]. In addi ion, i is possible o execu e Machine Lea ning o e en Deep Lea ning algo i hms in he Edge nodes, so ha a mo e di ec se ice can be gi en o use s, educing esponse imes [ 6 ]. I is also possible o con inue p o iding se ice empo a ily du ing communica ion b eaks wi h he Cloud [ 12 ]. Figu e 1shows he basic scheme o an edge compu ing a chi ec u e whe e he edge nodes allow use applica ion p ocesses o be execu ed close o he da a sou ces [ 30 ]. The edge nodes pe o m compu e asks such as il e ing, p ocessing, caching, load balancing by educing da a sen o ecei ed om he cloud and eques ing se ices and in o ma ion [13]. Figu e 1. Edge Compu ing a chi ec u e, based on he wo k o Yu e al. [30]. The e is a wide a ie y o scena ios whe e solu ions based on IoT and Edge Compu ing a e being applied. Among he mos ele an applica ions, we ind Sma Fa ming [ 6 ], Sma Ene gy [ 33 ] o Indus y 4.0 solu ions [ 11 ], among many o he s. Mo eo e , al hough he e a e scena ios in which Edge Compu ing is applied o a single en i onmen as an ad-hoc sys em, he e a e also de elopmen s aimed a p o iding Edge unc ionali ies as a pla o m. In his way, he ep oducibili y o he solu ion is inc eased. Likewise, he e a e di e en e e ence a chi ec u es wi hin he scope o Edge Compu ing applied in indus ial en i onmen s o Indus y 4.0. In ac , one o hem is he Global Edge Compu ing A chi ec u e [ 11 ], on which his wo k has been based. This a chi ec u e, in u n, was he esul o analyzing ou o he mos impo an e e ence a chi ec u es in he ield o Edge-IoT in Indus y 4.0. The i s o hese a chi ec u es is FAR-Edge [ 34 ]. One o he aspec s aken by GECA om FAR-Edge is ha bo h inco po a e blockchain unc ionali ies. Howe e , in he case o FAR-Edge he blockchain is Sus ainabili y 2020,12, 5706 5 o 23 inco po a ed in i s in e media e laye , while in GECA i is implemen ed om he base laye , ha is, a he same ime ha he da a a e gene a ed by he IoT senso s. Ano he a chi ec u e ha has se ed as a e e ence o GECA is INTEL-SAP Re e ence A chi ec u e [ 35 ]. GECA is inspi ed by he SAP Cloud T us Cen e concep , which aims o e i y he owne ship o de ices and egis e hem in he sys em o pla o m implemen ed, assigning a ce i ica e o iden i y (au hen ici y) o each owne and keeping an upda ed lis o success ully egis e ed de ice. In he case o GECA his p ocess is done in he Cloud, be o e a new de ice can access a GECA based sys em o pla o m. Thi dly, he e is he a chi ec u e o he Edge Compu ing Conso ium [ 36 ]. GECA is based on his a chi ec u e in e ms o he impo ance o ollowing a s anda d. The e o e, GECA’s design is based on he IEC 61499 s anda d [ 37 ], ollowing a s uc u e o unc ional blocks, each wi h i s co esponding inpu s, p ocesses and ou pu s. Finally, he a chi ec u e o he Indus ial In e ne Conso ium (IIC) p oposes he inclusion o an En e p ise Laye , on which GECA’s Business Solu ion Laye is based [38]. The e a e wo main models when designing Edge Compu ing solu ions [ 30 ]. In he hie a chical model, he edge a chi ec u e is di ided in o a hie a chy in which unc ions a e de ined based on dis ance and esou ces. In he hie a chical model, he e o e, he di e en edge and cloudle se e s a e deployed a di e en dis ances om he end use s. Examples o he hie a chical model a e he wo k o Ja a weh e al. [39] , who p opose a model based on he in eg a ion o cloudle s and Mobile Edge Compu ing (MEC) se e s. The GECA a chi ec u e on which his wo k is based also ollowed he hie a chical model be o e he upda e p oposed in his pape [ 11 ]. The second one is he So wa e-De ined model [8], desc ibed in he nex sec ion. 2.3. So wa e-De ined Ne wo king and Ne wo k Func ion Vi ualiza ion in Edge-IoT Scena ios In o de o p o ide a mo e e icien use o esou ces in IoT ne wo ks, new solu ions a e eme ging o i ualize esou ces [ 40 ]. In his sense, concep s such as Ne wo k Func ion Vi ualiza ion (NFV) a ise, o ien ed o he i ualiza ion o he di e en componen s o he ne wo k [7]. In ac , ETSI MEC (Mobile Edge Compu ing) is based on he NFV concep , and i s applica ion in IoT scena ios has been explo ed widely [ 41 ]. In a way closely ela ed o he NFV, and o en used complemen ing each o he , So wa e-De ined Ne wo ks also eme ge [ 8 ]. App oaches such as So wa e-De ined Ne wo ks and Ne wo k Func ion Vi ualiza ion enable cos sa ings by employing gene al pu pose de ices a he han mo e expensi e ne wo k-speci ic de ices ha may need o be eplaced i ne wo k con igu a ion needs change o e ime [9]. Figu e 2shows he ypical a chi ec u e o So wa e-De ined Ne wo ks [ 42 ]. In his ype o a chi ec u e, he ne wo k is sepa a ed in o a da a plane and a con ol plane [ 43 ]. The da a plane consis s o COTS o wa ding nodes wi h gene al pu pose capabili ies, a he han speci ic pu pose ha dwa e (e.g., ou e s, ga eways, e c.) [ 9 ]. The emo e con igu a ion o he nodes is done om a con olle (i.e., SDN Con olle ) in he con ol plane. In his way, he ne wo k adminis a o uses a cen alized con ol console o econ igu e he a ic low on he ne wo k wi hou ha ing o physically modi y he ne wo k nodes [ 42 ]. Fo his pu pose, all hese nodes ha e a common in e ace o emo e con igu a ion, called sou hbound in e ace [ 44 ]. Sou hbound in e aces allow abs ac ing he unc ionali y o he COTS o wa ding nodes and allow he con olle o in e ac wi h hem. In his ega d, OpenFlow is he mos ep esen a i e o sou hbound in e aces [8]. The open-sou ce OpenFlow p o ocol allows decoupling he con ol plane o he da a plane o he ne wo ks. Thanks o he uncoupling o hese wo planes, he con ol and managemen o he ne wo k can be ca ied ou emo ely in he cloud (in a cen alized o dis ibu ed way), while packe o wa ding is ca ied ou in he ha dwa e de ices ha make up he ne wo k [ 8 ]. The con ol plane commands ha dwa e de ices speci ying how o o wa d hese packe s be ween hei adjacen nodes. The e o e, i is no longe necessa y o build di e en ha dwa e de ices wi h speci ic unc ions (ASIC ci cui s) [ 7 ], making i possible o use cheape gene al-pu pose ha dwa e, wi h a lowe uni cos and ha can e ol e o e ime simply upda ing i s unc ions and so wa e emo ely, educing eplacemen and wa ehousing cos s. Fu he mo e, in he con ol plane o a SDN, a Ne wo k Ope a ing Sys em (NOS) is Sus ainabili y 2020,12, 5706 6 o 23 unning o e sou hbound and no hbound in e aces [ 45 ]. NOS desc ibe he a ailable so wa e-de ined ne wo king so wa e ools. O e he speci ic NOS, is possible o build applica ions aimed a con olling he ne wo k beha io and de ine high-le el ne wo k policies (i.e., he managemen plane) [ 8 ]. Fo his pu pose, no hbound in e aces allow hese applica ions o in e ac wi h he NOS. A human ne wo k adminis a o uses a cen alized con ol console o econ igu e he a ic low on he ne wo k wi hou ha ing o physically modi y he ne wo k nodes [ 42 ]. Howe e , he adminis a ion o he ne wo k can be ca ied ou in an au oma ed way by speci ic so wa e such as a cogni i e engine based on in elligen algo i hms. This is, in ac , one o he objec i es o his wo k. Examples o his ype o app oach a e he wo k o Baek e al. [46], o he wo k o Sampaio e al. [47], as we discuss la e in Sec ion 2.4. Figu e 2. So wa e De ined Ne wo k (SDN) a chi ec u e. The concep o ne wo k i ualiza ion is de ined acco ding o G anelli e al. [ 7 ] as he p ocess o combining ha dwa e and so wa e ne wo k esou ces, as well as he ne wo k’s own unc ionali ies, in o a single so wa e-based en i y ha is called a i ual ne wo k. Figu e 3shows his app oach [ 42 ]. As men ioned abo e, he in eg a ion o SDN and NFV is common in he ield o scien i ic and indus ial esea ch o ake ad an age o he bene i s o bo h [ 43 ]. Jus as he e a e in elligen algo i hms aimed a balancing esou ces, con igu ing ou ing and i ewall ules emo ely om he managemen plane using he no hbound in e ace in SDN scena ios, he e a e also esea ches ha p opose in elligen mechanisms aimed a p o isioning esou ces in NFVs, such as he wo k o Ruiz e al. [ 48 ] o Pei e al. [ 49 ]. Again, ou wo k is ocused on p o ide wi h new in elligen mechanisms ha allow p o isioning i ual ne wo k esou ces in SDN and NFV scena ios. Sus ainabili y 2020,12, 5706 7 o 23 Figu e 3. Ne wo k Func ion Vi ualiza ion (NFV) o e So wa e-De ined Ne wo ks. The e a e s udies in So wa e-De ined Ne wo ks [ 50 ], Wi eless So wa e-De ined Ne wo ks [ 51 ] and Ne wo k Func ion Vi ualiza ion [ 52 ] as complemen a y echnologies ha could wo k oge he wi h Edge Compu ing a chi ec u es. Ja a weh e al. [ 53 ] p opose a so wa e-de ined model aimed a he in eg a ion o Mobile Edge Compu ing (MEC) and SDN. Salman e al. [ 54 ] go one s ep u he , p oposing he in eg a ion o MEC, SDN and NFV, achie ing a be e MEC pe o mance in mobile ne wo ks and ha can be u he ex ended enabling IoT-wide deploymen scena ios. Mo e speci ically, he e a e di e en solu ions aimed a combining bo h pa adigms o u he op imize esou ces in IoT ne wo ks [ 55 ]. Wi hin he di e en app oaches, we ind HomeCloud [ 56 ], a amewo k ha combines he use o SDN and NFV wi h he aim o allowing e icien o ches a ion and deli e y o applica ions om he se e s ha a e deployed on he Edge i sel . Ca aguay e al. [ 40 ] p opose a SDN/NFV a chi ec u e speci ically o IoT ne wo ks and ocused on modi ying QoE/QoS lows in eal- ime by means o he con olle capabili ies. Likewise, Mon a ed e al. [ 57 ] p opose a wo- ie cloud a chi ec u e. In Mon a ed e al. [57] a chi ec u e, on he one hand, he e a e da a se e s in he cloud and, on he o he , he e a e edge de ices o p o ide da a close o use s. Fo he con ol and managemen o he a chi ec u e, a ne wo k in as uc u e de ined by he so wa e is p oposed. None heless, i is necessa y o ind inno a i e solu ions ha design and implemen in elligen algo i hms ha allow au oma ed asks such as esou ce balancing, ne wo k econ igu a ion o cybe a ack p e en ion in scena ios whe e IoT, Edge Compu ing and SDN/NFV a e combined. In his sense, he e a e some solu ions like he p oposal o Ruiz e al. [ 48 ], who p oposes a gene ic algo i hm o VNF p o isioning in NFV-Enabled Cloud/MEC RAN a chi ec u es. Howe e , among he in elligen algo i hms one o he mos p omising ields in ecen yea s is Rein o cemen Lea ning. This is he basis o app oaches such as ha o He e al. [ 58 ], which p oposes a deep ein o cemen lea ning app oach in SDN wi h MEC. In he nex sec ion we will b ie ly analyze he s a e o he a o he use o Deep Rein o cemen Lea ning in SDN/NFV scena ios. Sus ainabili y 2020,12, 5706 8 o 23 2.4. Rein o cemen Lea ning and Deep Rein o cemen Lea ning in SDN/NFV Scena ios Rein o cemen lea ning (and Deep Rein o cemen Lea ning) is one o he mos p omising ends in ecen yea s in he ield o machine lea ning [ 10 ]. Among i s applica ions a e sel -d i ing ca s [ 59 ], obo acuums [ 60 ], au oma ed ading [ 61 ], en e p ise esou ce managemen [ 62 , 63 ] o games. In his sense, in 2017 was eleased AlphaGo Ze o, an algo i hm ha was no ained by human compe i o da a. AlphaGo Ze o was only able o sense posi ions on he boa d, a he han ha ing p e ious da a. AlphaGo Ze o also an on a single machine wi h 4 TPUs and bea p e ious AlphaGo Lee [ 64 ] in h ee days by 100 games o 0. AlphaGo Ze o’s neu al ne wo k was p e iously ained using Tenso Flow wi h 64 GPU wo ke s and 19 CPU pa ame e se e s [ 65 ]. As can be seen, he po en ial o Deep Rein o cemen Lea ning in scena ios whe e he capaci y o ini ial aining da ase is no a ailable is eno mous. This allows en isioning solu ions in which o build de ices ha lea n om sc a ch he bes way o balance da a lows in a so wa e de ined ne wo k aking in o accoun he di e en QoS (Quali y o Se ice) and op imizing he use o esou ces [ 58 ], o op imize he use o ene gy in edge scheduling [ 66 ]. Likewise, hey ep esen algo i hms wi h a g ea po en ial o de ec new cybe a acks no ye known wi hou he need o ain new models as new h ea s a ise [67]. The Q-lea ning algo i hm [ 68 ] is one o he bes known model- ee echniques in ein o cemen lea ning, and has nume ous e olu ions and a ian s o he same [ 69 ]. Fo any Fini e Ma ko Decision P ocesses (FMDP), Q-lea ning (Qcomes om Quali y) inds a policy ha is op imal in he sense ha i maximizes he expec ed alue o he o al ewa d o e any and all successi e s eps, s a ing om he cu en s a e [ 70 ]. Thus, we ha e a alue unc ion Q(s , a) ha akes as inpu he cu en s a e s and he cu en ac ion a and e u ns he expec ed ewa d o ha ac ion and all subsequen ac ions. Ini ially, Q e u ns he same a bi a y alue. As he agen explo es he en i onmen , Q e u ns an inc easingly be e app oxima ion o he alue o an a ac ion, gi en a s s a e. Tha is, he unc ion Q is p og essi ely upda ed. So, he new alue Qnew(s , a ) is upda ed in each i e a ion om he old alue Q(s , a ) wi h a lea ning a e α , since he lea ned alue ( +γ·maxaQ(s +1 , a)) is known, whe e is he ewa d, γ he discoun ac o and maxaQ(s +1,a) he es ima e o he u u e op imal alue: Qnew(s ,a )←(1−α)Q(s ,a ) + α +γ·max aQ(s +1,a). (1) In his way, he agen has an es ima ed alue o each s a e-ac ion pai , and whose knowledge inc eases wi h ime. Wi h his in o ma ion, he agen can selec which ac ion o ca y ou in each momen acco ding o i s ac ion-selec ion s a egy, which can ake in o accoun an epsilon-g eedy s a egy, o example, o inc ease i s knowledge o he en i onmen . Deep lea ning has accele a ed p og ess in Rein o cemen Lea ning, wi h he use o deep lea ning algo i hms wi hin he RL ha de ine he ield o Deep Rein o cemen Lea ning [ 69 ]. Deep Lea ning allows Rein o cemen Lea ning o scale up o p e iously in ac able decision-making p oblems, ha is, en i onmen s wi h high dimensional s a es and ac ion spaces. The e a e se e al Deep Rein o cemen Lea ning algo i hms [ 69 ], including he a ian s o Deep Q-Lea ning (DQN), such as double DQN [ 66 ], dueling DQN, Deep Recu en Q-Ne wo ks (DQRN) and mul i-s ep DQN, among o he s, and hose based on policy g adien , such as REINFORCE, Ad an age Ac o -C i ic (A2C), Na u al Policy G adien (NPG), among many o he s. Thus, deep ein o cemen lea ning echniques p o ide g ea po en ial in IoT, edge and SDN scena ios. In he wo k by Ami i e al. [ 71 ], ein o cemen lea ning echniques a e used o ene gy managemen in he e ogeneous ne wo ks based on he QoS equi ed by each se ice in he ne wo k. Liu e al. [72] ocused on designing an IoT-based ene gy managemen sys em using an Edge Compu ing in as uc u e wi h deep ein o cemen lea ning. Fe dowsi and Saad [67] p oposes he use o deep ein o cemen lea ning using LSTM (Long and Sho -Te m Memo ies) [ 73 ] blocks o signal au hen ica ion and secu i y in massi e IoT sys ems. One o he mos ele an wo ks in his sense is he app oach by Mu e al. [ 74 ], which p oposes a SDN low en y managemen mechanism based on Deep Q-Ne wo ks and compa ed wi h classic Sus ainabili y 2020,12, 5706 9 o 23 Q-lea ning ein o cemen lea ning using Minine emula o [ 75 ]. In ou wo k, Deep Q-Ne wo ks is p oposed o manage he Ne wo k Func ion Vi ualiza ion in Edge-IoT scena ios. Unlike he wo k o Mu e al. [ 74 ], whose me hodology se es as a basis o he expe imen a ion s age o ou wo k, ou p oposal is o ien ed owa ds a speci ic SDN managemen solu ion in a e e ence a chi ec u e aimed o he design, implemen a ion and deploymen o Edge-IoT solu ions, in which he lows be ween nodes a e almos en i ely mice lows ( ha ca ies small amoun s o da a). 3. Managemen o SDN Flow En ies in he Global Edge Compu ing A chi ec u e by means o Deep Rein o cemen Lea ning In his sec ion we will desc ibe he con ibu ion o his wo k. Fi s , Sec ion 3.1 will depic he GECA a chi ec u e in i s p e ious s a e be o e in oducing SDN and NFV managemen . A e ha , Sec ion 3.2 will desc ibe he new componen s in oduced in he a chi ec u e in o de o op imize he managemen o da a lows. In his sense, a low managemen mechanism will be in oduced in he Cloud laye (Business Solu ion Laye ) based on Deep Rein o cemen Lea ning based on ewa ds indica ed by he Edge nodes based on hei sa is ac ion wi h he obse ed QoS. The mechanism will be based on Deep Q-Ne wo ks, and i s concep design will be explained in Sec ion 3.3. 3.1. The Global Edge Compu ing A chi ec u e 1.0 This sec ion depic s he Global Edge Compu ing A chi ec u e (GECA), an a chi ec u e designed and p esen ed by Si ón-Candanedo e al. [ 11 ] in a modula and ie ed manne , based on he concep o unc ional blocks. GECA is s uc u ed in h ee laye s: IoT,Edge and Business Solu ion laye s. The main objec i e o he a chi ec u e is o inco po a e low-cos bu high-capaci y compu ing esou ces in i s edge laye o allow p e-p ocessing and il e ing o he olume o da a sen in a adi ional IoT en i onmen by he se o connec ed senso s o de ices. The p incipal ea u es and componen s o he laye s o he a chi ec u e, shown in Figu e 4a e desc ibed in he ollowing o de : • IoT Laye : in GECA, his laye is made up o he se o elemen s ha de ine an en i onmen like IoT, ha is, objec s o connec ed hings ha cons an ly gene a e da a. Among hem a e senso s, ac ua o s, con olle s o IoT ga eways. The da a ansmission p ocess in his laye is done h ough he mos used communica ion s anda ds, such as: Wi-Fi, ZigBee, LoRa o SigFox. In addi ion o he IoT de ices, his laye includes he componen s ha p o ide secu i y o he a chi ec u e. The inco po a ion o a basic blockchain scheme in which he sma con ac s h ough o acles in e ac wi h he physical componen s o he laye , allows he da a o be ans e ed sa ely and ollowing he p ede ined e ms in each con ac . • Edge Laye : This laye p oposes he use o low-cos and high-capaci y boa ds such as he Raspbe y Pi, whose cha ac e is ics allow i o unc ion as an edge node o p ocess and il e he da a collec ed and sen by he de ices deployed in he IoT laye . The componen s o hese boa ds: I/O po s, 32 o 64 bi Linux ope a ing sys em, RAM memo y up o 4GB, USB 2.0 o 3.0 po s o SD ca ds, allow he ins alla ion o lib a ies such as Tenso Flow Li e o simila used o apply Machine Lea ning echniques o da a managemen [ 76 ]. Machine Lea ning echniques con ibu e o he deploymen o he Da a Analy ics in he Edge laye , making i easie o use s o ob ain aluable da a and esponses wi h lowe la ency, educing he cos s associa ed wi h Cloud Compu ing such as shipping, p ocessing, da a s o age as well as bandwid h consump ion. • Business Solu ion Laye : The se ices and applica ions associa ed wi h Business In elligence, which a e in a classic Cloud Compu ing a chi ec u e, a e included in his laye o GECA. A his le el he a chi ec u e acili a es he deploymen o public o p i a e se ices, also allowing he inclusion o componen s o : analysis (case based easoning, da a analysis and isualiza ion algo i hms), au hen ica ion ( o ensu e secu i y) knowledge base (using i ual agen o ganiza ions o decision suppo sys ems) and APIs (so ha se ices a e a ailable in any s anda d web b owse ). Sus ainabili y 2020,12, 5706 16 o 23 (Equa ion (7)). Fo he ou pu laye we conside an ou pu pe each possible ac ion (Equa ion (10)). Fo he in e media e (hidden) laye s, we ha e conside ed di e en ne wo k a chi ec u es (dis inc combina ion o laye s and neu ons pe laye ) o he es s o compa e hei esul s, bu using always ully-connec ed inne p oduc laye s. As in Re e ence [ 74 ], we use ReLU (Rec i ie Lineal Uni ) unc ions as ac i a ion unc ions, as p o ided by Equa ion (11). (x) = (0 i x<0 xi x≥0. (11) Figu e 6. Q-Ne wo k used in Global Edge Compu ing A chi ec u e (GECA) 2.0 SDN mechanism. Fo he expe imen a ion a Minine 2.2.2 i ual machine on an Ubun u 20.4 LTS unning di ec ly on O acle Cloud wi h a 1 CPU and 1GB o RAM has been used. In his base expe imen , Py hon 3.8.2 has been used o implemen he new mechanism o he Deep Q-Ne wo k ac ing in he SDN con olle o dynamically con igu e he ables in he emo e nodes, as his allows o use POX as Ne wo k Ope a ing Sys em o implemen he SDN [ 8 ]. Py hon has also been used o emula e he emo e nodes sending pe iodic esponse imes o he SDN Con olle . In he expe imen , we ha e emula ed 1 cen al SDN con olle , 4 ully-connec ed Fog Fo wa ding Ga eways, 8 Edge Ga eways (2 pe each Fog Fo wa ding Ga eway) and 16 IoT nodes (2 pe each Edge Ga eway), as depic ed in Figu e 7. We ha e only conside ed mice lows ( ha ca ies small amoun s o da a) o jus using Minine ’s buil -in ipe ool o gene a e andom a ic pa e ns, speci ically Poisson a ic a an a e age a e o 128 kbps, conside ing he wo k o Mu e al. [ 74 ] as design guide. We ha e no conside ed elephan lows in his expe imen as hey a e no ele an in he majo i y o IoT en i onmen s, specially in communica ions among IoT nodes. Two di e en es s we e pe o med wi h his IoT ne wo k, using wo di e en neu al ne wo k a chi ec u es in each case. In Tes 1, h ee in e media e (hidden) laye s we e used, wi h 8, 4 and 4 neu ons espec i ely. In Tes 2, h ee in e media e laye s we e also used, bu his ime wi h 8, 8 and 4 neu ons, espec i ely. In nei he o he wo scena ios we e he ne wo ks ained p e iously wi h some samples. Tha is, he ini ial alues o he weigh s we e chosen using andom alues. Fo each o he wo cases a maximum numbe o 1000 episodes was used. In bo h cases he same hype -pa ame e s we e used, using ypical alues in Deep Q-Lea ning—discoun ac o γ= 0.95, lea ning a e α= 0.1 and an ini ial explo a ion a e e =0= 1.00, dec easing o a minimum explo a ion a e e =1000 = 0.01 in he las episode. In o de o calcula e he loss alue, he S ochas ic G adien Descen (SGD) me hod is used. Sus ainabili y 2020,12, 5706 17 o 23 In bo h cases he goal o achie e was o inc ease by 30% he pe o mance o he IoT ne wo k, as measu ed by Equa ion (7), ha is, o educe by 30% he sum o o al esponse imes wi h espec o he s a ing si ua ion in he episode =0. Each o he es s was un 10 imes in o de o a e age he esul s ob ained. The summa y o he con igu a ion used and he esul s ob ained is shown in he Table 1. Figu e 7. Emula ed Fog-Edge-In e ne o Things (IoT) ne wo k used o he expe imen . As can be seen, he Tes 2 neu al ne wo k achie es be e esul s han he Tes 1 ne wo k o he con igu a ion used in he expe imen . Howe e , he compu a ional load is signi ican ly highe and he es s a e execu ed no ably slowe in he case o Tes 2. I is necessa y o pe o m mo e es s wi h di e en con igu a ions (di e en IoT ne wo k opologies, di e en a ic lows be ween nodes and di e en hype -pa ame e s) and o inco po a e quan i a i e measu es o he compu a ional load equi ed o es ima e he cos -bene i balance o using a dense neu al ne wo k. Sus ainabili y 2020,12, 5706 18 o 23 Table 1. Compa ison o con igu a ion and esul s ob ained wi h he di e en Q-Ne wo ks. Pa ame e Desc ip ion Tes 1 Tes 2 Neu al ne wo k a chi ec u e Neu ons pe laye 4:8:4:4:4 4:8:8:4:4 γDiscoun ac o 0.95 0.95 αLea ning a e 0.1 0.1 N Max episodes 1000 1000 e =0Ini ial explo a ion a e 1.00 1.00 e =1000 Minimum explo a ion a e 0.01 0.01 Goal Dec ease in o al esponse imes 30% 30% Episode in which goal is me Maximum 198 142 A e age 183 102 Minimum 163 92 5. Conclusions and Fu u e Wo k SDNs ad NFV make easie o deploy and dis ibu e applica ions by d ama ically educing in as uc u e o e head and cos s. SDNs enable cloud a chi ec u es h ough au oma ed and scalable applica ion dis ibu ion and mobili y. Mo eo e , VFN inc ease lexibili y and esou ce u iliza ion on SDNs by means o da a cen e i ualiza ion. Thanks o he applica ion o SDNs on IoT scena ios, i is possible o sepa a e he da a plane om he ne wo k con ol plane and in oduce a logically cen alized con ol plane, called a con olle , o abs ac con ol unc ions om ne wo king. P og ammable con ol mechanisms o so wa e-de ined ne wo ks make hem an al e na i e o educing he complexi y o Edge Compu ing (EC) a chi ec u es by enabling mo e e icien use o a ailable compu ing esou ces. By using SDNs he da a a ic o igina ing om Edge se e s can be dynamically ou ed eeing Edge de ices om he execu ion o complex ne wo k ac i i ies such as se ice de ec ion, o ches a ion, and QoS (pe o mance-delay) equi emen s. Howe e , hese ad an ages a e accompanied by new challenges in e ms o managing i ual esou ces. In his sense, i is necessa y o de elop in elligen mechanisms ha allow he au oma ed and dynamic managemen o he i ual communica ions es ablished in he SDNs by he di e en use nodes. The e a e di e en p oposals in his ega d based on machine lea ning, such as gene ic algo i hms o deep lea ning. Recen ly, new app oaches based on Deep Rein o cemen Lea ning ha e demons a ed o o e g ea po en ial in sol ing his challenge, especially due o he ad an ages o no needing p e ious aining da a. Fu u e wo k includes he implemen a ion and alida ion o he Global Edge Compu ing A chi ec u e (GECA) 2.0, wi h he possibili y o implemen ing So wa e-De ined Ne wo ks, as well as Ne wo k Func ion Vi ualiza ion. Thanks o he modula and scalable design o he a chi ec u e, he in oduc ion o he new sub-laye s and componen s is a s aigh o wa d p ocess. The Deep Q-Ne wo ks will be implemen ed, compa ed and es ed in he labo a o y. A e ha , an Indus y 4.0 pla o m will be deployed using he new e sion o he a chi ec u e in a eal scena io (mixed dai y a m) whe e GECA has been o me ly applied [6] o alida e he new in elligen mechanism. Also, in his i s e sion o GECA 2.0 has been conside ed a solu ion in which he SDN Con olle is cen alized in he Cloud. Howe e , as u u e wo k we will in es iga e solu ions in which he aining o he Q-Ne wo ks is ca ied ou in he Cloud, bu GECA 2.0 is p o ided wi h a mechanism o ans e he models o he Fog Fo wa ding Ga eways and Edge Ga eways. This will allow he econ igu a ion o he ou ing ables in he nodes o be ca ied ou in a dis ibu ed way and wi h less delay, being mo e app op ia e in scena ios whe e econ igu a ion speed and esponse imes a e c i ical. Mo eo e , u u e wo k also includes he esea ch and implemen a ion o new componen s wi hin he GECA 2.0 e e ence a chi ec u e, including he de elopmen o i s NFV ea u es, such as he Managemen and O ches a ion (MANO), as well as in elligen mechanisms based on Edge Compu ing and Machine Lea ning o implemen he p o isioning and con igu a ion o he VNFs and he Edge laye . Sus ainabili y 2020,12, 5706 19 o 23 Au ho Con ibu ions: Concep ualiza ion, R.S.A., I.S.-C. and R.C.-V.; In es iga ion, R.S.A., I.S.-C. and R.C.-V.; Me hodology, J.P. and J.M.C.; W i ing—o iginal d a , R.S.A., I.S.-C. and R.C.-V.; W i ing— e iew and edi ing, J.P. and J.M.C. All au ho s ha e ead and ag eed o he published e sion o he manusc ip . Funding: This wo k has been pa ially suppo ed by he Eu opean Regional De elopmen Fund (ERDF) h ough he In e eg Spain-Po ugal V-A P og am (POCTEP) unde g an 0677_DISRUPTIVE_2_E (In ensi ying he ac i i y o Digi al Inno a ion Hubs wi hin he PocTep egion o boos he de elopmen o dis up i e and las gene a ion ICTs h ough c oss-bo de coope a ion). Inés Si ón-Candanedo has been suppo ed by schola ship p og am: IFARHU-SENACYT (Go e nmen o Panama). Acknowledgmen s: Some icons in Figu es by Copy igh (c) 2018 Sand o Pe ei a (unde MIT License). Con lic s o In e es : The au ho s decla e no con lic o in e es . Re e ences 1. Radanlie , P.; De Rou e, D.C.; Nicolescu, R.; Hu h, M.; Mon al o, R.M.; Cannady, S.; Bu nap, P. Fu u e de elopmen s in cybe isk assessmen o he in e ne o hings. Compu . Ind. 2018 ,102, 14–22. [C ossRe ] 2. De la P ie a, F.; Rod íguez-González, S.; Chamoso, P.; Co chado, J.M.; Bajo, J. Su ey o agen -based cloud compu ing applica ions. Fu u e Gene . Compu . Sys . 2019,100, 223–236. [C ossRe ] 3. Mohammadi, M.; Al-Fuqaha, A.; So ou , S.; Guizani, M. Deep lea ning o IoT big da a and s eaming analy ics: A su ey. IEEE Commun. Su . Tu o . 2018,20, 2923–2960. [C ossRe ] 4. Ga cía, O.; Alonso, R.S.; P ie o, J.; Co chado, J.M. Ene gy E iciency in Public Buildings h ough Con ex -Awa e Social Compu ing. Senso s 2017,17, 826. [C ossRe ] 5. Shi, W.; Schah am, D. The p omise o Edge Compu ing. Compu e 2016,49, 78–81. [C ossRe ] 6. Alonso, R.S.; Si ón-Candanedo, I.; Ga cía, Ó.; P ie o, J.; Rod íguez-González, S. An in elligen Edge-IoT pla o m o moni o ing li es ock and c ops in a dai y a ming scena io. Ad Hoc Ne w. 2020 ,98, 102047. [C ossRe ] 7. G anelli, F.; Geb ema iam, A.A.; Usman, M.; Cugini, F.; S ama i, V.; Ali ska, M.; Cha zimisios, P. So wa e de ined and i ualized wi eless access in u u e wi eless ne wo ks: Scena ios and s anda ds. IEEE Commun. Mag. 2015,53, 26–34. [C ossRe ] 8. Puen e Fe nández, J.A.; Ga cía Villalba, L.J.; Kim, T.H. So wa e De ined Ne wo ks in Wi eless Senso A chi ec u es. En opy 2018,20, 225. [C ossRe ] 9. Alenezi, M.; Almus a a, K.; Mee ja, K.A. Cloud based SDN and NFV a chi ec u es o IoT in as uc u e. Egyp . In o m. J. 2019,20, 1–10. [C ossRe ] 10. Leike, J.; K uege , D.; E e i , T.; Ma ic, M.; Maini, V.; Legg, S. Scalable agen alignmen ia ewa d modeling: A esea ch di ec ion. a Xi 2018, a Xi :1811.07871. 11. Si ón-Candanedo, I.; Alonso, R.S.; Co chado, J.M.; Rod íguez-González, S.; Casado-Va a, R. A e iew o edge compu ing e e ence a chi ec u es and a new global edge p oposal. Fu u e Gene . Compu . Sys . 2019 , 99, 278–294. [C ossRe ] 12. Si ón-Candanedo, I.; Alonso, R.S.; Ga cía, Ó.; Muñoz, L.; Rod íguez-González, S. Edge compu ing, io and social compu ing in sma ene gy scena ios. Senso s 2019,19, 3353. [C ossRe ] [PubMed] 13. Alonso, R.S.; Si ón-Candanedo, I.; Rod íguez-González, S.; Ga cía, Ó.; P ie o, J. A Su ey on So wa e-De ined Ne wo ks and Edge Compu ing o e IoT. In Highligh s o P ac ical Applica ions o Su i able Agen s and Mul i-Agen Sys ems; The PAAMS Collec ion; Communica ions in Compu e and In o ma ion Science; De La P ie a, F., González-B iones, A., Pawleski, P., Cal a esi, D., Del Val, E., Lopes, F., Julian, V., Osaba, E., Sánchez-Ibo a, R., Eds.; Sp inge : Cham, Swi ze land, 2019; pp. 289–301. [C ossRe ] 14. Van Hassel , H.; Guez, A.; Sil e , D. Deep ein o cemen lea ning wi h double q-lea ning. In P oceedings o he Thi ie h AAAI Con e ence on A i icial In elligence, Phoenix, AZ, USA, 12–17 Feb ua y 2016. 15. Alonso, R.S.; Tapia, D.I.; Bajo, J.; Ga cía, Ó.; de Paz, J.F.; Co chado, J.M. Implemen ing a ha dwa e-embedded eac i e agen s pla o m based on a se ice-o ien ed a chi ec u e o e he e ogeneous wi eless senso ne wo ks. Ad Hoc Ne w. 2013,11, 151–166. [C ossRe ] 16. Ko, H.; Bae, K.; Ma ei os, G.; Kim, H.; Yoe, H.; Ramos, C. A S udy on he Key Managemen S a egy o Wi eless Senso Ne wo ks. ADCAIJ Ad . Dis ib. Compu . A i . In ell. J. 2015,3, 43–53. [C ossRe ] 17. Alonso, R.S.; Ga cía, Ó.; Saa ed a, A.; Tapia, D.I.; de Paz, J.F.; Co chado, J.M. He e ogeneous wi eless senso ne wo ks in a ele-moni o ing sys em o homeca e. In P oceedings o he In e na ional Wo k-Con e ence Sus ainabili y 2020,12, 5706 20 o 23 on A i icial Neu al Ne wo ks, Limassol, Cyp us, 14–17 Sep embe 2009; Sp inge : Be lin, Ge many, 2009; pp. 663–670. 18. Ga cía, Ó.; Alonso, R.S.; Tapia, D.I.; Co chado, J.M. Elec ical powe consump ion moni o ing in ho els using he n-co e pla o m. In P oceedings o he 2016 Clemson Uni e si y Powe Sys ems Con e ence (PSC), Clemson, CA, USA, 8–11 Ma ch 2016; IEEE: Pisca away, NJ, USA, 2016; pp. 1–6. 19. Ash on, K. Tha ‘in e ne o hings’ hing. RFID J. 2009,22, 97–114. 20. S inidhi, N.N.; Dilip Kuma , S.M.; Venugopal, K.R. Ne wo k op imiza ions in he In e ne o Things: A e iew. Eng. Sci. Technol. In . J. 2018,22, 1–21. [C ossRe ] 21. Ke ha eswa an, V.; Ram, C.S. An Indian Pe spec i e on he ad e se impac o In e ne o Things (IoT). ADCAIJ Ad . Dis ib. Compu . A i . In ell. J. 2017,6, 35–40. [C ossRe ] 22. Chamoso, P.; P ie a, F.D.L. Swa m-Based Sma Ci y Pla o m: A T a ic Applica ion. ADCAIJ Ad . Dis ib. Compu . A i . In ell. J. 2015,4, 89–98. [C ossRe ] 23. González-B iones, A.; De La P ie a, F.; Mohamad, M.; Oma u, S.; Co chado, J. Mul i-agen sys ems applica ions in ene gy op imiza ion p oblems: A s a e-o - he-a e iew. Ene gies 2018 ,11, 1928. [C ossRe ] 24. Chamoso, P.; González-B iones, A.; Rod íguez, S.; Co chado, J.M. Tendencies o Technologies and Pla o ms in Sma Ci ies: A S a e-o - he-A Re iew. Wi el. Commun. Mob. Compu . 2018. [C ossRe ] 25. Si ón-Candanedo, I.; He nández-Nie es, E.; Rod íguez-González, S.; San os-Ma ín, M.T.; González-B iones, A. Machine lea ning p edic i e model o indus y 4.0. In P oceedings o he In e na ional Con e ence on Knowledge Managemen in O ganiza ions, Žilina, Slo akia, 6–10 Augus 2018; Sp inge : Be lin, Ge many, 2018; pp. 501–510. 26. Liao, Y.; Lou es, E.d.F.R.; Deschamps, F. Indus ial In e ne o Things: A sys ema ic li e a u e e iew and insigh s. IEEE In e ne Things J. 2018,5, 4515–4525. [C ossRe ] 27. T appey, A.J.; T appey, C.V.; Go inda ajan, U.H.; Chuang, A.C.; Sun, J.J. A e iew o essen ial s anda ds and pa en landscapes o he In e ne o Things: A key enable o Indus y 4.0. Ad . Eng. In o m. 2017 , 33, 208–229. [C ossRe ] 28. Rod íguez, S.; De Paz, J.F.; Villa ubia, G.; Za o, C.; Bajo, J.; Co chado, J.M. Mul i-agen in o ma ion usion sys em o manage da a om a WSN in a esiden ial home. In . Fusion 2015,23, 43–57. [C ossRe ] 29. Si ón, I.; Rod íguez, S. Pa e n ex ac ion o he design o p edic i e models in indus y 4.0. In P oceedings o he In e na ional Con e ence on P ac ical Applica ions o Agen s and Mul i-Agen Sys ems, Po o, Po ugal, 21–23 June 2017; Sp inge : Be lin, Ge many, 2017; pp. 258–261. 30. Yu, W.; Liang, F.; He, X.; Ha che , W.G.; Lu, C.; Lin, J.; Yang, X. A Su ey on he Edge Compu ing o he In e ne o Things. IEEE Access 2017,6, 6900–6919. [C ossRe ] 31. Sanchez-Ibo a, R.; Sanchez-Gomez, J.; Ska me a, A. E ol ing IoT ne wo ks by he con luence o MEC and LP-WAN pa adigms. Fu u e Gene . Compu . Sys . 2018,88, 199–208. [C ossRe ] 32. Shi, W.; Cao, J.; Zhang, Q.; Li, Y.; Xu, L. Edge Compu ing: Vision and Challenges. IEEE In e ne Things J. 2016,3, 637–646. [C ossRe ] 33. Si ón-Candanedo, I.; Alonso, R.S.; Ga cía, Ó.; Gil, A.B.; Rod íguez-González, S. A Re iew on Edge Compu ing in Sma Ene gy by means o a Sys ema ic Mapping S udy. Elec onics 2020,9, 48. [C ossRe ] 34. FAR-EDGE-P. H2020 FAR-EDGE P ojec ; Fac o y Au oma ion Edge Compu ing Ope a ing Sys em Re e ence Implemen a ion (FAR-EDGE): Rome, I aly, 2017. 35. INTEL-SAP. IoT Join Re e ence A chi ec u e om In el and SAP. A ailable online: h ps: //www.in el.com/con en /dam/www/public/us/en/documen s/ e e ence-a chi ec u es/sap-io - e e ence-a chi ec u e.pd (accessed on 15 May 2020). 36. Edge Compu ing Conso ium; Alliance o Indus ial In e ne . Edge Compu ing Re e ence A chi ec u e 2.0; Technical Repo ; Edge Compu ing Conso ium and Alliance o Indus ial In e ne : Beijing, China, 2017. 37. Malik, A.; Roop, P.S.; Allen, N.; S ege , T. Emula ion o cybe -physical sys ems using IEC-61499. IEEE T ans. Ind. In o m. 2018,14, 380–389. [C ossRe ] 38. Tseng, M.; Cana an, T.E.; Cana an, L. In oduc ion o Edge Compu ing in IIoT; Technical Repo ; Indus ial In e ne Conso ium: Needham, MA, USA, 2018. 39. Ja a weh, Y.; Doula , A.; AlQudah, O.; Ahmed, E.; Al-Ayyoub, M.; Benkheli a, E. The u u e o mobile cloud compu ing: in eg a ing cloudle s and mobile edge compu ing. In P oceedings o he 2016 23 d In e na ional Con e ence on Telecommunica ions (ICT), Thessaloniki, G eece, 16–18 May 2016; IEEE: Pisca away, NJ, USA, 2016; pp. 1–5. Sus ainabili y 2020,12, 5706 21 o 23 40. Ca aguay, Á.L.V.; González, P.L.; Tandazo, R.T.; López, L.I.B. SDN/NFV A chi ec u e o IoT Ne wo ks. In P oceedings o he 14 h In e na ional Con e ence on Web In o ma ion Sys ems and Technologies (WEBIST 2018), Se ille, Spain, 18–20 Sep embe 2018; pp. 425–429. 41. Sabella, D.; Vaillan , A.; Kuu e, P.; Rauschenbach, U.; Gius , F. Mobile-edge compu ing a chi ec u e: The ole o MEC in he In e ne o Things. IEEE Consum. Elec on. Mag. 2016,5, 84–91. [C ossRe ] 42. Yang, M.; Li, Y.; Jin, D.; Zeng, L.; Wu, X.; Vasilakos, A.V. So wa e-De ined and Vi ualized Fu u e Mobile and Wi eless Ne wo ks: A Su ey. Mob. Ne w. Appl. 2015,20, 4–18. [C ossRe ] 43. Jammal, M.; Singh, T.; Shami, A.; Asal, R.; Li, Y. So wa e de ined ne wo king: S a e o he a and esea ch challenges. Compu . Ne w. 2014,72, 74–98. [C ossRe ] 44. Seze , S.; Sco -Haywa d, S.; Chouhan, P.K.; F ase , B.; Lake, D.; Finnegan, J.; Viljoen, N.; Mille , M.; Rao, N. A e we eady o SDN? Implemen a ion challenges o so wa e-de ined ne wo ks. IEEE Commun. Mag. 2013,51, 36–43. [C ossRe ] 45. Kim, H.; Feams e , N. Imp o ing ne wo k managemen wi h so wa e de ined ne wo king. IEEE Commun. Mag. 2013,51, 114–119. [C ossRe ] 46. Baek, J.Y.; Kaddoum, G.; Ga g, S.; Kau , K.; G a el, V. Managing og ne wo ks using ein o cemen lea ning based load balancing algo i hm. In P oceedings o he 2019 IEEE Wi eless Communica ions and Ne wo king Con e ence (WCNC), Ma akech, Mo occo, 15–18 Ap il 2019; IEEE: Pisca away, NJ, USA, 2019; pp. 1–7. 47. Sampaio, L.S.; Faus ini, P.H.; Sil a, A.S.; G an ille, L.Z.; Schae e -Filho, A. Using NFV and ein o cemen lea ning o anomalies de ec ion and mi iga ion in SDN. In P oceedings o he 2018 IEEE Symposium on Compu e s and Communica ions (ISCC), Na al, B azil, 25–28 June 2018; IEEE: Pisca away, NJ, USA, 2018; pp. 00432–00437. 48. Ruiz, L.; Du án, R.J.; De Miguel, I.; Khodashenas, P.S.; Ped eno-Man esa, J.J.; Me ayo, N.; Aguado, J.C.; Pa on-Ma ino, P.; Siddiqui, S.; Ma a, J.; e al. A gene ic algo i hm o VNF p o isioning in NFV-Enabled Cloud/MEC RAN a chi ec u es. Appl. Sci. 2018,8, 2614. [C ossRe ] 49. Pei, J.; Hong, P.; Li, D. Vi ual ne wo k unc ion selec ion and chaining based on deep lea ning in SDN and NFV-enabled ne wo ks. In P oceedings o he 2018 IEEE In e na ional Con e ence on Communica ions Wo kshops (ICC Wo kshops), Kansas Ci y, MO, USA, 20–24 May 2018; IEEE: Pisca away, NJ, USA, 2018; pp. 1–6. 50. Hu, F.; Hao, Q.; Bao, K. A Su ey on So wa e-De ined Ne wo k and OpenFlow: F om Concep o Implemen a ion. IEEE Commun. Su . Tu o . 2014,16, 2181–2206. [C ossRe ] 51. Jagadeesan, N.A.; K ishnamacha i, B. So wa e-De ined Ne wo king Pa adigms in Wi eless Ne wo ks: A Su ey. ACM Compu . Su . 2014,47, 27:1–27:11. [C ossRe ] 52. Mijumbi, R.; Se a , J.; Go icho, J.; Bou en, N.; Tu ck, F.D.; Bou aba, R. Ne wo k Func ion Vi ualiza ion: S a e-o - he-A and Resea ch Challenges. IEEE Commun. Su . Tu o . 2016,18, 236–262. [C ossRe ] 53. Ja a weh, Y.; Doula , A.; Da abseh, A.; Alsmi a , M.; Al-Ayyoub, M.; Benkheli a, E. SDMEC: So wa e de ined sys em o mobile edge compu ing. In P oceedings o he 2016 IEEE In e na ional Con e ence on Cloud Enginee ing Wo kshop (IC2EW), Be lin, Ge many, 4–8 Ap il 2016; IEEE: Pisca away, NJ, USA, 2016; pp. 88–93. 54. Salman, O.; Elhajj, I.; Kayssi, A.; Chehab, A. Edge compu ing enabling he In e ne o Things. In P oceedings o he 2015 IEEE 2nd Wo ld Fo um on In e ne o Things (WF-IoT), Milan, I aly, 14–16 Decembe 2015; IEEE: Pisca away, NJ, USA, 2015; pp. 603–608. 55. Bak i , A.C.; Ozgo de, A.; E soy, C. How Can Edge Compu ing Bene i om So wa e-De ined Ne wo king: A Su ey, Use Cases, and Fu u e Di ec ions. IEEE Commun. Su . Tu o . 2017,19, 2359–2391. [C ossRe ] 56. Pang, Z.; Sun, L.; Wang, Z.; Tian, E.; Yang, S. A Su ey o Cloudle Based Mobile Compu ing. In P oceedings o he 2015 In e na ional Con e ence on Cloud Compu ing and Big Da a (CCBD), Shanghai, China, 4–6 No embe 2015; pp. 268–275. [C ossRe ] 57. Mon a ed, S.; Bannazadeh, H.; Leon-Ga cia, A. So wa e de ined wi eless access o a wo- ie cloud sys em. In P oceedings o he 2015 IFIP/IEEE In e na ional Symposium on In eg a ed Ne wo k Managemen (IM), O awa, ON, USA, 11–15 May 2015; pp. 566–571. [C ossRe ] 58. He, Y.; Yu, F.R.; Zhao, N.; Leung, V.C.; Yin, H. So wa e-de ined ne wo ks wi h mobile edge compu ing and caching o sma ci ies: A big da a deep ein o cemen lea ning app oach. IEEE Commun. Mag. 2017 , 55, 31–37. [C ossRe ] Sus ainabili y 2020,12, 5706 22 o 23 59. Sallab, A.E.; Abdou, M.; Pe o , E.; Yogamani, S. Deep ein o cemen lea ning amewo k o au onomous d i ing. Elec on. Imaging 2017,2017, 70–76. [C ossRe ] 60. Sue ich, D.; Young, T. Machine Lea ning o Op imized Scheduling in Complex Semiconduc o Equipmen . In P oceedings o he 2019 30 h Annual SEMI Ad anced Semiconduc o Manu ac u ing Con e ence (ASMC), Sa a oga Sp ings, NY, USA, 6–9 May 2019; IEEE: Pisca away, NJ, USA, 2019, pp. 1–4. 61. Deng, Y.; Bao, F.; Kong, Y.; Ren, Z.; Dai, Q. Deep di ec ein o cemen lea ning o inancial signal ep esen a ion and ading. IEEE T ans. Neu al Ne w. Lea n. Sys . 2016,28, 653–664. [C ossRe ] [PubMed] 62. Ma, L.; Zhang, Z.; Ko, B.; S i a sa, M.; Leung, K.K. Resou ce managemen in dis ibu ed SDN using ein o cemen lea ning. In G ound/Ai Mul isenso In e ope abili y, In eg a ion, and Ne wo king o Pe sis en ISR IX; In e na ional Socie y o Op ics and Pho onics: Bellingham, WA, USA, 2018; Volume 10635, p. 106350M. 63. Ami i, R.; Almasi, M.A.; And ews, J.G.; Meh pouyan, H. Rein o cemen lea ning o sel o ganiza ion and powe con ol o wo- ie he e ogeneous ne wo ks. IEEE T ans. Wi el. Commun. 2019 ,18, 3933–3947. [C ossRe ] 64. Sil e , D.; Huang, A.; Maddison, C.J.; Guez, A.; Si e, L.; Van Den D iessche, G.; Sch i wiese , J.; An onoglou, I.; Pannee shel am, V.; Lanc o , M.; e al. Mas e ing he game o Go wi h deep neu al ne wo ks and ee sea ch. Na u e 2016,529, 484. [C ossRe ] [PubMed] 65. Sil e , D.; Sch i wiese , J.; Simonyan, K.; An onoglou, I.; Huang, A.; Guez, A.; Hube , T.; Bake , L.; Lai, M.; Bol on, A.; e al. Mas e ing he game o go wi hou human knowledge. Na u e 2017 ,550, 354–359. [C ossRe ] [PubMed] 66. Zhang, Q.; Lin, M.; Yang, L.T.; Chen, Z.; Khan, S.U.; Li, P. A double deep Q-lea ning model o ene gy-e icien edge scheduling. IEEE T ans. Se . Compu . 2018,12, 739–749. [C ossRe ] 67. Fe dowsi, A.; Saad, W. Deep lea ning o signal au hen ica ion and secu i y in massi e in e ne -o - hings sys ems. IEEE T ans. Commun. 2018,67, 1371–1387. [C ossRe ] 68. Wa kins, C.J.; Dayan, P. Q-lea ning. Mach. Lea n. 1992,8, 279–292. [C ossRe ] 69. A ulkuma an, K.; Deisen o h, M.P.; B undage, M.; Bha a h, A.A. Deep ein o cemen lea ning: A b ie su ey. IEEE Signal P ocess. Mag. 2017,34, 26–38. [C ossRe ] 70. Melo, F.S. Con e gence o Q-Lea ning: A Simple P oo ; Ins i u e o Sys ems and Robo ics, Ins i u o Supe io Técnico: Lisboa, Po ugal, 2001. 71. Ami i, R.; Meh pouyan, H.; F idman, L.; Mallik, R.K.; Nallana han, A.; Ma olak, D. A machine lea ning app oach o powe alloca ion in He Ne s conside ing QoS. In P oceedings o he 2018 IEEE In e na ional Con e ence on Communica ions (ICC), Kansas Ci y, MO, USA, 20–24 May 2018; IEEE: Pisca away, NJ, USA, 2018; pp. 1–7. 72. Liu, Y.; Yang, C.; Jiang, L.; Xie, S.; Zhang, Y. In elligen Edge Compu ing o IoT-Based Ene gy Managemen in Sma Ci ies. IEEE Ne w. 2019,33, 111–117. [C ossRe ] 73. Kim, J.; Kim, J.; Lee, S.; Pa k, J.; Hahn, M. Vowel based oice ac i i y de ec ion wi h LSTM ecu en neu al ne wo k. In P oceedings o he 8 h In e na ional Con e ence on Signal P ocessing Sys ems, U umqi, China, 20–22 July 2019; ACM: New Yo k, NY, USA, 2016; pp. 134–137. 74. Mu, T.Y.; Al-Fuqaha, A.; Shuaib, K.; Sallabi, F.M.; Qadi , J. SDN low en y managemen using ein o cemen lea ning. ACM T ans. Au on. Adap . Sys . (TAAS) 2018,13, 1–23. [C ossRe ] 75. De Oli ei a, R.L.S.; Schwei ze , C.M.; Shinoda, A.A.; P e e, L.R. Using minine o emula ion and p o o yping so wa e-de ined ne wo ks. In P oceedings o he 2014 IEEE Colombian Con e ence on Communica ions and Compu ing (COLCOM), Bogo a, Colombia, 4–6 June 2014; IEEE: Pisca away, NJ, USA, 2014; pp. 1–6. 76. Tang, J. In elligen Mobile P ojec s wi h Tenso Flow: Build 10+ A i icial In elligence Apps Using Tenso Flow Mobile and Li e o IOS, And oid, and Raspbe y Pi; Pack Publishing L d.: Bi mingham, UK, 2018. 77. AlE oud, A.; Alsmadi, I. Iden i ying cybe -a acks on so wa e de ined ne wo ks: An in e ence-based in usion de ec ion app oach. J. Ne w. Compu . Appl. 2017,80, 152–164. [C ossRe ] 78. Casado-Va a, R.; González-B iones, A.; P ie o, J.; Co chado, J.M. Sma Con ac o Moni o ing and Con ol o Logis ics Ac i i ies: Pha maceu ical U ili ies Case S udy. In Ad ances in In elligen Sys ems and Compu ing P oceedings o he In e na ional Join Con e ence SOCO’18-CISIS’18-ICEUTE’18, San Sebas ián, Spain, 6–8 June 2018; G aña, M., López-Guede, J.M., E xaniz, O., He e o, Á., Sáez, J.A., Quin ián, H., Co chado, E., Eds.; Sp inge : Be lin, Ge many, 2019; pp. 509–517. 79. Hassel , H.V. Double Q-lea ning. In Ad ances in Neu al In o ma ion P ocessing Sys ems 23, P oceedings o he 23 d In e na ional Con e ence on Neu al In o ma ion P ocessing Sys ems, Vancou e , Canada, 6-11 Decembe 2010; Sus ainabili y 2020,12, 5706 23 o 23 La e y, J.D., Williams, C.K.I., Shawe-Taylo , J., Zemel, R.S., Culo a, A., Eds.; Neu al In o ma ion P ocessing Sys ems: San Diego, CA, USA, 2010; pp. 2613–2621. 80. Shoeibi, N.; Shoeibi, N. Fu u e o Sma Pa king: Au oma ed Vale Pa king Using Deep Q-Lea ning. In P oceedings o he In e na ional Symposium on Dis ibu ed Compu ing and A i icial In elligence, A ila, Spain, 26–28 June 2019; Sp inge : Be lin, Ge many, 2019; pp. 177–182. 81. Mnih, V.; Ka ukcuoglu, K.; Sil e , D.; Rusu, A.A.; Veness, J.; Bellema e, M.G.; G a es, A.; Riedmille , M.; Fidjeland, A.K.; Os o ski, G.; e al. Human-le el con ol h ough deep ein o cemen lea ning. Na u e 2015 , 518, 529. [C ossRe ] [PubMed] c 2020 by he au ho s. Licensee MDPI, Basel, Swi ze land. This a icle is an open access a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons A ibu ion (CC BY) license (h p://c ea i ecommons.o g/licenses/by/4.0/).