Optimizing Flying Base Station Connectivity by RAN Slicing and Reinforcement Learning

Recei ed Ap il 22, 2022, accep ed May 10, 2022, da e o publica ion May 16, 2022, da e o cu en e sion May 24, 2022. Digi al Objec Iden i ie 10.1109/ACCESS.2022.3175487 Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning DICK CARRILLO MELGAREJO 1,2,3, (Membe , IEEE), JIRI POKORNY4,5, PAVEL SEDA4, ARUN NARAYANAN 1, (Membe , IEEE), PEDRO H. J. NARDELLI 1, (Senio Membe , IEEE), MEHDI RASTI1, (Senio Membe , IEEE), JIRI HOSEK 4, (Senio Membe , IEEE), MILOS SEDA6, DEMÓSTENES Z. RODRÍGUEZ 7, (Senio Membe , IEEE), YEVGENI KOUCHERYAVY 5, AND GUSTAVO FRAIDENRAICH 2 1Depa men o Elec ical Enginee ing, School o Ene gy Sys ems, Lappeen an a-Lah i Uni e si y o Technology (LUT), 53850 Lappeen an a, Finland 2School o Elec ical and Compu e Enginee ing, S a e Uni e si y o Campinas (UNICAMP), Campinas 13083, B azil 3Nokia Bell Labs, 02610 Espoo, Finland 4Depa men o Telecommunica ions, Facul y o Elec ical Enginee ing and Communica ion, B no Uni e si y o Technology, 601 90 B no, Czech Republic 5Uni o Elec ical Enginee ing, Tampe e Uni e si y, 337 20 Tampe e, Finland 6Ins i u e o Au oma ion and Compu e Science, Facul y o Mechanical Enginee ing, B no Uni e si y o Technology, 601 90 B no, Czech Republic 7Depa men o Compu e Science, Fede al Uni e si y o La as, La as, 37200 Minas Ge ais, B azil Co esponding au ho : Dick Ca illo Melga ejo ([email p o ec ed]) This wo k was suppo ed in pa by he Academy o Finland h ough he F amewo k o he Iden i ica ion o Ra e E en s ia MAchine lea ning and IoT Ne wo ks (FIREMAN) Conso ium unde G an CHIST-ERA-17-BDSI-003/n.326270 and Ene gyNe Resea ch Fellowship unde G an 321265/n.328869, and in pa by he Jane and Aa os E kko Founda ion h ough he Swa ming Technology o Reliable and Ene gy-awa e Ae ial Missions (STREAM) P ojec . ABSTRACT The applica ion o lying base s a ions (FBS) in wi eless communica ion is becoming a key enable o imp o e cellula wi eless connec i i y. Following his endency, his esea ch wo k aims o enhance he spec al e iciency o FBSs using he adio access ne wo k (RAN) slicing amewo k; his op imiza ion conside s ha FBSs’ loca ion was al eady de ined p e iously. This amewo k spli s he physical adio esou ces in o h ee RAN slices. These RAN slices schedule esou ces by op imizing indi idual slice spec al e iciency by using a deep ein o cemen lea ning app oach. The simula ion indica es ha he p oposed amewo k gene ally ou pe o ms he spec al e iciency o he ne wo k ha only conside s he heu is ic p ede ined FBS loca ion, al hough he gains a e no always signi ican in some speci ic cases. Finally, spec al e iciency is analyzed o each RAN slice esou ce and e alua ed in e ms o se ice-le el ag eemen (SLA) o indica e he pe o mance o he amewo k. INDEX TERMS Flying base s a ions, UAVs, loca ion op imiza ion, wi eless communica ion, deep- ein o cemen lea ning. I. INTRODUCTION Ex ensi e de elopmen s in he ield o unmanned ae ial ehicles (UAVs) ha e opened many oppo uni ies o new applica ions in bo h p i a e and public domains, such as su eillance, anspo a ion, en i onmen al moni o ing, indus ial moni o ing, ag icul u e se ices, and disas e elie [1], [2]. Recen ly, he inc easing numbe o use cases employ UAVs as wi eless ho spo s o elays o ex end ne wo k co e age in a eas whe e i is equi ed. Mo e- o e , nowadays, he e a e UAV applica ions used as a ool o communica ions a he applica ion le el, o example, The associa e edi o coo dina ing he e iew o his manusc ip and app o ing i o publica ion was Jiankang Zhang . in o ma ion sha ing in social media o sea ching o missing pe sons. Ano he example is he ecen loods in Ge many [3], which showed ha he in as uc u e is s ill qui e ulne a- ble. The e o e, i is wo h pu suing solu ions o o e come p oblems when he egula communica ion in as uc u e s ops wo king. Thus, he use o UAVs p o ides an essen ial esou ce o allowing he con inui y o communica ions and suppo ing human ope a o s o con inue o communica ing in sea ch and escue ope a ions, he eby gua an eeing e icien ope a ion [4]. In such scena ios, he op ion o apidly and e icien ly deploying a lee o d ones is c ucial in quickly es ablishing a communica ion ne wo k capable o sa ing li es, especially as i migh be di icul o use e es ial means comp ising empo a y ne wo king equipmen , such as 53746 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ VOLUME 10, 2022 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning a cell on wheels in na u al disas e s. This ea u e makes UAVs unique and c ucial o deploymen in such use cases [5]. In addi ion, deploying UAVs as lying base s a ions (FBS) has also ecen ly eme ged as a easible esponse o highly localized a ic demands in nex -gene a ion cellula ne wo ks [6], [7]. Using UAVs in such a way p o ides an oppo uni y o exploi hei agili y o mo ion o imp o e he ai - o-g ound link capaci y by op imal ai placemen [8], [9]. Typically, he abo e-men ioned use cases conside sig- ni ican ly la ge a eas whe e mul iple UAVs mus be used. Howe e , his leads o wo majo p oblems. Fi s , he UAVs mus be posi ioned o op imally co e as many use s as possible [10], [11]. Second, he in acell and in e cell in e - e ence mus be mi iga ed [12]. The i s p oblem can be e ec i ely app oached by using heu is ic algo i hms. These algo i hms can p o ide a solu ion wi h a low compu a ional ime and good esul s, as shown in [13], o example. In he case o in acell in e e ence, he sys em pe o mance can be imp o ed wi h a a ie y o mul iple access echniques, such as o hogonal equency-di ision mul iple access (OFDMA). When in e cell in e e ence is aken in o accoun , some popu- la schemes, such as equency euse, g aph heo y, and coop- e a i e mul i-poin (CoMP) [9], [14]–[18], can be employed. In ou p e ious wo k [13], we add essed he UAVs posi- ion op imali y using an heu is ic me hodology. Howe e , he adio channel in e e ence p oblem was no add essed in de ail. The p esen wo k ex ends [13] by employing a ep e- sen a i e wi eless channel model. In addi ion, we p oposed a adio access ne wo k (RAN) slicing amewo k ha enables he alloca ion o adio esou ces (slices) ca ying speci ic da a se ices. Ou p oposed amewo k aims o accommoda e a di e si y o se ices o e a single sha ed i h gene a ion (5G) in as uc u e and lays he ounda ion o ine-g ained se ice managemen in FBS ne wo ks. We ha e conside ed ha an agile RAN slicing amewo k is an app op ia e solu ion o achie e he pe o mance equi emen s in oduced by e icals on 5G communica ion ne wo ks. The RAN slicing ame- wo k comp ises se e al in e wo king unc ional componen s, aiming a a lexible ins an ia ion o adio se ices, ha can cope wi h he inc easing complexi y o suppo ing FBS se - ices. In ou wo k, we conside h ee slices: enhanced mobile b oadband (eMBB), ul a- eliable low-la ency communica- ion (URLLC), and massi e machine- ype communica ions (mMTC). The alloca ion o hese slices is achie ed by op imizing a cos unc ion ha is di ec ly ela ed o he spec al e iciency (SE) o he downlink da a ansmission, which is cons ained by he maximum powe ansmission and he numbe o RAN slices. A cellula ne wo k based on subchannels usually has a high p obabili y o in e cell in e e ences in he edge cell. To sol e his in ica e alloca ion p oblem, we in o- duce an in elligen componen in he amewo k—i.e., a deep ein o cemen lea ning (DRL) model— ha imp o es he sys em pe o mance and manages he adio esou ce allo- ca ion minimizing he in e e ence. By using ou p oposed in e e ence managemen me hodology o op imize he SE on each RAN slice, speci ic se ice-le el ag eemen (SLA)1 can be achie ed be ween he ne wo k se ice p o ide and he cus ome . To acili a e eade s comp ehension o his pape , he main con ibu ions o his pape a e summa ized as ollows: •enhancemen o he UAV loca ion dis ibu ion algo- i hm p oposed in [13], using a p ope ai - o-g ound channel model o enable aa app op ia e in e e ence analysis. •a no el RAN slicing amewo k is p oposed o enable he use o ad anced machine lea ning echniques, such as DRL. •we p opose a dis ibu ed DRL app oach o mi iga e he downlink in e e ence, in which each FBS ope a es as an independen lea ning agen . • h ee ep esen a i e scena ios o FBSs a e desc ibed and analyzed in de ail o compa e he SLA pe o mance be ween he DRL and he benchma k. •a mul iagen lea ning echnique is p oposed o op imize a noncon ex p oblem in he FBS sys em model. The es o he pape is o ganized as ollows. The model inding op imal placemen o UAVs in a gi en a ea, used as benchma k in his esea ch wo k, is p esen ed in Sec ion II. In Sec ion III, he RAN slicing amewo k is de ined, including he sys em model and he DRL me hodology o alloca e he adio esou ces. A de ailed desc ip ion o he op imiza ion sequence is gi en in Sec ion IV. The simula ion se up is p esen ed in Sec ion V. Nume ical esul s oge he wi h a ho ough compa a i e pe o mance analysis a e discussed in Sec ion VI. Ou concluding ema ks and u u e wo k a e p esen ed in Sec ion VII. II. UAV LOCATION OPTIMIZATION The e ec i e deploymen o UAVs ac oss a selec ed a ea is a di icul ask ha alls in o he ca ego y o N P−comple e class o p oblems [19]. To add ess his ask, we enhance he model p esen ed in [13] o loca ion co e ing o he UAV deploymen in on-demand connec i i y scena ios. The enhancemen ocuses in he elimina ion o in e e ence o all pai s o newly added cen es and o new and exis ing cen es. The p oposal in [13] did an ex ended explana ion o he UAV loca ion me hodology. The main idea o his deploymen was o selec a easible loca ions whe e UAVs can be loca ed by his heu is ic me hodology. Based on ha , he op imiza ion algo i hm selec ed he sui able UAV loca ion o compose a lis o UAVs and hei espec i e loca ions. A. DEPLOYMENT MODEL To acili a e he unde s anding o he model, we p o ide he e minology used in he es o his pape adap ed o he e ms used in he li e a u e in Table 1. To localize he sui - able posi ions o he FBS deploymen , loca ion op imiza ion 1SLAs es ablish cus ome expec a ions ega ding he se ice p o ide ’s pe o mance and o e all quali y. I is a con ac be ween he ne wo k se - ice (NS) p o ide and he cus ome . VOLUME 10, 2022 53747 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning TABLE 1. Mapping ma hema ical e minology o communica ion ne wo ks e minology. p oblems is aken as inspi a ion. Cu en ly, he e exis se e al acili y loca ion p oblems dealing wi h many eal-wo ld use cases. Simply, hey can be di ided in o loca ion se co e - ing p oblems (LSCPs) [20], and maximal co e ing loca ion p oblems (MCLPs) [21]. The LSCP a ge s he minimiza ion ques ion in which he numbe o acili ies ha sa is y he ne - wo k equi emen s and he need o be loca ed is minimized. On he o he hand, in he MCLP, a p ede ined numbe o esou ces ies o maximize i s co e age. The main di ision is based on he a ailable esou ces. Because hese models ha e been used o a wide ange o applica ions, hey a e no ied o elecommunica ion ne wo k deploymen only. Hence, o he bes o ou knowledge, he e is a gap in he li e a u e ha [13] aims o b idge o hese models and he use case o UAV deploymen . As a gap, we see he ollowing ac o s (o hei combina ion in one model): (i) Sepa a ing he capaci y o acili ies o loca ions co e ing bo h downlink and uplink. This may di e o each loca ion o acili y. (ii) Conside a ion o he exis ing se ices; o he use case o UAVs, i is essen ial o conside he exis ing in as- uc u e ha can se e a leas some demand om he loca ions o be co e ed ad hoc. (iii) Spli ing capaci y equi emen s om one loca ion o only one acili y a a gi en momen . (i ) Co e ing some loca ions wi h ze o o a highe numbe o acili ies. This is c ucial o he mus -ha e loca ions whe e i is no accep able o lose he connec i i y. ( ) The o e simpli ied wi eless in e e ence is based on he o e laps be ween cellula cells. We elimina e he in e e ence by Eq. (8) and (9). In [13], i is assumed ha co e age a ailabili y is gua - an eed. Capaci y conside a ions a e c i ical in he 5G-and- beyond deploymen s ha expec a signi ican inc ease in ne wo k a ic. This is due o he g ow h o se ices ha ha e conside ably highe ne wo k h oughpu equi emen s, such as he g ow h o high-de ini ion ideos, augmen ed eali y (AR) / i ual eali y (VR), machine- o-machine communica- ion, and o he e y in ensi e o demanding se ices in e ms o ne wo k equi emen s. In pa icula , we ha e o deal wi h a high densi y o use s ha a e simul aneously connec ed. Fo he exis ing acili ies E and hei co esponding decision a iables xi, whe e i∈E , we se his pa ame e o 1, which means ha all he exis ing acili ies a e aken in o accoun . The alloca ion o capaci y equi emen s be ween uploads and downloads ep esen s a spli o 100 Mbps o 80 Mbps o download (mo e ex ensi e) and 20 Mbps o upload. In addi- ion, we s ill need o sa is y he equi emen ha he demand jbo h o download and upload mus be assigned o he same acili y i. To de i e a ma hema ical model, le us se he ollowing no a ion: •I=a se o acili y si es (UAV o FBS) 1,2,...,m; •J=a se o demand a eas (cus ome s) 1,2,...,n; •dij = he sho es dis ance be ween acili y iand demand j; •Dmax =maximum dis ance which will be accep ed o ope a ion be ween he acili ies and demands; •lj=numbe o acili ies equi ed o se icing demand j; •xi∈ {0,1}, whe e xi=1 means ha acili y iis selec ed, while xi=0 means ha i is no selec ed. •Nj= {i|dij ≤Dmax} = he se o acili ies i ha can co e he demand loca ion j; •Cu i=upload capaci y o acili y i; •Cd i=download capaci y o acili y i; •au j=upload amoun o demand a j; •ad j=download amoun o demand a j; •yij ∈ {0,1} = non agmen ed demand om loca ion jis assigned (1) o is no assigned (0) o acili y i. Now, we se ou he ollowing model ex ac ed om [13] o minimize he numbe o equi ed FBSs and maximize he cellula co e age a ea. min X i∈I xi,(1) subjec o ∀j∈J:X i∈Nj xi≥lj(2) ∀j∈J:X i∈Nj yij =1 (3) ∀i∈Nj:Cu ixi≥X j∈J yijau j(4) ∀i∈Nj:Cd ixi≥X j∈J yijad j(5) (∀i∈I)(∀j∈J):yij ≤xi(6) ∀i∈E :xi=1 (7) (∀i∈I−E )(∀j∈I−E )(i6= j): dij ≥(xi+xj−1)dmin (8) (∀i∈I−E )(∀j∈E ):dij >dminxi(9) ∀i∈I:xi∈ {0,1}(10) (∀i∈I)(∀j∈J):yij ∈ {0,1}(11) Cons ain (3) gua an ees ha he demand jis assigned o only one acili y a a gi en momen . All selec ed acili ies mus ha e a su icien sum o hei capaci ies o uploads and downloads o co e all upload and download demands (in 53748 VOLUME 10, 2022 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning p ac ice, his is an ideal case ha ne wo k ope a o s a e ying o each wi h he a ailable esou ces), his is gua an eed by cons ain s (4) and (5). I a acili y is selec ed o be emo ed om he ne wo k in as uc u e, none o he demand should be assigned o i ; his cons ain is gi en by (6). In [13], he in e e ence is simpli ied o minimize he co e age o e laps de ined by he cells. Finally, conside he ollowing: i dij, i∈I,j∈Iis he dis ance be ween he acili ies iand j, hen we can se ha o all pai s o selec ed acili ies, he acili ies will ha e a dis ance g ea e o equal han a ce ain h eshold, which is gua an eed by cons ain (8). Fu he , as i is ypical in he s a e o he a dealing wi h loca ion co e age wi h model enhancemen s, au ho s in [13] p o ided an al e na i e maximiza ion model ha conside s a p ede ined numbe o new acili ies (no ye op imized) o be loca ed and co e ed as much a ea as possible: max X i∈NjX j∈J yij(au j+ad j),(12) subjec o he same cons ain s as in he minimiza ion model, bu wi h he addi ion o he ollowing cons ain o a p ede ined numbe o new acili ies. X i/∈E xi=p(13) No e ha his model a ge s he localiza ion o FBS nodes, which de ines he benchma k. This in o ma ion is used as inpu in he op imiza ion o FBS based on (RAN) slicing amewo k, which is de ailed in Sec ion III-E. The b ie explana ion o he algo i hm implemen ing he model is de ailed in Algo i hm 1. The Ge Inpu Da a pa ep- esen s he lis exis ing acili ies, expec ed/exis ing demands wi h coo dina es, and addi ional impo an me ada a. Fo he Co e ingModel compu a ion pa he heu is ics needs o implemen epai ope a o sa is ying all he model cons ain s (e.g., adding new UAVs o FBSs o he lis o solu ion o sa is y he capaci y equi emen s). B. CONSIDERATIONS OF THE COMPUTATIONAL COMPLEXITY MODEL The size o he sea ch space is de e mined by he numbe o all possible selec ions o acili ies. Fo m acili ies, acco ding o he binomial heo em, i is equal o m 1+m 2+ · · · + m m=(1+1)m−1=O(2m).(14) Fu he mo e, we need o ind he mos complex con- di ion in ex ended models o m<n(whe e nis he numbe o demand a eas) o ind he esul ing compu a- ional complexi y. In he minimiza ion model, hese a e (6) and (11) in he co esponding equa ions o he maximiza ion model, which equi e m·nope a ions. This is based on he ac ha he esul ing ime complexi y o hese models is O(2mmn) [13]. Algo i hm 1 Algo i hm Tha Op imizes he UAVs o FBSs Loca ion FMain pa 1: unc ion indLoca ions 2: Ge Inpu Da a 3: Gene a e heo e ical possible UAVs loca ions 4: Apply Co e ingModel() wi h hese da a FCo e ing model 5: unc ion Co e ingModel() 6: Gene a e possible solu ions 7: Apply epai ope a o p o iding easible solu ions 8: Apply selec ed heu is ics o ind op imal solu ion Sui able loca ions o he UAV deploymen III. RAN SLICING FRAMEWORK To complemen he benchma k desc ibed in Sec ion II, in his sec ion we desc ibe he p oposed RAN slicing amewo k. We employ he s anda dized de ini ions o RAN slicing in 5G, he sys em channel model, he adio op imiza ion p ob- lem o mula ion, and ou p oposed app oach using DRL. A. FRAMEWORK DESCRIPTION The di e se pe o mance equi emen s in oduced by 5G communica ion ne wo ks a e e ile g ound o he appli- ca ion o an agile RAN slicing amewo k. Ou p oposed amewo k aims o accommoda e a di e si y o se ices o e a single sha ed 5G in as uc u e and lays he ounda ion o ine-g ained se ice managemen in FBS ne wo ks. This RAN slicing amewo k p ima ily comp ises se e al in e - wo king unc ional componen s, aiming a a lexible ins an i- a ion o adio se ices. The p oposed a chi ec u e is de ised o cope wi h he ising complexi y o suppo ing FBS se - ices, achie ing no only mo e manageable RAN slices bu also con o ming he business p oposi ions sough by ne wo k ope a o s and se ice p o ide s akeholde s. This amewo k comp ises o hogonal physical esou ces ha spli he a ailable bandwid h o suppo a speci ic numbe o ne wo k slices. In his speci ic wo k, we conside h ee slices: eMBB, URLLC, and mMTC. In a cellula ne wo k based on subchannels, he RAN slicing amewo k is exposed o a high p obabili y o in e cell in e e ence, specially in he edge cell. To add ess his issue, we inco po a e an in elligen componen in he amewo k o manage he adio esou ce alloca ion using DRL. This in e e ence managemen aims o achie e speci ic SLA policies be ween he ne wo k se ice p o ide and he cus ome by op imizing he SE on each RAN slice. A g aphical desc ip ion o he concep o he p oposed amewo k is sha ed in Fig. 1. The sys em model and he DRL me hodology a e de ailed in he ollowing sec ion. B. SYSTEM MODEL Conside a se Io IFBS p o iding downlink wi eless se - ice o a g oup o use equipmen s (UEs) in a geog aphical a ea A. Each FBS i∈Ise es an a ea Ai, such ha VOLUME 10, 2022 53749 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning FIGURE 1. The RAN Slicing amewo k emphasizing he ole o he DRL module. In his pa icula case we conside ha each slice uses dedica ed physical esou ces in each FBS. Thus, he cogni ion elemen op imizes he in e e ence educ ion using he deep- ein o cemen lea ning app oach. ∪∀i∈IAi=Aand Ai∩Ak6= ∅ o any i6= k∈I. In o he wo ds, we conside ha when UAVs a e alloca ed by he op imiza ion algo i hm p esen ed in Sec ion II, i is possible ha some cells ha e a signi ican in e sec ion be ween hem. The pa h loss o he ai - o-g ound communica ion link om a ypical FBS loca ed a xi∈R3 o a ypical g ound UE ha is loca ed a y∈R3is gi en as ollows [22]: h( ) →n[dB](xi,y)=20 log10 4π ckxi−yk c+ξ(xi,y), (15) whe e cis he ca ie equency o FBS downlink com- munica ions, kxi−ykis he FBS–UE dis ance, cis he speed o ligh , and ξ(xi,y) is he addi ional pa h loss o he ai - o-g ound channel, compa ed wi h he ee space p op- aga ion. The alue o ξ(xi,y) can be modeled as a Gaus- sian dis ibu ion wi h di e en pa ame e s (µLOS, σ2 LOS) and (µNLOS, σ2 NLOS) o line-o -sigh (LOS) and non-line-o -sigh (NLOS) links, espec i ely. Then, he downlink spec al e i- ciency achie ed by he RAN slice m, he use n, a he ime slo om he FBS loca ed a xi o a UE loca ed a y∈Aiis C( ) n,m(xi,y)=log21+γ( ) n,m(xi,y),(16) whe e γ( ) n,m(xi,y) is he signal- o-in e e ence-plus-noise (SINR) a he use n, on he RAN slice m, a he ime slo , which is de ined by (17) γ( ) n,m(xi,y)=β( ) l,mg( ) l→n,m(xl,y)p( ) ll=n P 6=l β( ) ,mg( ) →n,m(x ,y)p( ) +σ2 n ,(17) whe e β( ) ,mis he bina y a iable ha indica es he RAN slicing selec ion m ansmi ed om he UAV a ime , g( ) →n,m(x ,y) indica es he downlink channel gain om he FBS o he use non he RAN slice min he ime slo when he UE is loca ed in he posi ion yand he FBS in he posi ion x ∈R3,p( ) is he ansmi powe o he UAV in he ime slo , and σ2is he addi i e whi e Gaussian noise powe spec al densi y a he use ecei e n. g( ) →n,m(x ,yn)=h( ) →n(x ,yn)α( ) n→l,m 2 =1,2,· · · , (18) whe e h →n(x ,yn) is he pa h loss in a linea scale, which is calcula ed in (15), and α( ) n→l,mis he small-scale Rayleigh ading. The p obabili y o ha ing an LOS link be ween he FBS j loca ed a xjand he UE loca ed a yis gi en by [22]: PLOS(xj,yi)=1 1+aexp −b180 πσ(xj,y)−a,(19) 53750 VOLUME 10, 2022 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning whe e aand ba e cons an alues ha depend on he com- munica ion en i onmen , σ(xj,y)=sin−1Hj kxj−ykis he ele a ion angle, and Hjis he al i ude o he FBS j. Then, he a e age downlink SE be ween an FBS iand he UE a nloca ed in yn∈Aiwill be: ¯ C( ) n,m(xi,yn)=PLOS(xi,yn)C( )LOS n,m(xi,yn) +(1 −PLOS(xi,yn))C( )NLOS n,m(xi,yn).(20) C. RAN SLICING IN 5G A simpli ied 5G logical a chi ec u e is composed o a co e cloud, an edge cloud, and an RAN. The co e cloud p o- ides gene ic con ol plane signaliza ion, slice managemen , mobili y managemen , and au hen ica ion. The edge cloud pe o ms some use plane unc ions as a packe /se ice ga e- way (P/S-GW) o imp o e la ency communica ion on c i - ical applica ions. I also enables da a o wa ding, con ol plane unc ions, and mobile edge compu ing pla o ms, such as con en s o age se e s. In he adio access plane, he 3 d Gene a ion Pa ne ship P ojec (3GPP) de ines he nex - gene a ion RAN (NG-RAN), which is comp ised o nex - gene a ion NodeBs (gNBs) connec ed o he co e ne wo k. This a chi ec u e is used o suppo he ne wo k slicing app oach p oposed in 5G. In his aspec , he e a e wo ypes o subne s in he 5G slicing a chi ec u e: co e ne wo k slice subne s and RAN slice subne s. In he co e ne wo k slice subne s, he ne wo k slicing ope - a ion used in he co e ne wo k is con olled by he ne wo k slicing managemen . I is composed o he i ualized ne - wo k unc ion managemen (VNFM), he so wa e-de ined ne wo k (SDN) con olle , he managemen and o ches a ion uni , and he i ualized in as uc u e managemen (VIM). The VNFM maps he physical ne wo k unc ions o i ual machines (VMs); he SDN con olle manages and ope - a es he en i e i ual ne wo k; he VIM alloca es i ualized esou ces o VMs; and he managemen and o ches a ion uni c ea es, ac i a es, and dele es ne wo k slices based on he se ice equi emen s. In he RAN slice subne s, he gNB is a c ucial enable o ne wo k slices. I p o ides RAN slice subne s ha a e composed o a cen alized uni (CU), mul iple dis ibu ed uni s (DUs), and mul iple adio uni s (RUs). The gNB unc- ionali ies a e dis ibu ed in a lexible manne be ween he CU, DUs, and RUs. To manage hei li e cycles, he s anda d speci ies he RAN ne wo k slice subne empla e (NSST) and wo managemen en i ies, such as he RAN ne wo k slice subne managemen unc ion (NSSMF) and he ne wo k unc ion managemen unc ions (NFMFs) [23]. The co e ne wo k slice subne s ha e been s udied and de eloped in he cu en 5G wi h ou s anding esul s. How- e e , he RAN slicing is s ill an open opic, and i is no ye s anda dized. The RAN slicing aims o imp o e he e icien usage o a ailable physical adio esou ces and simul aneously gua an ees he SLA policies imposed in each slice. D. REINFORCEMENT LEARNING AIDED UAVs Machine lea ning is an app oach ha has become inc eas- ingly popula o sequen ial decision-making on wi eless communica ion ne wo ks wi h applica ions in many di e se a eas, such as sma g ids, sel -d i ing ca s, and obo ics. The e a e h ee machine lea ning ca ego ies, depending on he na u e o he in o ma ion o eedback a ailable o he lea ning sys em: (i) supe ised lea ning; (ii) unsupe ised lea ning; and (iii) ein o cemen lea ning (RL). In his pape , we use RL as he p ima y app oach o op imizing he cos unc ion based on spec al e iciency. RL is a echnique ha is conce ned wi h how agen s should de e mine he sequences o ac ions in an en i onmen ha will maximize cumula i e ewa ds [24], [25]. I is a ial-and-e o p ocess whe e an agen in e ac s wi h an unknown en i onmen in a sequence o disc e e ime s eps o achie e a ask. A ime , he agen i s obse es he cu en s a e o he en i onmen , which is a uple o ele an en i onmen ea u es and is deno ed as S( )∈S, whe e Sis a se o possible s a es. I hen akes an ac ion a( )∈A om an allowed se o ac ions A acco ding o a policy ha can be ei he s ochas ic, i.e., π wi h a( )∼π(.|S( )) o de e minis ic, i.e., µwi h a( )= µ(S( )). Because he in e ac ions a e o en modeled as a Ma ko decision p ocess, he en i onmen mo es o a nex s a e S( +1) ollowing an unknown ansi ion ma ix ha maps s a e–ac ion pai s on o a dis ibu ion o successi e s a es, and he agen ecei es a ewa d S( +1). O e all, he abo e p ocess is desc ibed as an expe ience a +1 deno ed as e( +1) = (S( ),a( ), ( +1),s( +1)). The goal is o lea n a policy ha maximizes he cumula i e discoun ed ewa d a ime , de ined as ollows: R( )= ∞ X τ=0 γτ ( +τ+1),(21) whe e γ∈(0;1] is he discoun ac o . RL has been g owing in popula i y because i does no equi e an ex ensi e ne wo k model. Ins ead, i s lea ning p ocess is based on he in e ac ions wi h he en i onmen ha p oduces i s op imal s a egies. Owing o he possibili y o combining RL wi h deep lea ning [26], DRL is a highly sui able me hod o sol ing p oblems wi h a high numbe o s a es and low p io knowledge, which is he case o he esou ce alloca ion scena io in RAN slicing. DRL has ecen ly been used in p oblems ela ed o UAVs [27]–[31]. Mo eo e , he DRL app oach has been exploi ed and applied o he p oblem o UAV posi ion and esou ce alloca ion. In [30], he au ho s p oposed a DRL algo- i hm based on echo s a e ne wo k (ESN) cells o op imizing he UAV pa h, cell associa ion o minimize he in e cell in e - e ence le el, ansmission delay, and ansmi powe le el. In [32], he ESN algo i hm was used based on a mul iagen Q-lea ning app oach, which was employed o p edic he u u e posi ions o UEs and de e mine he posi ions o UAVs. Howe e , his wo k did no conside UAV coope a ion and capaci y limi a ions o on haul links be ween UAVs and VOLUME 10, 2022 53751 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning egula base s a ions. Fu he , o he op imiza ion o FBS placemen , he s udies in [27], [28], [33] used an RL algo- i hm. In his wo k, we p esen an RAN slicing amewo k based on a DRL me hodology ha complemen s he loca ion op imiza ion model ob ained in [13] by adding a adio chan- nel model and op imizing he RAN slice esou ces be ween mul iple FBSs co e ing an a bi a y a ea. E. RADIO OPTIMIZATION PROBLEM FORMULATION To apply he DRL me hodology explained in he p e ious subsec ion, we de ine he adio op imiza ion p oblem ha his pape aims o op imize. Thus, de ails o he cos unc ion and i s cons ain s a e de ined in he ollowing. Deno ing RAN slices and powe ec o s in he ime slo as β( )=hβ( ) 1,1, β( ) 1,2,· · · , β( ) N,MiTand p( )=hp( ) 1,· · · ,p( ) NiT espec i ely, we de ine he sum- a e maximiza ion p oblem as max p( ),α( ) N X n=1 ¯ C( ) n(xi,yn) s. . 0 ≤p( ) n≤Pmax,∀n∈N, β( ) n,m∈ {0,1},∀n∈N,∀m∈M, X m∈M β( ) n,m=1,∀n∈N,(22) whe e ¯ C( ) n=PM m=1¯ C( ) n,m(xi,yn). The noncon ex p oblem in (22) equi es a highly complex app oach ha could also inc ease he compu a ional com- plexi y. To handle his noncon ex p oblem, we conside a mul iagen lea ning scheme, whe e each ansmi e , moun ed in each FBS, ope a es as an independen lea ning agen . Each agen success ully execu es wo policies o de e mine i s asso- cia ed RAN slice and ansmission powe le el. The p oposed mul iagen app oach is easily scalable o mo e ex ensi e ne - wo ks and can ope a e wi h local in o ma ion a e aining. The componen s o he DRL me hodology conside ed based on he sys em model desc ibed be o e is composed by: •Agen s: in he mul iple lea ning app oach, he FBSs ep esen he agen s. •Policies: wo well de ined policies a e conside ed. π1 o choose an speci ic RAN slice, and he π2 o selec a p ope powe le el o each use . •Ac ions: we conside wo well de ined g oup o ac ions. The disc e e ac ions ela ed o he selec ion o RAN slices, and he con inuous ac ion o choose he powe ansmission o each indi idual use . •S a es: I is composed by a uple o in o ma ion ela ed o he RAN slice alloca ion, he SE, in e e ence in each indi idual use , gain and in e e ence in each use . •Rewa ds: a p opo ional alue o he SE in each ecei e (UE) is used as ewa d. I conside s he ollowing c i e- ia: he SE is e alua ed in e e y use wi h he condi ion o one neighbou hood base s a ion (BS) o agen is no ansmi ing. Thus, i he SE alue is signi ican , hen he BS being e alua ed is penalized. In con as , i he SE emains, hen he BS is ewa ded. A he beginning o each ime slo , each agen successi ely execu es wo policies o de e mine i s associa ed ansmis- sion powe le el and RAN slice selec ion. Fo his pu pose, he DRL conside s wo op imiza ion app oaches. The i s conside s a Deep Q-ne wo k o op imize a s ochas ic policy ha aims o imp o e he RAN slice selec ion. A second Deep Q-ne wo k op imizes a de e minis ic policy o selec a sui able powe ansmission alue. The agen o he second Deep Q-ne wo k equi es he RAN slice decision o he i s app oach o de e mine i s s a e inpu be o e se ing he ans- mi powe o he agen . A b ie explana ion o his app oach is done in Algo i hm 2. Algo i hm 2 Algo i hm o he DRL App oach FMain Loop 1: while S op C i e ia no me do 2: RAN slice Selec ion() 3: Powe Con ol() FRAN slice selec ion 4: unc ion RAN slice Selec ion() FAc ion Selec ion 5: ac ion ←a( ) n∈ARAN-Slice = {1,· · · ,M} = M 6: S a e se design ←s( ) n,m 7: T aining by a deep-Q Ne wo k 8: Rewa d unc ion design 9: Upda e Policies: π1 FPowe Con ol 10: unc ion Powe Con ol() FAc ion Selec ion 11: ac ion ←a( ) n,a( ) n ∈Apowe =[0,1] 12: S a e se design ←s( ) n,a( ) n 13: T aining by a deep-Q Ne wo k 14: Rewa d unc ion design 15: Upda e Policies: π2 IV. DESCRIPTION OF THE PROPOSED SOLUTION This pape aims o complemen and enhance he ou pu ob ained in [13], which is no op imal i e alua ed in a eal scena io. Thus, he esou ce alloca ion ha we p opose is condi ioned o he p eloca ion o each FBS ob ained using he me hodology p esen ed in Sec ion II. As he bench- ma k does no conside any channel model o e alua e he in acell and in e cell in e e ence, i can ge subop- imal esul s in p ac ical wi eless scena ios. In pa icula , we aim o add ess he ollowing esea ch ques ions ha a ise du ing he FBS ne wo k deploymen in a eal scena io: •Wha can be he po en ial imp o emen s based on he FBSs p eloca ion de ined in [13]?, 53752 VOLUME 10, 2022 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning FIGURE 2. Op imiza ion low diag am showing majo s eps and s ages. He e, we emphasis he op imiza ion done in he benchma k (in yellow) and highligh he DRL app oach (blue). •Wha is he pe o mance o he simula ion se up when di e en se ices a e suppo ed by he FBS ne wo k?, •Is i possible o de elop a p ac ical op imiza ion me hod ha is capable o imp o ing he pe o mance o FBSs?. To add ess hese ques ions, we use a s anda d simula ion model de ined by 3GPP and a RAN slicing amewo k p o- posed in Sec ion III, which is p oposed o analyze he ne wo k pe o mance using a DRL me hodology o op imize he adio esou ces. The whole op imiza ion p ocess is di ided in o h ee phases— da a ma ix gene a ion, FBS loca ion minimiza ion, and RAN slicing op imiza ion—as shown in Fig. 2. He e, S ages 1 and 2 ha e al eady been applied and e i ied in he o iginal publica ion [13]. De ails o each s age a e desc ibed in he ollowing pa ag aphs. A. STAGE 1—DATA MATRIX GENERATION In his s age, use s a e gene a ed andomly and uni o mly in one speci ic and ixed a ea. The FBSs a e gene a ed acco d- ing o he desi ed adius. The la ge he adius is, he lowe he numbe o FBSs is equi ed. A la ge adius causes a highe pe cen age o o e laps. No e ha S ages 1 and 2 do no ake in e e ence in o conside a ion. Thus, in e e ence has o be a oided by limi ing he adio o e lapping. The use s a e gi en andom da a a e equi emen s acco ding o he a ic mix. The ou pu o his s age is a ma ix whose ows ep esen all FBSs and columns ep esen all use s. The ma ix is illed wi h ze os i he use is inside he speci ic FBS cell adius, and o he wise, i is illed wi h ones. This ma ix is comple ed and used as he inpu o S age 2. B. STAGE 2—FBS LOCALIZATION This s age is composed o a so wa e se ice ha uses he ou pu o S age 1. This s age aims o op imize he ows o he gene a ed ma ix so ha he cus ome equi emen s a e me . In his s age, we apply he op imiza ion model om Sec ion II-A. Because he complexi y o selec ing he op imal ows (FBSs) om he ma ix is O(2m), heu is ic algo i hms a e applied. As a consequence, we use he di e en ial e o- lu ion and cuckoo sea ch algo i hms wi h epai ope a o s p esen ed in [13]. The ou pu o he compu a ion is a se o FBSs (wi h hei loca ions om he p e ious s age) ha should be used in he ne wo k deploymen . Fu he , he loca- ion o each FBS is used as an inpu o S age 3, conside ing a ep esen a i e in e e ence channel model. C. STAGE 3—OPTIMIZATION OF RAN SLICES We add an in e e ence model based on he 3GPP s anda d o complemen he p e ious s ages o analyze and mi iga e downlink in e e ence. Then, we calcula e he SINR, de ined in (17), o each use conside ing he signal s eng h coming om he se ing UAV and om he FBSs ha a e in e e ing wi h ha speci ic use n. Using (22), ou simula ion model calcula es an op imal spec al e iciency o each use n ollowing he p oposed DRL app oach on he RAN slicing amewo k de ined in Sec ion III. V. SIMULATION SETUP Two scena ios we e designed o emphasize he gain o he op imiza ion models ha we e p e iously explained. In bo h cases, 1000 use s we e deployed ollowing a uni o m dis i- bu ion in a speci ic a ea. Subsequen ly, in acco dance wi h he cons ain s in each scena io, he FBSs loca ion op imize de e mined he coo dina es o each FBS. The cons ain s a e mos ly ela ed o he cell adius, ha depends on he adio equency ope a ion, and h oughpu demand o each use o g oup o use s. Using he p ede e mined FBSs loca ion, we conside a speci ic wi eless sys em model. This sys em model consid- e s an in ini y backhaul capaci y on each FBS. Howe e , he o e ed bandwid h is limi ed in each FBS. The wi eless sys em simula ion conside s ha only a speci ic numbe o use s a e capable o ge ing he access s a um (AS)2in e - ace. Thus, mos o he use s a e in he RRC_IDLE and RRC_INACTIVE s a es, as de ined in he 5G new adio (NR) s anda d. The emaining use s ha passed he andom access p ocedu e a he medium access con ol (MAC) laye a e in he RRC_CONNECTED s a e. To acili a e he analysis, we conside ha each FBS is capable o suppo ing a ixed numbe o use s Nin he 2AS is a unc ional in e ace ha is esponsible o anspo ing da a o e he wi eless connec ion and managing adio esou ces. VOLUME 10, 2022 53753 D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning FIGURE 3. Gene ic simula ion sys em model, conside ing he key cons ain s o be sol ed by he deep ein o cemen lea ning app oach. RRC_CONNECTED s a e. Based on his assump ion, he cellula ne wo k is simula ed wi h LFBSs, and each FBS sup- po s KRAN slices. As we conside only downlink ansmis- sion, he in e e ence ha each use su e s om nonse ing FBSs is calcula ed ollowing he assump ions de ined in he 3GPP TR 36.931 Ve . 16 speci ica ions. The heigh al i ude o all FBSs is 20 m. The DRL and he wi eless sys em model a e deployed in Py hon. The machine lea ning implemen a ion is done using Tenso Flow lib a ies o implemen he deep Q-ne wo k se up. The deep neu al ne wo k used in he simula ion has 3 hidden laye s wi h 200, 100, 50 ully connec ed neu ons. The ba ch size is 128. The epsilon-G eedy Algo i hm, used in his wo k, conside s a maximum equal o 0.1 and an decay o 0.9995. The implemen a ion and u he hype -pa ame e s a e a ailable in he ollowing Gi hub URL h p://www.gi hub. com/TBD. Th ee di e en scena ios we e de ined, conside ing aspec s such as a ixed numbe o FBSs (Scena io 1), and co e age maximiza ion wi h a ixed numbe o use s in he desi ed co e age a ea and in each cell (Scena ios 2 and 3, espec i ely). A. SCENARIO 1—FIXED NUMBER OF AVAILABLE FBSs This scena io conside s L=20 FBSs co e ing a speci ic a ea and suppo ing N=100 use s. To acili a e he analysis, i is conside ed ha each FBS has N/L=5 a ached use s. This scena io aims o iden i y an op imal cell adius suppo ing N use s in he RRC_CONNECTED s a e. A g aphical desc ip- ion o his scena io is shown in Fig. 4. B. SCENARIO 2—MAXIMIZING THE NETWORK COVERAGE I In his case, he numbe o FBSs (L) is de ined by he op i- miza ion ou pu s o S ages 1 and 2, which we e desc ibed in Sec ion IV. Using Fig. 3as a e e ence, he alues o X and Y a e 400 m. Thus, he numbe o FBSs is a unc ion o cell adius; e.g., he la ge he cell adius is, he smalle he numbe o FBSs needs o be deployed. In his scena io, he numbe o use s Nin he RRC_CONNECTED s a e is TABLE 2. Compa ison o pa ame e s used in Scena ios 2 and 3. always he same in he a ea de ined by X and Y. This scena io is illus a ed in Fig. 5. C. SCENARIO 3—MAXIMIZING THE NETWORK COVERAGE II In his case, he numbe o UAVs (L) is de ined by he op imiza ion ou pu s o S ages 1 and 2, which we e desc ibed in Sec ion IV. Based on Fig. 3, he alue o X and Y is ixed a 400 m. In his scena io, he numbe o Nuse s in he RRC_CONNECTED s a e a ies when he numbe o FBSs changes, as i is s a ed in Table 2. Each cell adius has a ixed numbe o Nuse s in he scena io. Fig. 6depic s his scena io. VI. ANALYSIS AND RESULTS A. ANALYSIS OF SCENARIO 1 The spec al e iciency ob ained in Scena io 1 is p esen ed in Fig. 7. He e, di e en cell adii be ween 50 m and 300 m a e e alua ed. In he same igu e, he i s 500 i e a ions o he i s episode3(be ween 1 and 10000 i e a ions) a e compa ed wi h he i s 500 i e a ions o he second episode (be ween 10001 and 20000 i e a ions). The policy, de ined in Subsec ion III-D, is almos null in he i s i e a ions; he spec al e iciency is e y low, simila o a andom alloca ion o RAN slicing esou ces. A he beginning o episode 2, he spec al e iciency imp o es quickly despi e ha a new use deploymen is conside ed. To acili a e he in e p e a ion o he simula ion esul s, he a e age spec al e iciency in each scena io is calcula ed. Fo ins ance, Fig. 8p esen s he esul s o Scena io 1 be o e and a e applying he RAN slicing esou ce alloca ion. Be o e applying he esou ce alloca ion, he maximum spec al e i- ciency is almos 1.5 bps/Hz o he cell adius o 50 m. The spec al e iciency dec eases o he o he cell adii when he cell adius is inc eased. This ou pu ep esen s he ne wo k pe o mance when only S ages 1 and 2, desc ibed in Sec ion IV, a e conside ed. Howe e , he pe o mance is imp o ed in all cell adii a e applying he DRL app oach. In all cases, he spec al e iciency is imp o ed. Fo ins ance, he cell adius o 75 m achie es a spec al e iciency o mo e han 3.5 bps/Hz, ep esen ing a gain o 2 bps/Hz compa ed wi h he se up wi hou op imiza ion. 3Wi h episode we e e o a comple e sequence o in e ac ions, om s a o inish. In ou simula ions, one episode is comple ed a e 10 000 i e a ions. 53754 VOLUME 10, 2022