Recei ed Ap il 22, 2022, accep ed May 10, 2022, da e o publica ion May 16, 2022, da e o cu en e sion May 24, 2022.
Digi al Objec Iden i ie 10.1109/ACCESS.2022.3175487
Op imizing Flying Base S a ion Connec i i y by
RAN Slicing and Rein o cemen Lea ning
DICK CARRILLO MELGAREJO 1,2,3, (Membe , IEEE), JIRI POKORNY4,5, PAVEL SEDA4,
ARUN NARAYANAN 1, (Membe , IEEE), PEDRO H. J. NARDELLI 1, (Senio Membe , IEEE),
MEHDI RASTI1, (Senio Membe , IEEE), JIRI HOSEK 4, (Senio Membe , IEEE),
MILOS SEDA6, DEMÓSTENES Z. RODRÍGUEZ 7, (Senio Membe , IEEE),
YEVGENI KOUCHERYAVY 5, AND GUSTAVO FRAIDENRAICH 2
1Depa men o Elec ical Enginee ing, School o Ene gy Sys ems, Lappeen an a-Lah i Uni e si y o Technology (LUT), 53850 Lappeen an a, Finland
2School o Elec ical and Compu e Enginee ing, S a e Uni e si y o Campinas (UNICAMP), Campinas 13083, B azil
3Nokia Bell Labs, 02610 Espoo, Finland
4Depa men o Telecommunica ions, Facul y o Elec ical Enginee ing and Communica ion, B no Uni e si y o Technology, 601 90 B no, Czech Republic
5Uni o Elec ical Enginee ing, Tampe e Uni e si y, 337 20 Tampe e, Finland
6Ins i u e o Au oma ion and Compu e Science, Facul y o Mechanical Enginee ing, B no Uni e si y o Technology, 601 90 B no, Czech Republic
7Depa men o Compu e Science, Fede al Uni e si y o La as, La as, 37200 Minas Ge ais, B azil
Co esponding au ho : Dick Ca illo Melga ejo ([email p o ec ed])
This wo k was suppo ed in pa by he Academy o Finland h ough he F amewo k o he Iden i ica ion o Ra e E en s ia MAchine
lea ning and IoT Ne wo ks (FIREMAN) Conso ium unde G an CHIST-ERA-17-BDSI-003/n.326270 and Ene gyNe Resea ch
Fellowship unde G an 321265/n.328869, and in pa by he Jane and Aa os E kko Founda ion h ough he Swa ming Technology o
Reliable and Ene gy-awa e Ae ial Missions (STREAM) P ojec .
ABSTRACT The applica ion o lying base s a ions (FBS) in wi eless communica ion is becoming a key
enable o imp o e cellula wi eless connec i i y. Following his endency, his esea ch wo k aims o enhance
he spec al e iciency o FBSs using he adio access ne wo k (RAN) slicing amewo k; his op imiza ion
conside s ha FBSs’ loca ion was al eady de ined p e iously. This amewo k spli s he physical adio
esou ces in o h ee RAN slices. These RAN slices schedule esou ces by op imizing indi idual slice
spec al e iciency by using a deep ein o cemen lea ning app oach. The simula ion indica es ha he
p oposed amewo k gene ally ou pe o ms he spec al e iciency o he ne wo k ha only conside s he
heu is ic p ede ined FBS loca ion, al hough he gains a e no always signi ican in some speci ic cases.
Finally, spec al e iciency is analyzed o each RAN slice esou ce and e alua ed in e ms o se ice-le el
ag eemen (SLA) o indica e he pe o mance o he amewo k.
INDEX TERMS Flying base s a ions, UAVs, loca ion op imiza ion, wi eless communica ion, deep-
ein o cemen lea ning.
I. INTRODUCTION
Ex ensi e de elopmen s in he ield o unmanned ae ial
ehicles (UAVs) ha e opened many oppo uni ies o new
applica ions in bo h p i a e and public domains, such
as su eillance, anspo a ion, en i onmen al moni o ing,
indus ial moni o ing, ag icul u e se ices, and disas e
elie [1], [2]. Recen ly, he inc easing numbe o use cases
employ UAVs as wi eless ho spo s o elays o ex end
ne wo k co e age in a eas whe e i is equi ed. Mo e-
o e , nowadays, he e a e UAV applica ions used as a ool
o communica ions a he applica ion le el, o example,
The associa e edi o coo dina ing he e iew o his manusc ip and
app o ing i o publica ion was Jiankang Zhang .
in o ma ion sha ing in social media o sea ching o missing
pe sons. Ano he example is he ecen loods in Ge many [3],
which showed ha he in as uc u e is s ill qui e ulne a-
ble. The e o e, i is wo h pu suing solu ions o o e come
p oblems when he egula communica ion in as uc u e
s ops wo king. Thus, he use o UAVs p o ides an essen ial
esou ce o allowing he con inui y o communica ions and
suppo ing human ope a o s o con inue o communica ing in
sea ch and escue ope a ions, he eby gua an eeing e icien
ope a ion [4]. In such scena ios, he op ion o apidly and
e icien ly deploying a lee o d ones is c ucial in quickly
es ablishing a communica ion ne wo k capable o sa ing
li es, especially as i migh be di icul o use e es ial
means comp ising empo a y ne wo king equipmen , such as
53746 This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/ VOLUME 10, 2022
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
a cell on wheels in na u al disas e s. This ea u e makes UAVs
unique and c ucial o deploymen in such use cases [5].
In addi ion, deploying UAVs as lying base s a ions
(FBS) has also ecen ly eme ged as a easible esponse o
highly localized a ic demands in nex -gene a ion cellula
ne wo ks [6], [7]. Using UAVs in such a way p o ides an
oppo uni y o exploi hei agili y o mo ion o imp o e he
ai - o-g ound link capaci y by op imal ai placemen [8],
[9]. Typically, he abo e-men ioned use cases conside sig-
ni ican ly la ge a eas whe e mul iple UAVs mus be used.
Howe e , his leads o wo majo p oblems. Fi s , he UAVs
mus be posi ioned o op imally co e as many use s as
possible [10], [11]. Second, he in acell and in e cell in e -
e ence mus be mi iga ed [12]. The i s p oblem can be
e ec i ely app oached by using heu is ic algo i hms. These
algo i hms can p o ide a solu ion wi h a low compu a ional
ime and good esul s, as shown in [13], o example. In he
case o in acell in e e ence, he sys em pe o mance can be
imp o ed wi h a a ie y o mul iple access echniques, such
as o hogonal equency-di ision mul iple access (OFDMA).
When in e cell in e e ence is aken in o accoun , some popu-
la schemes, such as equency euse, g aph heo y, and coop-
e a i e mul i-poin (CoMP) [9], [14]–[18], can be employed.
In ou p e ious wo k [13], we add essed he UAVs posi-
ion op imali y using an heu is ic me hodology. Howe e ,
he adio channel in e e ence p oblem was no add essed in
de ail. The p esen wo k ex ends [13] by employing a ep e-
sen a i e wi eless channel model. In addi ion, we p oposed a
adio access ne wo k (RAN) slicing amewo k ha enables
he alloca ion o adio esou ces (slices) ca ying speci ic da a
se ices. Ou p oposed amewo k aims o accommoda e a
di e si y o se ices o e a single sha ed i h gene a ion (5G)
in as uc u e and lays he ounda ion o ine-g ained se ice
managemen in FBS ne wo ks. We ha e conside ed ha an
agile RAN slicing amewo k is an app op ia e solu ion o
achie e he pe o mance equi emen s in oduced by e icals
on 5G communica ion ne wo ks. The RAN slicing ame-
wo k comp ises se e al in e wo king unc ional componen s,
aiming a a lexible ins an ia ion o adio se ices, ha can
cope wi h he inc easing complexi y o suppo ing FBS se -
ices. In ou wo k, we conside h ee slices: enhanced mobile
b oadband (eMBB), ul a- eliable low-la ency communica-
ion (URLLC), and massi e machine- ype communica ions
(mMTC).
The alloca ion o hese slices is achie ed by op imizing a
cos unc ion ha is di ec ly ela ed o he spec al e iciency
(SE) o he downlink da a ansmission, which is cons ained
by he maximum powe ansmission and he numbe o
RAN slices. A cellula ne wo k based on subchannels usually
has a high p obabili y o in e cell in e e ences in he edge
cell. To sol e his in ica e alloca ion p oblem, we in o-
duce an in elligen componen in he amewo k—i.e., a
deep ein o cemen lea ning (DRL) model— ha imp o es
he sys em pe o mance and manages he adio esou ce allo-
ca ion minimizing he in e e ence. By using ou p oposed
in e e ence managemen me hodology o op imize he SE
on each RAN slice, speci ic se ice-le el ag eemen (SLA)1
can be achie ed be ween he ne wo k se ice p o ide and he
cus ome .
To acili a e eade s comp ehension o his pape , he main
con ibu ions o his pape a e summa ized as ollows:
•enhancemen o he UAV loca ion dis ibu ion algo-
i hm p oposed in [13], using a p ope ai - o-g ound
channel model o enable aa app op ia e in e e ence
analysis.
•a no el RAN slicing amewo k is p oposed o enable
he use o ad anced machine lea ning echniques, such
as DRL.
•we p opose a dis ibu ed DRL app oach o mi iga e he
downlink in e e ence, in which each FBS ope a es as an
independen lea ning agen .
• h ee ep esen a i e scena ios o FBSs a e desc ibed and
analyzed in de ail o compa e he SLA pe o mance
be ween he DRL and he benchma k.
•a mul iagen lea ning echnique is p oposed o op imize
a noncon ex p oblem in he FBS sys em model.
The es o he pape is o ganized as ollows. The model
inding op imal placemen o UAVs in a gi en a ea, used as
benchma k in his esea ch wo k, is p esen ed in Sec ion II. In
Sec ion III, he RAN slicing amewo k is de ined, including
he sys em model and he DRL me hodology o alloca e he
adio esou ces. A de ailed desc ip ion o he op imiza ion
sequence is gi en in Sec ion IV. The simula ion se up is
p esen ed in Sec ion V. Nume ical esul s oge he wi h a
ho ough compa a i e pe o mance analysis a e discussed in
Sec ion VI. Ou concluding ema ks and u u e wo k a e
p esen ed in Sec ion VII.
II. UAV LOCATION OPTIMIZATION
The e ec i e deploymen o UAVs ac oss a selec ed a ea is a
di icul ask ha alls in o he ca ego y o N P−comple e
class o p oblems [19]. To add ess his ask, we enhance
he model p esen ed in [13] o loca ion co e ing o he
UAV deploymen in on-demand connec i i y scena ios. The
enhancemen ocuses in he elimina ion o in e e ence o all
pai s o newly added cen es and o new and exis ing cen es.
The p oposal in [13] did an ex ended explana ion o he UAV
loca ion me hodology. The main idea o his deploymen was
o selec a easible loca ions whe e UAVs can be loca ed by
his heu is ic me hodology. Based on ha , he op imiza ion
algo i hm selec ed he sui able UAV loca ion o compose a
lis o UAVs and hei espec i e loca ions.
A. DEPLOYMENT MODEL
To acili a e he unde s anding o he model, we p o ide he
e minology used in he es o his pape adap ed o he
e ms used in he li e a u e in Table 1. To localize he sui -
able posi ions o he FBS deploymen , loca ion op imiza ion
1SLAs es ablish cus ome expec a ions ega ding he se ice p o ide ’s
pe o mance and o e all quali y. I is a con ac be ween he ne wo k se -
ice (NS) p o ide and he cus ome .
VOLUME 10, 2022 53747
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
TABLE 1. Mapping ma hema ical e minology o communica ion
ne wo ks e minology.
p oblems is aken as inspi a ion. Cu en ly, he e exis se e al
acili y loca ion p oblems dealing wi h many eal-wo ld use
cases. Simply, hey can be di ided in o loca ion se co e -
ing p oblems (LSCPs) [20], and maximal co e ing loca ion
p oblems (MCLPs) [21]. The LSCP a ge s he minimiza ion
ques ion in which he numbe o acili ies ha sa is y he ne -
wo k equi emen s and he need o be loca ed is minimized.
On he o he hand, in he MCLP, a p ede ined numbe o
esou ces ies o maximize i s co e age. The main di ision
is based on he a ailable esou ces. Because hese models
ha e been used o a wide ange o applica ions, hey a e no
ied o elecommunica ion ne wo k deploymen only. Hence,
o he bes o ou knowledge, he e is a gap in he li e a u e
ha [13] aims o b idge o hese models and he use case o
UAV deploymen . As a gap, we see he ollowing ac o s (o
hei combina ion in one model):
(i) Sepa a ing he capaci y o acili ies o loca ions co e ing
bo h downlink and uplink. This may di e o each
loca ion o acili y.
(ii) Conside a ion o he exis ing se ices; o he use case
o UAVs, i is essen ial o conside he exis ing in as-
uc u e ha can se e a leas some demand om he
loca ions o be co e ed ad hoc.
(iii) Spli ing capaci y equi emen s om one loca ion o
only one acili y a a gi en momen .
(i ) Co e ing some loca ions wi h ze o o a highe numbe
o acili ies. This is c ucial o he mus -ha e loca ions
whe e i is no accep able o lose he connec i i y.
( ) The o e simpli ied wi eless in e e ence is based on
he o e laps be ween cellula cells. We elimina e he
in e e ence by Eq. (8) and (9).
In [13], i is assumed ha co e age a ailabili y is gua -
an eed. Capaci y conside a ions a e c i ical in he 5G-and-
beyond deploymen s ha expec a signi ican inc ease in
ne wo k a ic. This is due o he g ow h o se ices ha ha e
conside ably highe ne wo k h oughpu equi emen s, such
as he g ow h o high-de ini ion ideos, augmen ed eali y
(AR) / i ual eali y (VR), machine- o-machine communica-
ion, and o he e y in ensi e o demanding se ices in e ms
o ne wo k equi emen s. In pa icula , we ha e o deal wi h
a high densi y o use s ha a e simul aneously connec ed.
Fo he exis ing acili ies E and hei co esponding decision
a iables xi, whe e i∈E , we se his pa ame e o 1, which
means ha all he exis ing acili ies a e aken in o accoun .
The alloca ion o capaci y equi emen s be ween uploads and
downloads ep esen s a spli o 100 Mbps o 80 Mbps o
download (mo e ex ensi e) and 20 Mbps o upload. In addi-
ion, we s ill need o sa is y he equi emen ha he demand
jbo h o download and upload mus be assigned o he same
acili y i.
To de i e a ma hema ical model, le us se he ollowing
no a ion:
•I=a se o acili y si es (UAV o FBS) 1,2,...,m;
•J=a se o demand a eas (cus ome s) 1,2,...,n;
•dij = he sho es dis ance be ween acili y iand
demand j;
•Dmax =maximum dis ance which will be accep ed o
ope a ion be ween he acili ies and demands;
•lj=numbe o acili ies equi ed o se icing demand j;
•xi∈ {0,1}, whe e xi=1 means ha acili y iis selec ed,
while xi=0 means ha i is no selec ed.
•Nj= {i|dij ≤Dmax} = he se o acili ies i ha can
co e he demand loca ion j;
•Cu
i=upload capaci y o acili y i;
•Cd
i=download capaci y o acili y i;
•au
j=upload amoun o demand a j;
•ad
j=download amoun o demand a j;
•yij ∈ {0,1} = non agmen ed demand om loca ion jis
assigned (1) o is no assigned (0) o acili y i.
Now, we se ou he ollowing model ex ac ed om [13]
o minimize he numbe o equi ed FBSs and maximize he
cellula co e age a ea.
min X
i∈I
xi,(1)
subjec o ∀j∈J:X
i∈Nj
xi≥lj(2)
∀j∈J:X
i∈Nj
yij =1 (3)
∀i∈Nj:Cu
ixi≥X
j∈J
yijau
j(4)
∀i∈Nj:Cd
ixi≥X
j∈J
yijad
j(5)
(∀i∈I)(∀j∈J):yij ≤xi(6)
∀i∈E :xi=1 (7)
(∀i∈I−E )(∀j∈I−E )(i6= j):
dij ≥(xi+xj−1)dmin (8)
(∀i∈I−E )(∀j∈E ):dij >dminxi(9)
∀i∈I:xi∈ {0,1}(10)
(∀i∈I)(∀j∈J):yij ∈ {0,1}(11)
Cons ain (3) gua an ees ha he demand jis assigned o
only one acili y a a gi en momen . All selec ed acili ies
mus ha e a su icien sum o hei capaci ies o uploads and
downloads o co e all upload and download demands (in
53748 VOLUME 10, 2022
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
p ac ice, his is an ideal case ha ne wo k ope a o s a e ying
o each wi h he a ailable esou ces), his is gua an eed by
cons ain s (4) and (5). I a acili y is selec ed o be emo ed
om he ne wo k in as uc u e, none o he demand should
be assigned o i ; his cons ain is gi en by (6). In [13], he
in e e ence is simpli ied o minimize he co e age o e laps
de ined by he cells. Finally, conside he ollowing: i dij,
i∈I,j∈Iis he dis ance be ween he acili ies iand j, hen
we can se ha o all pai s o selec ed acili ies, he acili ies
will ha e a dis ance g ea e o equal han a ce ain h eshold,
which is gua an eed by cons ain (8).
Fu he , as i is ypical in he s a e o he a dealing wi h
loca ion co e age wi h model enhancemen s, au ho s in [13]
p o ided an al e na i e maximiza ion model ha conside s a
p ede ined numbe o new acili ies (no ye op imized) o be
loca ed and co e ed as much a ea as possible:
max X
i∈NjX
j∈J
yij(au
j+ad
j),(12)
subjec o he same cons ain s as in he minimiza ion
model, bu wi h he addi ion o he ollowing cons ain o
a p ede ined numbe o new acili ies.
X
i/∈E
xi=p(13)
No e ha his model a ge s he localiza ion o FBS nodes,
which de ines he benchma k. This in o ma ion is used as
inpu in he op imiza ion o FBS based on (RAN) slicing
amewo k, which is de ailed in Sec ion III-E.
The b ie explana ion o he algo i hm implemen ing he
model is de ailed in Algo i hm 1. The Ge Inpu Da a pa ep-
esen s he lis exis ing acili ies, expec ed/exis ing demands
wi h coo dina es, and addi ional impo an me ada a. Fo
he Co e ingModel compu a ion pa he heu is ics needs o
implemen epai ope a o sa is ying all he model cons ain s
(e.g., adding new UAVs o FBSs o he lis o solu ion o
sa is y he capaci y equi emen s).
B. CONSIDERATIONS OF THE COMPUTATIONAL
COMPLEXITY MODEL
The size o he sea ch space is de e mined by he numbe o
all possible selec ions o acili ies. Fo m acili ies, acco ding
o he binomial heo em, i is equal o
m
1+m
2+ · · · + m
m=(1+1)m−1=O(2m).(14)
Fu he mo e, we need o ind he mos complex con-
di ion in ex ended models o m<n(whe e nis he
numbe o demand a eas) o ind he esul ing compu a-
ional complexi y. In he minimiza ion model, hese a e (6)
and (11) in he co esponding equa ions o he maximiza ion
model, which equi e m·nope a ions. This is based on
he ac ha he esul ing ime complexi y o hese models
is O(2mmn) [13].
Algo i hm 1 Algo i hm Tha Op imizes he UAVs o FBSs
Loca ion
FMain pa
1: unc ion indLoca ions
2: Ge Inpu Da a
3: Gene a e heo e ical possible UAVs loca ions
4: Apply Co e ingModel() wi h hese da a
FCo e ing model
5: unc ion Co e ingModel()
6: Gene a e possible solu ions
7: Apply epai ope a o p o iding easible solu ions
8: Apply selec ed heu is ics o ind op imal solu ion
Sui able loca ions o he UAV deploymen
III. RAN SLICING FRAMEWORK
To complemen he benchma k desc ibed in Sec ion II, in his
sec ion we desc ibe he p oposed RAN slicing amewo k.
We employ he s anda dized de ini ions o RAN slicing in
5G, he sys em channel model, he adio op imiza ion p ob-
lem o mula ion, and ou p oposed app oach using DRL.
A. FRAMEWORK DESCRIPTION
The di e se pe o mance equi emen s in oduced by 5G
communica ion ne wo ks a e e ile g ound o he appli-
ca ion o an agile RAN slicing amewo k. Ou p oposed
amewo k aims o accommoda e a di e si y o se ices o e
a single sha ed 5G in as uc u e and lays he ounda ion
o ine-g ained se ice managemen in FBS ne wo ks. This
RAN slicing amewo k p ima ily comp ises se e al in e -
wo king unc ional componen s, aiming a a lexible ins an i-
a ion o adio se ices. The p oposed a chi ec u e is de ised
o cope wi h he ising complexi y o suppo ing FBS se -
ices, achie ing no only mo e manageable RAN slices bu
also con o ming he business p oposi ions sough by ne wo k
ope a o s and se ice p o ide s akeholde s.
This amewo k comp ises o hogonal physical esou ces
ha spli he a ailable bandwid h o suppo a speci ic numbe
o ne wo k slices. In his speci ic wo k, we conside h ee
slices: eMBB, URLLC, and mMTC. In a cellula ne wo k
based on subchannels, he RAN slicing amewo k is exposed
o a high p obabili y o in e cell in e e ence, specially in he
edge cell. To add ess his issue, we inco po a e an in elligen
componen in he amewo k o manage he adio esou ce
alloca ion using DRL. This in e e ence managemen aims o
achie e speci ic SLA policies be ween he ne wo k se ice
p o ide and he cus ome by op imizing he SE on each RAN
slice. A g aphical desc ip ion o he concep o he p oposed
amewo k is sha ed in Fig. 1. The sys em model and he DRL
me hodology a e de ailed in he ollowing sec ion.
B. SYSTEM MODEL
Conside a se Io IFBS p o iding downlink wi eless se -
ice o a g oup o use equipmen s (UEs) in a geog aphical
a ea A. Each FBS i∈Ise es an a ea Ai, such ha
VOLUME 10, 2022 53749
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
FIGURE 1. The RAN Slicing amewo k emphasizing he ole o he DRL module. In his pa icula case we conside ha each slice uses dedica ed
physical esou ces in each FBS. Thus, he cogni ion elemen op imizes he in e e ence educ ion using he deep- ein o cemen lea ning app oach.
∪∀i∈IAi=Aand Ai∩Ak6= ∅ o any i6= k∈I. In o he
wo ds, we conside ha when UAVs a e alloca ed by he
op imiza ion algo i hm p esen ed in Sec ion II, i is possible
ha some cells ha e a signi ican in e sec ion be ween hem.
The pa h loss o he ai - o-g ound communica ion link
om a ypical FBS loca ed a xi∈R3 o a ypical g ound
UE ha is loca ed a y∈R3is gi en as ollows [22]:
h( )
→n[dB](xi,y)=20 log10 4π ckxi−yk
c+ξ(xi,y),
(15)
whe e cis he ca ie equency o FBS downlink com-
munica ions, kxi−ykis he FBS–UE dis ance, cis he
speed o ligh , and ξ(xi,y) is he addi ional pa h loss o he
ai - o-g ound channel, compa ed wi h he ee space p op-
aga ion. The alue o ξ(xi,y) can be modeled as a Gaus-
sian dis ibu ion wi h di e en pa ame e s (µLOS, σ2
LOS) and
(µNLOS, σ2
NLOS) o line-o -sigh (LOS) and non-line-o -sigh
(NLOS) links, espec i ely. Then, he downlink spec al e i-
ciency achie ed by he RAN slice m, he use n, a he ime
slo om he FBS loca ed a xi o a UE loca ed a y∈Aiis
C( )
n,m(xi,y)=log21+γ( )
n,m(xi,y),(16)
whe e γ( )
n,m(xi,y) is he signal- o-in e e ence-plus-noise
(SINR) a he use n, on he RAN slice m, a he ime slo ,
which is de ined by (17)
γ( )
n,m(xi,y)=β( )
l,mg( )
l→n,m(xl,y)p( )
ll=n
P
6=l
β( )
,mg( )
→n,m(x ,y)p( )
+σ2
n
,(17)
whe e β( )
,mis he bina y a iable ha indica es he RAN
slicing selec ion m ansmi ed om he UAV a ime ,
g( )
→n,m(x ,y) indica es he downlink channel gain om he
FBS o he use non he RAN slice min he ime slo
when he UE is loca ed in he posi ion yand he FBS in he
posi ion x ∈R3,p( )
is he ansmi powe o he UAV in
he ime slo , and σ2is he addi i e whi e Gaussian noise
powe spec al densi y a he use ecei e n.
g( )
→n,m(x ,yn)=h( )
→n(x ,yn)α( )
n→l,m
2 =1,2,· · · ,
(18)
whe e h →n(x ,yn) is he pa h loss in a linea scale, which
is calcula ed in (15), and α( )
n→l,mis he small-scale Rayleigh
ading.
The p obabili y o ha ing an LOS link be ween he FBS j
loca ed a xjand he UE loca ed a yis gi en by [22]:
PLOS(xj,yi)=1
1+aexp −b180
πσ(xj,y)−a,(19)
53750 VOLUME 10, 2022

D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
whe e aand ba e cons an alues ha depend on he com-
munica ion en i onmen , σ(xj,y)=sin−1Hj
kxj−ykis he
ele a ion angle, and Hjis he al i ude o he FBS j. Then,
he a e age downlink SE be ween an FBS iand he UE a
nloca ed in yn∈Aiwill be:
¯
C( )
n,m(xi,yn)=PLOS(xi,yn)C( )LOS
n,m(xi,yn)
+(1 −PLOS(xi,yn))C( )NLOS
n,m(xi,yn).(20)
C. RAN SLICING IN 5G
A simpli ied 5G logical a chi ec u e is composed o a co e
cloud, an edge cloud, and an RAN. The co e cloud p o-
ides gene ic con ol plane signaliza ion, slice managemen ,
mobili y managemen , and au hen ica ion. The edge cloud
pe o ms some use plane unc ions as a packe /se ice ga e-
way (P/S-GW) o imp o e la ency communica ion on c i -
ical applica ions. I also enables da a o wa ding, con ol
plane unc ions, and mobile edge compu ing pla o ms, such
as con en s o age se e s. In he adio access plane, he
3 d Gene a ion Pa ne ship P ojec (3GPP) de ines he nex -
gene a ion RAN (NG-RAN), which is comp ised o nex -
gene a ion NodeBs (gNBs) connec ed o he co e ne wo k.
This a chi ec u e is used o suppo he ne wo k slicing
app oach p oposed in 5G. In his aspec , he e a e wo ypes
o subne s in he 5G slicing a chi ec u e: co e ne wo k slice
subne s and RAN slice subne s.
In he co e ne wo k slice subne s, he ne wo k slicing ope -
a ion used in he co e ne wo k is con olled by he ne wo k
slicing managemen . I is composed o he i ualized ne -
wo k unc ion managemen (VNFM), he so wa e-de ined
ne wo k (SDN) con olle , he managemen and o ches a ion
uni , and he i ualized in as uc u e managemen (VIM).
The VNFM maps he physical ne wo k unc ions o i ual
machines (VMs); he SDN con olle manages and ope -
a es he en i e i ual ne wo k; he VIM alloca es i ualized
esou ces o VMs; and he managemen and o ches a ion uni
c ea es, ac i a es, and dele es ne wo k slices based on he
se ice equi emen s.
In he RAN slice subne s, he gNB is a c ucial enable
o ne wo k slices. I p o ides RAN slice subne s ha a e
composed o a cen alized uni (CU), mul iple dis ibu ed
uni s (DUs), and mul iple adio uni s (RUs). The gNB unc-
ionali ies a e dis ibu ed in a lexible manne be ween he
CU, DUs, and RUs. To manage hei li e cycles, he s anda d
speci ies he RAN ne wo k slice subne empla e (NSST)
and wo managemen en i ies, such as he RAN ne wo k
slice subne managemen unc ion (NSSMF) and he ne wo k
unc ion managemen unc ions (NFMFs) [23].
The co e ne wo k slice subne s ha e been s udied and
de eloped in he cu en 5G wi h ou s anding esul s. How-
e e , he RAN slicing is s ill an open opic, and i is no
ye s anda dized. The RAN slicing aims o imp o e he
e icien usage o a ailable physical adio esou ces and
simul aneously gua an ees he SLA policies imposed in each
slice.
D. REINFORCEMENT LEARNING AIDED UAVs
Machine lea ning is an app oach ha has become inc eas-
ingly popula o sequen ial decision-making on wi eless
communica ion ne wo ks wi h applica ions in many di e se
a eas, such as sma g ids, sel -d i ing ca s, and obo ics.
The e a e h ee machine lea ning ca ego ies, depending on
he na u e o he in o ma ion o eedback a ailable o he
lea ning sys em: (i) supe ised lea ning; (ii) unsupe ised
lea ning; and (iii) ein o cemen lea ning (RL). In his pape ,
we use RL as he p ima y app oach o op imizing he cos
unc ion based on spec al e iciency. RL is a echnique ha
is conce ned wi h how agen s should de e mine he sequences
o ac ions in an en i onmen ha will maximize cumula i e
ewa ds [24], [25]. I is a ial-and-e o p ocess whe e an
agen in e ac s wi h an unknown en i onmen in a sequence
o disc e e ime s eps o achie e a ask. A ime , he agen
i s obse es he cu en s a e o he en i onmen , which
is a uple o ele an en i onmen ea u es and is deno ed
as S( )∈S, whe e Sis a se o possible s a es. I hen
akes an ac ion a( )∈A om an allowed se o ac ions A
acco ding o a policy ha can be ei he s ochas ic, i.e., π
wi h a( )∼π(.|S( )) o de e minis ic, i.e., µwi h a( )=
µ(S( )). Because he in e ac ions a e o en modeled as a
Ma ko decision p ocess, he en i onmen mo es o a nex
s a e S( +1) ollowing an unknown ansi ion ma ix ha maps
s a e–ac ion pai s on o a dis ibu ion o successi e s a es, and
he agen ecei es a ewa d S( +1). O e all, he abo e p ocess
is desc ibed as an expe ience a +1 deno ed as e( +1) =
(S( ),a( ), ( +1),s( +1)).
The goal is o lea n a policy ha maximizes he cumula i e
discoun ed ewa d a ime , de ined as ollows:
R( )=
∞
X
τ=0
γτ ( +τ+1),(21)
whe e γ∈(0;1] is he discoun ac o .
RL has been g owing in popula i y because i does no
equi e an ex ensi e ne wo k model. Ins ead, i s lea ning
p ocess is based on he in e ac ions wi h he en i onmen
ha p oduces i s op imal s a egies. Owing o he possibili y
o combining RL wi h deep lea ning [26], DRL is a highly
sui able me hod o sol ing p oblems wi h a high numbe
o s a es and low p io knowledge, which is he case o he
esou ce alloca ion scena io in RAN slicing.
DRL has ecen ly been used in p oblems ela ed o
UAVs [27]–[31]. Mo eo e , he DRL app oach has been
exploi ed and applied o he p oblem o UAV posi ion and
esou ce alloca ion. In [30], he au ho s p oposed a DRL algo-
i hm based on echo s a e ne wo k (ESN) cells o op imizing
he UAV pa h, cell associa ion o minimize he in e cell in e -
e ence le el, ansmission delay, and ansmi powe le el.
In [32], he ESN algo i hm was used based on a mul iagen
Q-lea ning app oach, which was employed o p edic he
u u e posi ions o UEs and de e mine he posi ions o UAVs.
Howe e , his wo k did no conside UAV coope a ion and
capaci y limi a ions o on haul links be ween UAVs and
VOLUME 10, 2022 53751
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
egula base s a ions. Fu he , o he op imiza ion o FBS
placemen , he s udies in [27], [28], [33] used an RL algo-
i hm. In his wo k, we p esen an RAN slicing amewo k
based on a DRL me hodology ha complemen s he loca ion
op imiza ion model ob ained in [13] by adding a adio chan-
nel model and op imizing he RAN slice esou ces be ween
mul iple FBSs co e ing an a bi a y a ea.
E. RADIO OPTIMIZATION PROBLEM FORMULATION
To apply he DRL me hodology explained in he p e ious
subsec ion, we de ine he adio op imiza ion p oblem ha his
pape aims o op imize. Thus, de ails o he cos unc ion and
i s cons ain s a e de ined in he ollowing.
Deno ing RAN slices and powe ec o s in he ime slo as
β( )=hβ( )
1,1, β( )
1,2,· · · , β( )
N,MiTand p( )=hp( )
1,· · · ,p( )
NiT
espec i ely, we de ine he sum- a e maximiza ion p oblem
as
max
p( ),α( )
N
X
n=1
¯
C( )
n(xi,yn)
s. . 0 ≤p( )
n≤Pmax,∀n∈N,
β( )
n,m∈ {0,1},∀n∈N,∀m∈M,
X
m∈M
β( )
n,m=1,∀n∈N,(22)
whe e ¯
C( )
n=PM
m=1¯
C( )
n,m(xi,yn).
The noncon ex p oblem in (22) equi es a highly complex
app oach ha could also inc ease he compu a ional com-
plexi y. To handle his noncon ex p oblem, we conside a
mul iagen lea ning scheme, whe e each ansmi e , moun ed
in each FBS, ope a es as an independen lea ning agen . Each
agen success ully execu es wo policies o de e mine i s asso-
cia ed RAN slice and ansmission powe le el. The p oposed
mul iagen app oach is easily scalable o mo e ex ensi e ne -
wo ks and can ope a e wi h local in o ma ion a e aining.
The componen s o he DRL me hodology conside ed
based on he sys em model desc ibed be o e is composed by:
•Agen s: in he mul iple lea ning app oach, he FBSs
ep esen he agen s.
•Policies: wo well de ined policies a e conside ed. π1 o
choose an speci ic RAN slice, and he π2 o selec a
p ope powe le el o each use .
•Ac ions: we conside wo well de ined g oup o ac ions.
The disc e e ac ions ela ed o he selec ion o RAN
slices, and he con inuous ac ion o choose he powe
ansmission o each indi idual use .
•S a es: I is composed by a uple o in o ma ion ela ed
o he RAN slice alloca ion, he SE, in e e ence in each
indi idual use , gain and in e e ence in each use .
•Rewa ds: a p opo ional alue o he SE in each ecei e
(UE) is used as ewa d. I conside s he ollowing c i e-
ia: he SE is e alua ed in e e y use wi h he condi ion
o one neighbou hood base s a ion (BS) o agen is no
ansmi ing. Thus, i he SE alue is signi ican , hen
he BS being e alua ed is penalized. In con as , i he
SE emains, hen he BS is ewa ded.
A he beginning o each ime slo , each agen successi ely
execu es wo policies o de e mine i s associa ed ansmis-
sion powe le el and RAN slice selec ion. Fo his pu pose,
he DRL conside s wo op imiza ion app oaches. The i s
conside s a Deep Q-ne wo k o op imize a s ochas ic policy
ha aims o imp o e he RAN slice selec ion. A second
Deep Q-ne wo k op imizes a de e minis ic policy o selec a
sui able powe ansmission alue. The agen o he second
Deep Q-ne wo k equi es he RAN slice decision o he i s
app oach o de e mine i s s a e inpu be o e se ing he ans-
mi powe o he agen . A b ie explana ion o his app oach
is done in Algo i hm 2.
Algo i hm 2 Algo i hm o he DRL App oach
FMain Loop
1: while S op C i e ia no me do
2: RAN slice Selec ion()
3: Powe Con ol()
FRAN slice selec ion
4: unc ion RAN slice Selec ion()
FAc ion Selec ion
5: ac ion ←a( )
n∈ARAN-Slice = {1,· · · ,M} = M
6: S a e se design ←s( )
n,m
7: T aining by a deep-Q Ne wo k
8: Rewa d unc ion design
9: Upda e Policies: π1
FPowe Con ol
10: unc ion Powe Con ol()
FAc ion Selec ion
11: ac ion ←a( )
n,a( )
n
∈Apowe =[0,1]
12: S a e se design ←s( )
n,a( )
n
13: T aining by a deep-Q Ne wo k
14: Rewa d unc ion design
15: Upda e Policies: π2
IV. DESCRIPTION OF THE PROPOSED SOLUTION
This pape aims o complemen and enhance he ou pu
ob ained in [13], which is no op imal i e alua ed in a eal
scena io. Thus, he esou ce alloca ion ha we p opose is
condi ioned o he p eloca ion o each FBS ob ained using
he me hodology p esen ed in Sec ion II. As he bench-
ma k does no conside any channel model o e alua e
he in acell and in e cell in e e ence, i can ge subop-
imal esul s in p ac ical wi eless scena ios. In pa icula ,
we aim o add ess he ollowing esea ch ques ions ha
a ise du ing he FBS ne wo k deploymen in a eal
scena io:
•Wha can be he po en ial imp o emen s based on he
FBSs p eloca ion de ined in [13]?,
53752 VOLUME 10, 2022
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
FIGURE 2. Op imiza ion low diag am showing majo s eps and s ages.
He e, we emphasis he op imiza ion done in he benchma k (in yellow)
and highligh he DRL app oach (blue).
•Wha is he pe o mance o he simula ion se up when
di e en se ices a e suppo ed by he FBS ne wo k?,
•Is i possible o de elop a p ac ical op imiza ion me hod
ha is capable o imp o ing he pe o mance o FBSs?.
To add ess hese ques ions, we use a s anda d simula ion
model de ined by 3GPP and a RAN slicing amewo k p o-
posed in Sec ion III, which is p oposed o analyze he ne wo k
pe o mance using a DRL me hodology o op imize he adio
esou ces.
The whole op imiza ion p ocess is di ided in o h ee
phases— da a ma ix gene a ion, FBS loca ion minimiza ion,
and RAN slicing op imiza ion—as shown in Fig. 2. He e,
S ages 1 and 2 ha e al eady been applied and e i ied in he
o iginal publica ion [13]. De ails o each s age a e desc ibed
in he ollowing pa ag aphs.
A. STAGE 1—DATA MATRIX GENERATION
In his s age, use s a e gene a ed andomly and uni o mly in
one speci ic and ixed a ea. The FBSs a e gene a ed acco d-
ing o he desi ed adius. The la ge he adius is, he lowe he
numbe o FBSs is equi ed. A la ge adius causes a highe
pe cen age o o e laps. No e ha S ages 1 and 2 do no ake
in e e ence in o conside a ion. Thus, in e e ence has o be
a oided by limi ing he adio o e lapping. The use s a e gi en
andom da a a e equi emen s acco ding o he a ic mix.
The ou pu o his s age is a ma ix whose ows ep esen all
FBSs and columns ep esen all use s. The ma ix is illed
wi h ze os i he use is inside he speci ic FBS cell adius,
and o he wise, i is illed wi h ones. This ma ix is comple ed
and used as he inpu o S age 2.
B. STAGE 2—FBS LOCALIZATION
This s age is composed o a so wa e se ice ha uses he
ou pu o S age 1. This s age aims o op imize he ows o
he gene a ed ma ix so ha he cus ome equi emen s a e
me . In his s age, we apply he op imiza ion model om
Sec ion II-A. Because he complexi y o selec ing he op imal
ows (FBSs) om he ma ix is O(2m), heu is ic algo i hms
a e applied. As a consequence, we use he di e en ial e o-
lu ion and cuckoo sea ch algo i hms wi h epai ope a o s
p esen ed in [13]. The ou pu o he compu a ion is a se
o FBSs (wi h hei loca ions om he p e ious s age) ha
should be used in he ne wo k deploymen . Fu he , he loca-
ion o each FBS is used as an inpu o S age 3, conside ing a
ep esen a i e in e e ence channel model.
C. STAGE 3—OPTIMIZATION OF RAN SLICES
We add an in e e ence model based on he 3GPP s anda d
o complemen he p e ious s ages o analyze and mi iga e
downlink in e e ence. Then, we calcula e he SINR, de ined
in (17), o each use conside ing he signal s eng h coming
om he se ing UAV and om he FBSs ha a e in e e ing
wi h ha speci ic use n. Using (22), ou simula ion model
calcula es an op imal spec al e iciency o each use n
ollowing he p oposed DRL app oach on he RAN slicing
amewo k de ined in Sec ion III.
V. SIMULATION SETUP
Two scena ios we e designed o emphasize he gain o he
op imiza ion models ha we e p e iously explained. In bo h
cases, 1000 use s we e deployed ollowing a uni o m dis i-
bu ion in a speci ic a ea. Subsequen ly, in acco dance wi h
he cons ain s in each scena io, he FBSs loca ion op imize
de e mined he coo dina es o each FBS. The cons ain s a e
mos ly ela ed o he cell adius, ha depends on he adio
equency ope a ion, and h oughpu demand o each use o
g oup o use s.
Using he p ede e mined FBSs loca ion, we conside a
speci ic wi eless sys em model. This sys em model consid-
e s an in ini y backhaul capaci y on each FBS. Howe e ,
he o e ed bandwid h is limi ed in each FBS. The wi eless
sys em simula ion conside s ha only a speci ic numbe o
use s a e capable o ge ing he access s a um (AS)2in e -
ace. Thus, mos o he use s a e in he RRC_IDLE and
RRC_INACTIVE s a es, as de ined in he 5G new adio (NR)
s anda d. The emaining use s ha passed he andom access
p ocedu e a he medium access con ol (MAC) laye a e in
he RRC_CONNECTED s a e.
To acili a e he analysis, we conside ha each FBS is
capable o suppo ing a ixed numbe o use s Nin he
2AS is a unc ional in e ace ha is esponsible o anspo ing da a o e
he wi eless connec ion and managing adio esou ces.
VOLUME 10, 2022 53753
D. Ca illo Melga ejo e al.: Op imizing Flying Base S a ion Connec i i y by RAN Slicing and Rein o cemen Lea ning
FIGURE 3. Gene ic simula ion sys em model, conside ing he key
cons ain s o be sol ed by he deep ein o cemen lea ning app oach.
RRC_CONNECTED s a e. Based on his assump ion, he
cellula ne wo k is simula ed wi h LFBSs, and each FBS sup-
po s KRAN slices. As we conside only downlink ansmis-
sion, he in e e ence ha each use su e s om nonse ing
FBSs is calcula ed ollowing he assump ions de ined in he
3GPP TR 36.931 Ve . 16 speci ica ions. The heigh al i ude
o all FBSs is 20 m.
The DRL and he wi eless sys em model a e deployed
in Py hon. The machine lea ning implemen a ion is done
using Tenso Flow lib a ies o implemen he deep Q-ne wo k
se up. The deep neu al ne wo k used in he simula ion has
3 hidden laye s wi h 200, 100, 50 ully connec ed neu ons.
The ba ch size is 128. The epsilon-G eedy Algo i hm, used in
his wo k, conside s a maximum equal o 0.1 and an decay
o 0.9995. The implemen a ion and u he hype -pa ame e s
a e a ailable in he ollowing Gi hub URL h p://www.gi hub.
com/TBD.
Th ee di e en scena ios we e de ined, conside ing
aspec s such as a ixed numbe o FBSs (Scena io 1), and
co e age maximiza ion wi h a ixed numbe o use s in he
desi ed co e age a ea and in each cell (Scena ios 2 and 3,
espec i ely).
A. SCENARIO 1—FIXED NUMBER OF AVAILABLE FBSs
This scena io conside s L=20 FBSs co e ing a speci ic a ea
and suppo ing N=100 use s. To acili a e he analysis, i is
conside ed ha each FBS has N/L=5 a ached use s. This
scena io aims o iden i y an op imal cell adius suppo ing N
use s in he RRC_CONNECTED s a e. A g aphical desc ip-
ion o his scena io is shown in Fig. 4.
B. SCENARIO 2—MAXIMIZING THE NETWORK
COVERAGE I
In his case, he numbe o FBSs (L) is de ined by he op i-
miza ion ou pu s o S ages 1 and 2, which we e desc ibed
in Sec ion IV. Using Fig. 3as a e e ence, he alues o X
and Y a e 400 m. Thus, he numbe o FBSs is a unc ion
o cell adius; e.g., he la ge he cell adius is, he smalle
he numbe o FBSs needs o be deployed. In his scena io,
he numbe o use s Nin he RRC_CONNECTED s a e is
TABLE 2. Compa ison o pa ame e s used in Scena ios 2 and 3.
always he same in he a ea de ined by X and Y. This scena io
is illus a ed in Fig. 5.
C. SCENARIO 3—MAXIMIZING THE NETWORK
COVERAGE II
In his case, he numbe o UAVs (L) is de ined by he
op imiza ion ou pu s o S ages 1 and 2, which we e desc ibed
in Sec ion IV. Based on Fig. 3, he alue o X and Y is ixed
a 400 m. In his scena io, he numbe o Nuse s in he
RRC_CONNECTED s a e a ies when he numbe o FBSs
changes, as i is s a ed in Table 2. Each cell adius has a ixed
numbe o Nuse s in he scena io. Fig. 6depic s his scena io.
VI. ANALYSIS AND RESULTS
A. ANALYSIS OF SCENARIO 1
The spec al e iciency ob ained in Scena io 1 is p esen ed
in Fig. 7. He e, di e en cell adii be ween 50 m and 300 m
a e e alua ed. In he same igu e, he i s 500 i e a ions
o he i s episode3(be ween 1 and 10000 i e a ions) a e
compa ed wi h he i s 500 i e a ions o he second episode
(be ween 10001 and 20000 i e a ions). The policy, de ined
in Subsec ion III-D, is almos null in he i s i e a ions; he
spec al e iciency is e y low, simila o a andom alloca ion
o RAN slicing esou ces. A he beginning o episode 2, he
spec al e iciency imp o es quickly despi e ha a new use
deploymen is conside ed.
To acili a e he in e p e a ion o he simula ion esul s, he
a e age spec al e iciency in each scena io is calcula ed. Fo
ins ance, Fig. 8p esen s he esul s o Scena io 1 be o e and
a e applying he RAN slicing esou ce alloca ion. Be o e
applying he esou ce alloca ion, he maximum spec al e i-
ciency is almos 1.5 bps/Hz o he cell adius o 50 m.
The spec al e iciency dec eases o he o he cell adii
when he cell adius is inc eased. This ou pu ep esen s he
ne wo k pe o mance when only S ages 1 and 2, desc ibed
in Sec ion IV, a e conside ed. Howe e , he pe o mance is
imp o ed in all cell adii a e applying he DRL app oach.
In all cases, he spec al e iciency is imp o ed. Fo ins ance,
he cell adius o 75 m achie es a spec al e iciency o mo e
han 3.5 bps/Hz, ep esen ing a gain o 2 bps/Hz compa ed
wi h he se up wi hou op imiza ion.
3Wi h episode we e e o a comple e sequence o in e ac ions, om s a
o inish. In ou simula ions, one episode is comple ed a e 10 000 i e a ions.
53754 VOLUME 10, 2022