This is a sel -a chi ed e sion o an o iginal a icle. This e sion
may di e om he o iginal in pagina ion and ypog aphic de ails.
Au ho (s):
Ti le:
Yea :
Ve sion:
Copy igh :
Righ s:
Righ s u l:
Please ci e he o iginal e sion:
CC BY 4.0
h ps://c ea i ecommons.o g/licenses/by/4.0/
Adap ing o Dynamic LEO-B5G Sys ems : Me a-C i ic Lea ning Based E icien Resou ce
Scheduling
© Au ho s, 2022
Published e sion
Yuan, Yaxiong; Lei, Lei; Vu, Thang X.; Chang, Zheng; Cha zino as, Symeon; Sun,
Sumei
Yuan, Y., Lei, L., Vu, T. X., Chang, Z., Cha zino as, S., & Sun, S. (2022). Adap ing o Dynamic LEO-
B5G Sys ems : Me a-C i ic Lea ning Based E icien Resou ce Scheduling. IEEE T ansac ions on
Wi eless Communica ions, 21(11), 9582-9595. h ps://doi.o g/10.1109/TWC.2022.3178171
2022
Adap ing o Dynamic LEO-B5G Sys ems:
Me a-C i ic Lea ning Based E icien Resou ce
Scheduling
Yaxiong Yuan, S uden Membe , IEEE, Lei Lei, Membe , IEEE, Thang X. Vu, Membe , IEEE, Zheng Chang,
Senio Membe , IEEE, Symeon Cha zino as, Senio Membe , IEEE, and Sumei Sun, Fellow, IEEE
Abs ac —Low ea h o bi (LEO) sa elli e-assis ed communi-
ca ions ha e been conside ed as one o he key elemen s in
beyond 5G sys ems o p o ide wide co e age and cos -e icien
da a se ices. Such dynamic space- e es ial opologies impose
an exponen ial inc ease in he deg ees o eedom in ne wo k
managemen . In his pape , we add ess wo p ac ical issues o
an o e -loaded LEO- e es ial sys em. The i s challenge is how
o e icien ly schedule esou ces o se e a massi e numbe o
connec ed use s, such ha mo e da a and use s can be deli -
e ed/se ed. The second challenge is how o make he algo i hmic
solu ion mo e esilien in adap ing o dynamic wi eless en i on-
men s. We i s p opose an i e a i e subop imal algo i hm o
p o ide an o line benchma k. To adap o un o eseen a ia ions,
we p opose an enhanced me a-c i ic lea ning algo i hm (EMCL),
whe e a hyb id neu al ne wo k o pa ame e iza ion and he
Wolpe inge policy o ac ion mapping a e designed in EMCL.
The esul s demons a e EMCL’s e ec i eness and as - esponse
capabili ies in o e -loaded sys ems and in adap ing o dynamic
en i onmen s compa e o p e ious ac o -c i ic and me a-lea ning
me hods.
Index Te ms—LEO sa elli es, esou ce scheduling, ein o ce-
men lea ning, me a-c i ic lea ning, dynamic en i onmen .
I. INTRODUCTION
In beyond 5G ne wo ks (B5G), he massi e numbe o
connec ed use s and hei inc easing demands o high-da a-
a e se ices can lead o o e loading o e es ial base s a ions
(BSs), which in u n esul s in deg aded use expe ience,
e.g., longe delay in eques ing da a se ices o lowe da a
a e [1]. In o de o imp o e he ne wo k pe o mance and
use expe ience, he in eg a ion o sa elli es, e.g., low ea h
o bi (LEO) sa elli es, and e es ial sys ems is conside ed
as a p omising solu ion o p o ide cos -e icien da a se -
ices [2]. The solu ions o e es ial ne wo k op imiza ion
The wo k has been suppo ed by he ERC p ojec AGNOSTIC (742648),
by he FNR CORE p ojec s ROSETTA (C17/IS/11632107), FlexSAT
(C19/IS/13696663), Sma Space (C21/IS/16193290), and by he FNR bila e al
p ojec LARGOS (12173206). (Co esponding au ho : Lei Lei)
Yaxiong Yuan, Thang X. Vu, and Symeon Cha zino as a e wi h he
In e disciplina y Cen e o Secu i y, Reliabili y and T us , Luxembou g
Uni e si y, 1855 Ki chbe g, Luxembou g (e-mail: [email p o ec ed];
[email p o ec ed]; [email p o ec ed]).
Lei Lei is wi h he School o In o ma ion and Communica ions Enginee ing,
Xi’an Jiao ong Uni e si y, Xi’an 710049, China (e-mail: [email p o ec ed]).
Zheng Chang is wi h he School o Compu e Science and Enginee ing,
Uni e si y o Elec onic Science and Technology o China, Chengdu 610054,
China, and also wi h he Facul y o In o ma ion Technology, Uni e si y o
Jy ¨
askyl¨
a, FI-40014 Jy ¨
askyl¨
a, Finland (e-mail: [email p o ec ed]).
S. Sun is wi h he Ins i u e o In ocomm Resea ch, Agency o Sci-
ence, Technology, and Resea ch, Singapo e 138632 (e-mail: sunsm@i2 .a-
s a .edu.sg).
and esou ce managemen migh no be sui able o di ec
applica ion o in eg a ed sa elli e- e es ial sys ems [3]. In he
li e a u e, ailo ed schemes ha e been in es iga ed o imp o e
he ne wo ks’ pe o mance. In [4], he au ho s p oposed a
use scheduling scheme o maximize he sum- a e and he
numbe o accessed use s by u ilizing he LEO-based back-
haul. In [5], a join powe alloca ion and use scheduling
scheme was p oposed o maximize he ne wo k h oughpu in
hie a chical LEO sys ems wi h he cons ain o ansmission
delay. In [6], he au ho s de eloped a join esou ce block
alloca ion and powe alloca ion algo i hm o maximize he
o al ansmission a e o LEO sys ems. I is wo h no ing
ha he esou ce op imiza ion p oblems in LEO- e es ial
ne wo ks a e ypically combina o ial and non-con ex. The
con en ional i e a i e op imiza ion me hods, e.g., in [4]–[6],
a e una o dable o eal- ime ope a ions due o hei high
compu a ional complexi y.
A. Rela ed Wo ks: S a e-o - he-a and Limi a ions
Towa ds an e icien solu ion, a ious lea ning echniques
ha e been s udied. Compa ed o supe ised lea ning, ein-
o cemen lea ning (RL) lea ns he op imal policy om ob-
se ed samples wi hou p epa ing labeled da a. As one o he
p omising RL me hods, deep ein o cemen lea ning (DRL)
adop s deep neu al ne wo ks (DNNs) o pa ame e iza ion and
apid decision making. Recen wo ks ha e applied RL/DRL
o esou ce managemen in LEO- e es ial sys ems [7]–[9].
In [7], o maximize he achie able a e in LEO-assis ed elay
ne wo ks, a DQN-based algo i hm was p oposed o make
he online decisions o link associa ion. The au ho s in [8]
adop ed mul i-agen ein o cemen lea ning o minimize he
a e age numbe o hando e s and imp o e he e iciency
o channel u iliza ion o LEO sa elli e sys ems. In [9], he
au ho s applied an ac o -c i ic (AC) algo i hm o LEO esou ce
alloca ion, such as beam alloca ion and powe con ol. The
abo e RL algo i hms in p ac ical LEO sys ems a e limi ed
by he ollowing issue. Tha is, he pe o mance o a lea n-
ing model la gely depends on he da a o igina ed om he
expe ienced samples o he obse ed en i onmen , bu he
wi eless en i onmen is highly complex and dynamic. When
ne wo k pa ame e s a y d ama ically, he pe o mance o he
lea ning models can be deg aded. To emedy his, one has
o e-collec a la ge numbe o aining da a and e- ain he
lea ning models, which is ime-consuming and ine icien o
adap o as a ia ions [10].
1
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
To add ess his issue, a a ie y o s udies ocus on how o
make he lea ning models quickly espond o dynamic en i on-
men s. T ans e lea ning applies he knowledge acqui ed om
a sou ce lea ning ask o a a ge lea ning ask o speed up
he e- aining p ocess and educe he olume o he collec ed
new da a se s [11]. The pe o mance o ans e lea ning is
limi ed by inding co ela ed asks. Ano he app oach, join
lea ning, aims a ob aining a single model ha can be adap ed
o dynamic en i onmen s by op imizing he loss unc ion
o e mul iple asks [12]. Besides, con inual lea ning can also
accele a e he adap a ion o he new lea ning ask by adding he
expe ienced da a om he p e ious asks o he e- aining da a
se , hus a oiding comple ely o ge ing p e iously lea ned
models [13]. Join lea ning and con inual lea ning migh ha e
good lea ning pe o mance on a e age bu ha e limi ed gen-
e aliza ion abili ies when di e en asks a e highly di e si ied
[14]. In con as , me a-lea ning ex ac s me a-knowledge and
achie es good pe o mance o speci ic asks wi hou equi ing
he ela ed sou ce asks. The au ho s in [15] p oposed a
model-agnos ic me a-lea ning algo i hm (MAML) o ob ain
he model’s ini ial pa ame e s as me a-knowledge o quickly
adap o new asks. In [16], an algo i hm combining ac o -
c i ic wi h MAML (AC-MAML) was de eloped o lea n a
new ask om ewe expe ience da a se s. In [17], he au ho s
p oposed a p omising me a-c i ic lea ning amewo k wi h
be e pe o mance han con en ional AC and AC-MAML. In
[18], a me a-lea ning-based adap i e sensing algo i hm was
p oposed, which de e mines he nex mos in o ma i e sensing
loca ion in wi eless senso ne wo ks. In [19], me a-lea ning
was applied o ind a common ini ializa ion ec o ha enables
as aining o an au oencode o he ading channels. Mos
o he me a-lea ning me hods we e applied in he a eas o
pa e n ecogni ion [15], obo ics [16], [17], and physical laye
communica ions [19], which ypically add ess simple lea ning
asks wi h limi ed ac ion space. Howe e , when he lea ning
echniques, e.g., DRL, AC-MAML, o me a-c i ic lea ning,
a e applied o add ess combina o ial op imiza ion p oblems
in a dynamic LEO- e es ial ne wo k, he ac ion space can
be huge and he inpu -ou pu ela ionships can become mo e
complex. These may deg ade he e iciency o he abo e
lea ning me hods.
B. Mo i a ions and Con ibu ions
Mo ing beyond he s a e-o - he-a , his pape in ends o
add ess he ollowing ques ions:
•How o make he lea ning solu ions mo e adap i e o
dynamic LEO- e es ial ne wo ks?
•How o deal wi h he huge ac ion space and imp o e he
lea ning e iciency?
In his s udy, we design an enhanced me a-c i ic lea ning
algo i hm (EMCL) o enable e icien esou ce scheduling o
dynamic LEO- e es ial sys ems, and emphasize he solu ions
o deal wi h non-ideal dynamic en i onmen s. The majo
con ibu ions a e summa ized as ollows:
•We design a ailo ed me ic o o e -loaded LEO sys ems
wi h dense use dis ibu ion, aiming a se ing mo e use s
and deli e ing a highe olume o eques ed da a.
•We o mula e he esou ce scheduling p oblem as a
quad a ic in ege p og amming (QIP) and p o ide wo o -
line op imiza ion-based benchma ks, i.e., op imal b anch
and bound (B&B) algo i hm and subop imal al e na ing
di ec ion me hod o mul iplie s-based heu is ic algo i hm
(ADMM-HEU).
•Due o he combina o ial na u e and he high complexi y
o he o line solu ions, we sol e he p oblem om he
pe spec i e o DRL by e o mula ing a Ma ko decision
p ocess (MDP) o make online decisions wi h he iden i-
cal objec i e as he o iginal p oblem.
•To adap o dynamic en i onmen s, we p opose an EMCL
algo i hm based on a me a-c i ic amewo k. Compa ed
o con en ional me a lea ning, he no el y s ems om
ha : 1) The c i ic has good gene aliza ion abili ies o
e alua e any new ask such ha he lea ning agen can
adjus he policy imely when he en i onmen changes;
2) The ailo ed design o a hyb id neu al ne wo k ex ac s
he ea u es om he cu en and his o ical samples; 3) he
in eg a ed Wolpe inge policy allows he ac o o make
decisions mo e e icien ly in an exponen ially inc easing
ac ion space.
•We e alua e he p oposed EMCL wi h o he benchma ks
in h ee p ac ical dynamic scena ios, i.e., bu s y use
demands, d ama ically luc ua ed channel s a es, and use
depa u e/a i al. The nume ical esul s e i y EMCL’s
e ec i eness and as - esponse capabili ies in adap ing o
dynamic en i onmen s.
The es o he pape is o ganized as ollows. The sys em
model is p esen ed in Sec ion II. We o mula e a esou ce
scheduling p oblem and de elop op imal and subop imal so-
lu ions o pe o mance benchma ks in Sec ion III. In Sec ion
IV, we model he p oblem as an MDP and de elop an EMCL
algo i hm. Nume ical esul s a e demons a ed and analyzed
in Sec ion V. Finally, Sec ion VI concludes he pape .
II. SYSTEM MODEL AND PROBLEM FORMULATION
A. LEO-Te es ial Ne wo k
In p ac ice, e es ial BSs can become o e -loaded and
conges ed. This common issue has ecei ed conside able a en-
ion om academia, indus y, and s anda diza ion bodies, e.g.,
3GPP Release 17 [20]. In his wo k, we add ess his challeng-
ing issue ia de eloping sa elli e-aided solu ions. As shown in
Fig. 1, he BSs wi h limi ed esou ces migh no be able o
se e all he use s and deli e all he eques ed da a demands
wi hin a equi ed ansmission o queuing delay. To elie e he
bu den o he e es ial BSs, LEO sa elli es a e in oduced o
o load a ic om BSs o p o ide backhauling se ices. The
LEO employs a anspa en payload. Fo spec um usage, he
sys em keeps consis en wi h cu en ly deployed space and
g ound sys ems. Tha is, he LEO sa elli es ope a e a he Ka-
band o p o ide b oadband se ices o ad anced e minals,
e.g., equipped wi h e y small ape u e e minals (VSAT),
while he 5G e es ial sys em adop s sub-6GHz a he C-
band o se e no mal mobile de ices, e.g., sma phones [21].
We conside wo ypes o mobile e minals (MTs) in he
sys em. The i s ype is he no mal cellula e minals, e.g.,
2
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
Da a ansmission a (Ka band) Da a ansmission a (C band)
LEO BS Cellula e minals
Ga eway
Ga eway- LEO link Fibe link
Da a ansmission a +1 (C band)
Da a ansmission a +1 (Ka band)
TST
Co e Ne wo k
Dual-mode
e minals
Cen alized
Con olle
Fig. 1. An illus a i e LEO- e es ial communica ion sys em
cell phones, ha can be se ed by BSs o e es ial-sa elli e
e minals (TSTs), bu canno be se ed by LEO due o he
size limi a ion o dish an ennas. The o he is he dual-mode
e minals, e.g., ehicula e minals, which a e equipped wi h a
3GPP e es ial-non- e es ial ne wo k (TN-NTN) complian
dual-mode ha can be ei he se ed by LEO ia Ka-band (in
u al a eas) o by BS/TST h ough C-band (in u ban a eas)
[22]. Compa ed o con en ional cellula BS, TST is a small-
size e minal ha ac s as a lexible and cos -sa ing access
poin , e.g., S a link g ound e minals. A TST can ecei e
backhauling se ices om LEO o e Ka-band and ansmi
da a o MTs o e C-band [4]. The e es ial BSs can eques
da a om he co e ne wo k h ough op ical ibe links o om
he LEO sa elli es h ough he BS-LEO link. We ema k ha
Fig. 1 can be ex ended o a la ge-scale ne wo k wi h a massi e
numbe o MTs. Speci ically, an MT in Fig. 1 can ep esen a
clus e o densely-deployed de ices. Due o he p oximi y, he
channel s a es o he de ices wi hin a clus e can be assumed
iden ical. When a clus e is scheduled, all he de ices wi hin
he clus e will be scheduled by he TDMA (o FDMA) mode
o a oid in a-clus e in e e ence.
We deno e S,B,Mand Las he se o TSTs, BEs, MTs,
and LEOs, espec i ely, whe e Mis he union o se M1
(all he cellphone MTs) and M2(all he dual-mode MTs).
Thus, he union o ecei e s, i.e., g ound de ices (GDs), can
be exp essed as K=S ∪ B ∪ M ={1, ..., k, ..., K}, whe e
K=|S| +|B| +|M|. Simila ly, he union o ansmi e s is
w i en by N=S ∪ B ∪ L ={1, ..., n, ..., N}, whe e N=
|S|+|B|+|L|. The ime domain is di ided by ime slo s, i.e.,
T={1, ..., , ..., T}. In da a ansmission, each ansmi e
nse es a GD in unicas mode, i.e., no join ansmission
and no mul i-cas ansmission. Wi hin a ime slo , mul iple
ansmi e -GD links can be ac i a ed, o ming a link g oup.
We deno e G={1, ..., g, ..., G}as a se by enume a ing all
he alid link g oups.
To coo dina e he link scheduling be ween e es ial and
sa elli e pa s, a cen alized con olle is deployed in he sys em
[23]. Wi h he cen alized con olle , he in o ma ion om he
g ound and sa elli e can be collec ed and exchanged, which
acili a es he implemen a ion o scheduling decisions. In addi-
ion, e icien synch oniza ion app oaches can be implemen ed
on he ansmi e s and ecei e s o gua an ee ha he esou ce
scheduling upda es a e pe o med accu a ely in LEO sa elli e
sys ems [24].
B. Channel Modeling
We conside ime- a ying channels o bo h sa elli e and
e es ial communica ion. A ime slo , he channel s a e
be ween ecei e kand ansmi e ncan be modeled as:
hk,n, =(G(T)
leo ·G(C)
k,n, ·G(R), n ∈ L,
G(T)
e ·G(C)
k,n, ·G(R), n ∈ N L,(1)
whe e G(T)
leo and G(T)
e a e he ansmi an enna gain o LEO
and e es ial BS/TST, espec i ely. We assume ha all he
GDs a e equipped wi h a single ecei ing an enna, so ha hei
ecei e an enna gains G(R)a e uni o m. G(C)
k,n, ep esen s he
channel ading be ween ansmi e nand GD ka ime slo .
Fo LEO- o-GD channel, a widely used channel ading model
in [4], [6], [25] is adop ed, which includes ee-space pa h loss,
pi ch angle ading, a mosphe e ading, and Rician small-scale
ading:
G(C)
k,n, =c
4πdk,n, leo 2
·G(P)
k,n ·A(Ω) ·ϕ, (2)
whe e cis he speed o ligh , dk,n, is he p opaga ion dis ance
be ween LEO and he e minals, leo is he ca ie equency
o LEO, G(P)
k,n is he pi ch angle ading gain, and ϕis he
Rician ading gain. The a mosphe ic ading gain A(Ω) is he
unc ion o he angle Ω, whe e sin Ω = H/dk,n, , and His
he al i ude o LEO.
A(Ω) = 10(3χ
10 sin Ω ),(3)
whe e χ, in dB/km, is he a enua ion h ough he clouds
and ain. In downlink ansmission, we assume ha Dopple
shi caused by he high mobili y o LEO can be pe ec ly
p e(pos )-compensa ed in he ga eway based on he p edic able
sa elli e mo ion and speed [26]. Fo e es ial channels, i.e.,
TST/BS- o-MT, G(C)
k,n, consis s o he pa h loss and Rayleigh
small-scale ading [27], which is gi en by:
G(C)
k,n, =c
4πdk,n, e 2
·φ, (4)
whe e e is he ca ie equency o TST/BS and φis he
Rayleigh ading ac o .
Based on he adop ed channel ading models (2) and (4),
we u he model he ime- a ying channel as he ini e s a e
Ma ko channel (FSMC) o cap u e he ime-co ela ion cha -
ac e is ics and conduc ma hema ically ac able analysis. To
o m an FSMC, we i s disc e ize he channel s a e hk,n,
in o Lle els, i.e., H={h1, ..., hL}, whe e he h esholds
hl, ..., hLa e de e mined by he equal-p obabili y me hod [28].
Then he ansi ion p obabili y ma ix is de ined as:
P=


P1,1· · · P1,L
.
.
.....
.
.
PL,1· · · PL,L


,(5)
3
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
whe e he ansi ion p obabili y Pl,l0can be w i en as:
Pl,l0=P ob [hk,n, +1=hl0|hk,n, =hl], hl, hl0∈ H.(6)
Tha is, a a gi en ime slo , i hk,n, =hl,Pl,l0 e e s o
he p obabili y o channel s a e a he nex ime slo hk,n, +1
ansi ing om hl o hl0, which can be app oxima ed by he
a io be ween he le el c ossing a e and he a e age numbe
o symbol pe second [28].
C. Op imiza ion P oblem
We o mula e a esou ce scheduling p oblem o he consid-
e ed o e -loaded LEO-5G sys ems. We use bina y indica o s
αk,n,g o ep esen he ac i a ed links in g oup g∈ G, whe e
αk,n,g = 1 i he ansmi e -GD link (n, k)is included in
g oup gand will be ac i a ed when g oup gis scheduled, o h-
e wise, 0. Se Gand indica o s αk,n,g a e he necessa y inpu
pa ame e s o he op imiza ion p oblem P1. Following he
p inciples in (7)-(10), we enume a e alid links and candida e
g oups. In implemen a ion, e e y enume a ed link o g oup
will unde go a easibli y-check s ep o ensu e ha no links o
g oups iola e (7)-(10).
αk,n,g = 0,∀k∈ K M,∀n∈ N L,∀g∈ G,(7)
αk,n,g = 0,∀k∈ M1,∀n∈ L,∀g∈ G,(8)
Xn∈N αk,n,g ≤1,∀k∈ K,∀g∈ G,(9)
Xk∈K αk,n,g ≤1,∀n∈ N ,∀g∈ G.(10)
(7) and (8) exclude ce ain ypes o links, i.e., BS-BS,
TST-TST, BS-TST, TST-BS, and LEO-cellphone. (9) means
ha each GD kin g oup g ecei es da a om a mos one
ansmi e , and (10) ep esen s each ansmi e nin g oup g
se es no mo e han one GD. Fo example, conside a simple
sys em wi h 1 LEO, 1 TST, 1 BS, and 2 MTs (an MT1 in
M1, and an MT2 in M2). The e a e ou possible ecei e s,
i.e., TST, BS, MT1, and MT2, indexed by K={1,2,3,4},
espec i ely, and h ee possible ansmi e s, i.e., TST, BS,
and LEO, indexed by N={1,2,3}. Fil e ed by (7)-(8),
all he alid links a e (1,3) (TST o MT1), (1,4) (TST o
MT2), (2,3) (BS o MT1), (2,4) (BS o MT2), (3,1) (LEO
o TST), (3,2) (LEO o BS), and (3,4) (LEO o MT2).
Con ined by (9)-(10), a combina ion o he abo e links can
be a alid g oup g, e.g, a g oup {(3,4),(1,3)}con ains
wo links. Enume a ing all he alid g oups o ms se G=
{{(1,3),(2,4)},{(1,3),(3,4)}, ....., {(1,3),(2,4),(3,1)}},
which is se ed as he inpu se o decision making. No e
ha il e ed by cons ain s (7)-(10), a la ge numbe o
in alid links and g oups ha e been excluded. Fo e en la ge
ne wo ks, we ema k ha a ull enume a ion o g oups migh
be una o dable in implemen a ion. To deal wi h his issue,
some heu is ic enume a ion app oaches can be adop ed in
p e-p ocess s age o educe he complexi y o an a o dable
le el [29].
Con ined by (7) and (8), he SINR and he olume o
ansmi ed da a o GD kin g oup ga ime slo a e exp essed
in (11) and (12), espec i ely.
γk,g, =Pn∈L hk,n, αk,n,gpk,g
Pj∈K kPn∈L hj,n, αj,n,gpk,g +σ2
+Pn∈N L hk,n, αk,n,gpk,g
Pj∈K kPn∈N L hj,n, αj,n,gpk,g +σ2,(11)
and
Rk,g, = ΦBk,g log2(1 + γk,g, ),(12)
whe e pk,g is he ansmi powe o GD kin g oup gand Φis
he du a ion o each ime slo . We deno e Bleo and B e a e
he ixed bandwid h o LEO and BS/TST, espec i ely, such
ha he used bandwid h Bk,g o GD kin g oup gcan be
calcula ed by Bleo Pn∈L αk,n,g +B e Pn∈N L αk,n,g. We
de ine he decision a iables as x= [x1,1, ..., xg, , ..., xG,T ]
whe e
xg, =1,i g oup gis scheduled a ime slo ,
0,o he wise.
In a p ac ical o e -loaded scena io, no all he e minals can
be imely se ed and hei ac ual demands may no be ully
deli e ed in ime due o massi e access eques s compe ing
o limi ed esou ces. Unde his undesi able scena io, he
op imiza ion ask may shi om “se ing all he e minals
and sa is ying all he demands” o “se ing as many e minals
(and hei demands) as possible”. On his basis, we deno e
Dkand D0
k(< Dk)as he ac ual demand (in bi s) and he
h eshold, espec i ely. In he objec i e design, we conside
a composi e u ili y unc ion in (13), and de ine ha GD kis
se ed, i.e., k(x) = 1, when a h eshold D0
kis sa is ied.
k(x) =1
X
∈T X
g∈G
Rk,g, xg, −D0
k
,(13)
whe e 1(·)is an indica o unc ion such ha 1(β) =
1,i β > 0
0,i β≤0. We in oduce a h eshold D0
kin (13) since
in an o e -loaded scena io wi h densely deployed use s, he
sys em may no be able o sa is y all he ac ual demand Dk
wi hin one scheduling cycle. In implemen a ion, we p ede ine
D0
k=εDk, whe e 0≤ε≤1. The alue o εis selec ed om
he middle segmen o [0,1] o a oid oo high o low alue,
such ha D0
khas a conside able impac on he op imiza ion
esul s and he ade-o e ec .
We con e he non-linea unc ion k(x) o a linea unc ion
by in oducing auxilia y a iables y= [y1, ..., yk, ..., yK]and
linea cons ains (14d), whe e yk= k(x). The op imiza ion
p oblem is o mula ed as:
P1 : min
xg, ,yk
(x,y) = η0 X
k∈K
yk−K!2
+
X
k∈K
ηk
X
∈T X
g∈G
Rk,g, xg, −Dk

2
(14a)
s. . ¯γk−γk,g, ≤V 1−xg, X
n∈N
αk,n,g!,
4
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/

∀k∈ K, g ∈ G, ∈ T ,(14b)
X
g∈G
xg, ≤1,∀ ∈ T ,(14c)
D0
kyk≤X
∈T X
g∈G
Rk,g, xg, ,∀k∈ K,(14d)
xg, ∈ {0,1},∀g∈ G, ∈ T ,(14e)
yk∈ {0,1},∀k∈ K,(14 )
whe e ¯γkis he SINR h eshold o GD k,Vis a posi i e
su icien ly la ge alue, and η0, ..., ηKa e he weigh ac o s.
Conside ing he use s’ ai ness and esou ce u iliza ion in
an o e -loaded sys em, we design a ailo ed u ili y unc ion
(14a) consis ing o wo componen s. The i s e m encou ages
se ing mo e use s and mee ing hei minimum equi emen
D0
ksince sa is ying low- a ic use s a e mo e likely o ha e
ewa ds in he objec i e. The second e m aims a minimizing
he supply-demand gap such ha he schedule ends o
se e he use s wi h highe demand Dko highe weigh s
ηk(k= 1, ..., K). The p io i y o impo ance o he wo
pa s can be adjus ed by p e-de ined weigh alues acco ding
o di e en scena ios. Fo example, when a la ge numbe o
delay-sensi i e and low- a ic use s en e he ne wo k, he
schedule may gi e mo e p io i y by inc easing η0 o se e
his ype o use s as many as possible, while he delay- ole a e
se ices wi h high da a demand may ha e lowe p io i y (wi h
dec eased ηk) in his scheduling cycle.
•The cons ain s (14b) ep esen he SINR equi emen in
p ac ical sa elli e and 5G sys ems. I GD kin g oup gis
scheduled a ime slo , i.e., xg, Pn∈N αk,n,g = 1, he
SINR o GD kshould be highe han he h eshold ¯γk o
gua an ee he link quali y. This also implies ha schedul-
ing many links wi h s ong co-channel in e e ence may
no be a wise op ion in he op imal solu ion. The se ing
o ¯γk e e s o he s anda d o DVB-S2X [31] and 3GPP
Release 16 [32].
•The cons ain s (14c) ep esen no mo e han one g oup
can be scheduled in a ime slo .
•In cons ain s (14d), we de ine ha i GD kis se ed,
i.e., yk= 1, he ecei ed da a should be la ge han D0
k.
III. CHARACTERIZATION ON SOLUTION DEVELOPMENT
In his sec ion, we p opose an op imal me hod and a
heu is ic app oach as he o line benchma ks o small-medium
and la ge-scale ins ances, espec i ely. In addi ion, we ou line
con en ional online-lea ning solu ions and hei limi a ions.
A. The P oposed Op imal and Sub-op imal Solu ions
Towa ds he op imum o P1, we i s iden i y he con exi y
o P1 when he bina y a iables a e elaxed.
Lemma 1. The elaxa ion p oblem o P1 is con ex.
P oo . See Appendix A.
Based on Lemma 1, we conclude ha P1 is an in ege
con ex op imiza ion p oblem. The op imum can be ob ained
by B&B ha sol es a con ex elaxa ion p oblem a each
node, wi h he complexi y O(2G×T+K)[33]. Al hough he
complexi y inc eases exponen ially, he B&B-based app oach
can p o ide a pe o mance benchma k a leas o small-
medium ins ances.
To educe he complexi y in sol ing la ge-scale p oblems,
we de elop a subop imal algo i hm. We obse e ha P1 has
a a iable-spli ing s uc u e, which mo i a es he de elop-
men o ADMM based app oaches [34]. The algo i hm is
summa ized in Alg. 1, i s sol ing he con ex elaxa ion
p oblem o P1 based on ADMM (in lines 2-8), ollowed
by a ounding ope a ion (in lines 9-13). In ADMM, we
di ide he elaxed a iables in o T+ 1 blocks ˆ
x1, ..., ˆ
xT,ˆ
y,
whe e ˆ
x = [ˆx1, , ..., ˆxG, ], and in oduce auxilia y a iables
z= [z1, ..., zK], whe e
zk=D0
kˆyk−X
g∈G X
∈T
Rk,g, ˆxg, ,∀k∈ K.(15)
The inequali y cons ain s (14d) a e eplaced by:
zk≤0,∀k∈ K.(16)
The augmen ed Lag angian unc ion is exp essed as:
L(ˆ
x1, ..., ˆ
xT,ˆ
y,z,λ)
= (ˆ
x,ˆ
y) + X
k∈K
λk
zk−D0
kˆyk+X
g∈G X
∈T
Rk,g, ˆxg, 

+ρ
2X
k∈K
kzk−D0
kˆyk+X
g∈G X
∈T
Rk,g, ˆxg, k2,(17)
whe e ρ > 0is he penal y pa ame e and λ= [λ1, ..., λK]a e
he lag angian mul iplie s. We de ine Ii e as he o al numbe
o i e a ions o he algo i hm. In each i e a ion i, ADMM
upda es each a iable block as ollows (in line 5) and upda e
mul iplie s (in line 6):
ˆ
xi+1
= a gmin
ˆ
x ∈X
L(ˆ
xi
1, ..., ˆ
xi
T,ˆ
yi,zi,λi),∀ ∈ T ,(18)
ˆ
yi+1 = a gmin
ˆ
y∈Y
L(ˆ
xi
1, ..., ˆ
xi
T,ˆ
yi,zi,λi),(19)
zi+1 = a gmin
z∈Z
L(ˆ
xi
1, ..., ˆ
xi
T,ˆ
yi,zi,λi),(20)
whe e X ={x |(14b), (14c), (14d)},Y={y|0≤yk≤1}
and Z={z|zk≤0}. When ADMM e mina es, he con inu-
ous solu ion ˆxg, is ob ained in line 8. The ounding p ocess
is hen ca ied ou in lines 10-13 o con e he la ges ˆxg, in
each ime slo o 1 (selec ing he mos p omising g oup g o
each ) and keep o he s 0.
The de eloped ADMM-HEU can p o ide sub-op imal
benchma ks wi hin an accep able ime span, since he subp ob-
lems in (18)-(20) can be sol ed in a pa allel manne and wi h a
smalle size han he o iginal p oblem. Howe e , ADMM-HEU
equi es O(1/2)i e a ions o achie e -op imali y, whe e 
is se as µ
T(T+3) [35]. A each i e a ion, we can sol e he
T+ 2 a iable blocks by B&B wi h he ime complexi y o
O(T·2G+ 2 ·2K). Thus, he o al complexi y is gi en by
O(T5·2G+T4·2K), which migh no su icien o as
adap a ion o ne wo k a ia ions.
5
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
Algo i hm 1 ADMM-HEU
1: inpu : Dk,D0
kand Rk,n, .
2: Relax P1 o a con inuous p oblem P10.
3: Ini ialize ˆ
x0
,ˆ
y0,z0,λ0and i= 0.
4: o i= 0, ..., Ii e do
5: Upda e ˆ
x ,ˆ
yand zby Eq. (18), (19) and (20).
6: λi+1
k=λi
k+ρ zi
k−D0
kˆyi
k+P
g∈G P
∈T
Rk,g, ˆxi
g, !.
7: end o
8: Ob ain elaxed solu ion ˆxg, .
9: o ∈ T do
10: Find g†= a gmax
g∈G
{ˆx1, , ..., ˆxG, }.
11: Se x∗
g†, = 1 and x∗
g, = 0,∀g6=g†.
12: end o
13: Calcula e y∗
kbased on Eq. (13).
14: ou pu : x∗
g, and y∗
k
B. Con en ional Online-Lea ning Solu ions and Limi a ions
To enable an in elligen and online solu ion, we add ess he
p oblem om an RL pe spec i e. Fi s ly, we b ie ly in oduce
ac o -c i ic and me a-c i ic lea ning app oaches as a basis o
p esen he p oposed EMCL. AC is an RL algo i hm ha akes
ad an age o bo h alue-based me hods, e.g., Q-lea ning, and
policy-based me hods, e.g., REINFORCE, wi h as con e gen
p ope ies and he capabili y o deal wi h con inuous ac ion
spaces [36]. The lea ning agen in AC con ains wo com-
ponen s, whe e he ac o is esponsible o making decisions
while he c i ic is used o e alua ing he decisions by he alue
unc ions. Speci ically, a each lea ning s ep 1, he ac o akes
ac ion based on a s ochas ic policy, i.e., a ∼π(a|s ), whe e
π(a|s )is he p obabili y o aking an ac ion unde s a e s ,
ypically ollowing he Gaussian dis ibu ion [37]. The c i ic
is o gene a e a Q- alue unc ion Q(s , a ) = Eπ[¯ |s , a ],
whe e ¯ is he accumula ed ewa d a s ep , and Eπ[β]is
he expec ed alue o βo e he policy π. The goal o he
lea ning agen is o ind a policy o maximize he expec ed
accumula ed ewa d (o Q- alue).
A c i ical issue in con en ional lea ning app oaches, includ-
ing AC, is ha he pe o mance o a lea ning model la gely
depends on he adop ed aining o obse ed da a se s. To il-
lus a e he dynamic en i onmen and i s impac s, we conside
wo ypes o en i onmen al changes. The i s is “ o eseen
a ia ions”. A ypical example is a ime- a ying channel wi h
ce ain ime co ela ion and s a is ical cha ac e is ics. In his
case, a gene al machine lea ning algo i hm can cap u e he
egula pa e ns e ec i ely o esol e he mapping om he
en i onmen o he desi ed decision a iables. The second is
“un o eseen a ia ions”, which is much mo e challenging o
add ess. These changes a e usually unexpec ed and inclined o
b eak he s a is ical dis ibu ion o he o iginal en i onmen .
The p ac ical LEO-5G sys ems a e highly complex and dy-
namic, such as as and d ama ic a ia ions in channel s a es,
use demands, use a i al/depa u e, and ne wo k opologies.
This ypically causes he new inpu s o no longe be ele an
o he s a is ical p ope ies o he his o ical da a [38]. As a
consequence, he scheduling decisions made om he p e ious
1In his pape , a lea ning s ep co esponds o a ime slo .
lea ning model can become in alid and he model may need
o be e- ained o adap o he new en i onmen . To illus a e
his impac , we use Fig. 2, as an example, o depic a ypical
e olu ion o AC’s loss alue o e ime- a ying demands.
F om 0 o 100 ime slo s, he demand is ime- a ying bu
ollows his o ical s a is ical p ope ies, e.g., luc ua ing wi hin
a ce ain ange o ollowing a ce ain dis ibu ion, leading o
a well-adap ed AC wi h low and s able loss alues. When a
su ged demand is gene a ed a he 100- h ime slo , he new
inpu de ia es om he s a is ics. The AC model becomes
inapplicable o he new en i onmen , e idenced by he apidly
de e io a ing loss alues. When he agen in AC consumes a
conside able amoun o ime in new da a collec ion and e-
aining, he pe o mance can e u n o he p e ious le el.
0 50 100 150 200 250
Tim e
0.0
0.2
0.4
0.6
0.8
1.0
Loss Value
Lea ning pe o m ance w.o. e aining
Lea ning pe o m ance w. e aining
Fig. 2. E olu ion o loss o e ime- a ying demands.
To add ess his issue o “un o eseen change”, me a-c i ic-
based app oaches become an eme ging echnique ha akes
ad an age o a a ie y o p e iously obse ed asks o in e
he me a-knowledge, such ha a new lea ning ask can be
quickly ained wi h ew obse a ions [15]. Me a-c i ic lea n-
ing combines me a-lea ning wi h an AC amewo k o enhance
he gene aliza ion abili y. Howe e , con en ional me a-c i ic
lea ning is no e ec i e in dealing wi h he la ge disc e e space
in P1. In addi ion, he e is no uni o m s anda d o pa ame e -
ize he lea ning model and ex ac me a-knowledge in dynamic
en i onmen s. Thus, we p opose an EMCL algo i hm o enable
an e icien dynamic-adap i e solu ion.
IV. THE PROPOSED EMCL ALGORITHM
In his sec ion, we elabo a e he p oposed EMCL algo i hm,
i s ly s a ing om ou lining he EMCL amewo k, hen
de ailing he ailo ed design.
A. EMCL F amewo k
1) MDP Re o mula ion: Fi s , we e o mula e he o iginal
p oblem P1 as an MDP by de ining ac ion, s a e and ewa d.
•As he ac o is o selec a g oup om se Ga each ime
slo , he ac ion is de ined as an assigned link g oup,
a =g∈ G.(21)
•The s a e consis s o he channel coe icien s hk,n, ,
modeled as FSMC wi h he ansi ion p obabili y de ined
6
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
in (6), and he deli e ed da a o use kup o ime slo
, whe e bk, =bk, −1+Rk,a , .
s ={h1,1, , ..., hK,N, , b1, , ..., bK, }.(22)
All possible s a es a e included in he s a e space S. The
nex s a e only depends on he cu en s a e and ac ion bu
is i ele an o he pas , which means he s a e ansi ion
om s o ss+1 ollows he Ma ko p ope y [36].
•The ewa d is closely ela ed o he objec i e o P1. We
de ine he ewa d as (23).
=
K
X
k=0
ηk(∆2
k, −1−∆2
k, ),(23)
whe e ∆k, =


K
P
k=1
1(bk, −D0
k)−K, k = 0,
bk, −Dk, k 6= 0.
Then, he accumula ed ewa d a s ep is gi en by
¯ =PT
0= γ 0− 0, whe e γ∈[0,1] is a discoun ed
ac o .
Unde he designed MDP, we e i y he consis ency be ween
he goals o he RL algo i hm and he o iginal op imiza ion
p oblem such ha he policy p o ided by he lea ning agen
can minimize he objec i e in P1.
Lemma 2. When γ= 1, he objec i e o he lea ning agen
is equi alen o ha o he op imiza ion p oblem P1.
P oo . See Appendix B
2) Me a C i ic and Task-Speci ic Ac o : As shown in Fig.
3, we design a hie a chical s uc u e in EMCL con aining
a me a c i ic2and mul iple ac o s. Me a-lea ning uses da a
om p e iously obse ed mul iple asks, J(1), ..., J(I), o
in e a “me a-knowledge” wi h good gene aliza ion abili y and
accele a e he aining o a new ask. In he p oposed EMCL,
he “me a-knowledge” is he me a c i ic which can e alua e he
ask wi h a Q- alue, like he ole o he c i ic in adi ional
AC, and possesses a s ong gene aliza ion abili y o guide any
ask-speci ic ac o o p o ide a policy.
A ime s ep ,s(i)
,a(i)
, and (i)
ep esen he s a e,
ac ion, and ewa d o ask i, espec i ely. An episode
D(i)={s(i)
1, a(i)
1, (i)
1..., s(i)
T, a(i)
T, (i)
T}can be sampled om
he i s s ep o he e minal s ep T. We deno e D(i)
[u,w]
as a segmen o D(i) om s ep u o w, i.e., D(i)
[u,w]=
{s(i)
u, a(i)
u, (i)
u, ..., s(i)
w, a(i)
w, (i)
w}. Since he explici me a c i ic
and ac o s a e di icul o ob ain, we adop he unc-
ion app oxima ion me hod. The me a c i ic is pa ame e -
ized as a neu al ne wo k (NN) wi h he weigh s ω, i.e.,
Q(s(i)
, a(i)
,D(i)
[ −¯
, −1];ω). We no e ha , in addi ion o s(i)
and a(i)
, he inpu includes he mos ecen ¯
samples
D(i)
[ −¯
, −1]. Each ask-speci ic ac o is modeled as an NN
π(a|s(i)
;θ(i))wi h he weigh s θ(i).
To op imize he weigh s, we minimize he loss unc ions by
g adien descen . The loss unc ion o he me a c i ic L(ω)is
2In his pape , “me a-c i ic lea ning” e e s o an algo i hm ha combines
AC and me a-lea ning while “me a c i ic” e e s o he c i ic in he amewo k.
Task 1
Task I
...
s
(i)a
(i)
D[u,w]
(i)
LTSM
LTSM
LTSM
LTSM
LTSM
LTSM
...
...
s
(1)
...
Wolpe inge
mapping
s
(I)
Wolpe inge
mapping
a
(1)
a
(I)
me a c i ic
ac o 1
ac o I
Task-speci ic
Q- alue
Upda e me a c i ic
ω +1 = ω -ρ׏ωL(ω)
Upda e ac o θ +1
(1) = θ +1
(1)-ρ׏θJ(θ +1
(1))
Upda e ac o I θ +1
(I) = θ +1
(I)-ρ׏θJ(θ +1
(I))
π(a|s
(1); θ(1))
π(a|s
(I); θ(I))
Memo y
New Task
s
Wolpe inge
mapping
a
ac o
Upda e ac o θ +1 = θ +1-ρ׏θJ(θ +1)
π(a|s ; θ)
me a c i ic ω*
Memo y
ex ac well- ained me a c i ic
asĀme a-knowledgeā
Me a aining phase
Online lea ning phase
On
Q- alue
Task i
Gene al
Q- alue
ask iden i ica ion
embedding
Fig. 3. The p oposed EMCL amewo k.
de ined as he a e age empo al di e ence (TD) e o o e all
asks:
L(ω) =1
I
I
X
i=1
Eπ(θ(i))h(Q(s(i)
+1, a(i)
+1,D(i)
[ −¯
+1, ];ω)−
−γQ(s(i)
, a(i)
,D(i)
[ −¯
, −1];ω)i2,(24)
whe e he TD e o e lec s he simila i y be ween he es i-
ma ed Q- alue and ac ual Q- alue. Fo he ask-speci ic ac o ,
he loss unc ion J(θ(i))is he nega i e Q- alue:
J(θ(i)) = Eπ(θ(i))h−Q(s(i)
, a(i)
,D(i)
[ −¯
, −1];ω)i,(25)
such ha minimizing J(θ(i))is equi alen o maximizing he
expec ed accumula ed ewa d. The upda e ules a e gi en by:
ω +1 =ω −ρ∇ωL(ω),(26)
θ(i)
+1 =θ(i)
−ρ∇θ(i)J(θ(i)).(27)
Based on he undamen al esul s o he policy g adien
heo em [36], he g adien s o L(ω)and J(θ(i))a e:
∇ωL(ω) = 1
I
I
X
i=1 h2L(ω)∇ω(Q(s(i)
+1, a(i)
+1,D(i)
[ −¯
+1, ];ω)
−Q(s(i)
, a(i)
,D(i)
[ −¯
, −1];ω))i,(28)
∇θ(i)J(θ(i)) = −Q(s(i)
, a(i)
,D;ω)∇θ(i)log π(a|s(i)
;θ(i)).
(29)
3) Algo i hm Summa y: We summa ize he p oposed
EMCL in Alg. 2, which includes wo phases: he me a aining
7
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/
phase and he online lea ning phase. Fo he o me , he
me a c i ic is ained o e di e en lea ning asks. A each
lea ning episode, we sample Ilea ning asks. We ob ain he
app oxima ed Q- alue (in line 6) and s ochas ic policy (in
line 7) by he app oxima ion unc ions. The inal ac ions a e
de e mined by he Wolpe inge app oach in line 8, which
will be elabo a ed in he ollowing subsec ion. In line 9,
he memo y is used o s o e he expe ienced lea ning uples
{s(i)
, s(i)
+1, a(i)
, (i)
}. A each s ep, we ex ac a ba ch o
uples om he memo y as he aining da a o upda ing ω
and θ(i)by (26) and (27) in line 10 and 12, espec i ely. In
he online lea ning phase, gi en a new ask, he well- ained
me a c i ic ω∗can be di ec ly used o es ima e he Q- alue
and only he ac o needs o be e- ained. We no e ha he
adap a ion abili y o he me a-lea ning algo i hm depends on
he comple eness o he asks p o ided in he me a- aining
phase. In gene al, i is no p ac ical o collec all he possible
en i onmen s. As an al e na i e, he selec ed asks in he me a-
aining phase should keep he di e si y and ep esen a i eness
o achie e highe sampling e iciency.
Algo i hm 2 EMCL
Me a aining phase:
1: inpu : Mul iple ask samples; ini ial ω0.
2: o each lea ning episode do
3: Sample I asks and ini ialize ω0,θ(1)
0, ..., θ(I)
0.
4: o each lea ning s ep do
5: o each ask ido
6: Ob ain Q- alue by he me a c i ic in (32).
7: Ob ain s ochas ic policy by he ac o in (34).
8: Take ac ions a(i)
by he Wolpe inge app oach.
9: S o e uples {s(i)
, s(i)
+1, a(i)
, (i)
}in he memo y.
10: Take a ba ch o da a and upda e θ(i)by (27).
11: end o
12: Upda e ωby (26).
13: end o
14: end o
15: ou pu : The well- ained me a c i ic ω∗.
Online lea ning phase:
16: inpu : A new ask; ini ial θ0; well- ained me a c i ic ω∗.
17: o each lea ning episode do
18: o each lea ning s ep do
19: Ob ain Q- alue by he me a c i ic in (32).
20: Ob ain s ochas ic policy by he ac o in (34).
21: Take an ac ion a by he Wolpe inge app oach.
22: S o e uples {s , s +1, a , }in he memo y.
23: Take a ba ch o da a and upda e θby (27).
24: end o
25: end o
26: ou pu : The op imal ac o θ∗.
B. Tailo ed Designs in EMCL
1) Pa ame e iza ion wi h Hyb id Neu al Ne wo ks: The e
is no uni o m s anda d o pa ame e iza ion in con en ional
me a-c i ic lea ning. Conside ing dynamic en i onmen s, he
dis ibu ion o he new inpu da a and he p e ious obse a ions
may de ia e. Towa ds as adap a ion o he dynamic en i on-
men , he c i ic should be able o iden i y di e en asks, whe e
he in o ma ion o ask iden i ica ion can be e ined om he
expe ienced da a, which usually o ms ime- ela ed se ies [17].
The widely used DNN migh ha e limi a ions in e iciency and
in mining ea u es om ime-se ies da a due o he massi e
numbe o weigh s and eed- o wa d s uc u e. In he p oposed
EMCL, we design ailo ed neu al ne wo ks o enable he me a
c i ic and he ac o s o i he complex nonlinea ela ionships
and ex ac he me a-knowledge om his o ical da a.
As shown in Fig. 3, o he me a c i ic, a hyb id neu al ne -
wo k (HNN) combing con olu ional neu al ne wo k (CNN),
long-sho e m memo y (LSTM), and a i icial neu al ne wo k
(ANN) is applied o lea n he ea u es om he cu en s a e-
ac ion pai s and his o ical ajec o ies [39]. The ein o, CNN is
compu a ion-e icien ia adop ing he pa ame e sha ing and
pooling ope a ions, and is e ec i e o ex ac spa ial ea u es
om he inpu da a. These ad an ages enable CNN o educe
he pa ame e s o he model and alle ia e he p oblem o
o e i ing. LSTM, as a ype o ecu en neu al ne wo k, has
ad an ages in ex ac ing ea u es om ime- ela ed sequen ial
da a. Thus, in he designed me a c i ic, he CNN is used o
e alua e he decisions made by he ac o om he cu en
ac ion-s a e pai s(i)
, a(i)
. The LSTM is adop ed o iden i y
he ask based on he ime-se ies da a D(i)
[ −¯
, −1], such ha
he me a c i ic can accu a ely c i icize any ac o in changing
en i onmen and adap o he dynamic ne wo ks. We deno e
cnn(x;w), ls m(x;w)and ann(x;w)as he ou pu s o
CNN, LSTM, and ANN, espec i ely, which a e he unc ions
o inpu xand weigh w. The ea u es ou pu om CNN and
LSTM a e:
ξ1= cnn(s(i)
, a(i)
;ωcnn),(30)
ξ2= ls m(D(i)
[ −¯
, −1];ωls m),(31)
whe e ξ1and ξ2physically mean he gene al Q- alue and
he ask iden i ica ion embedding, espec i ely, which can be
ep esen ed by scala s [17]. Then, we ake he ea u es as
inpu s and pass hem h ough a ully-connec ed ANN o ob ain
he ask-speci ic Q- alue:
Qπ(s(i)
, a(i)
,D(i)
[ −¯
, −1];ω) = ann(ξ1, ξ2;ωann).(32)
Fo he ask-speci ic ac o s, we adop CNN as he app oxima-
o which akes he cu en s a e as he inpu and ou pu s he
mean µand a iance ϑ2o he s ochas ic policy. We assume
he s ochas ic policy ollows Gaussian dis ibu ion N(µ, ϑ2),
such ha
[µ, ϑ2] = cnn(s(i)
;θ(i)),(33)
π(a|s(i)
;θ(i)) = N(µ, ϑ2).(34)
2) Ac ion Mapping wi h he Wolpe inge Policy: The de-
cision a iables in P1 a e disc e e such ha we need o map
he ac ion om he s ochas ic policy o a disc e e ac ion space.
Howe e , he p e ious ac ion mapping policies in me a-c i ic
lea ning a e no e icien since he ac ion space is la ge o
P1. Thus, in EMCL, he Wolpe inge policy is adop ed o
as e con e gence [40].
Following he s ochas ic policy π, he ac o i s p oduces
an ac ion ˆawi h con inuous alue, i.e.,
π:S → ˆ
A, π(s) = ˆa, (35)
8
This a icle has been accep ed o publica ion in IEEE T ansac ions on Wi eless Communica ions. This is he au ho 's e sion which has no been ully edi ed and
con en may change p io o inal publica ion. Ci a ion in o ma ion: DOI 10.1109/TWC.2022.3178171
This wo k is licensed unde a C ea i e Commons A ibu ion 4.0 License. Fo mo e in o ma ion, see h ps://c ea i ecommons.o g/licenses/by/4.0/