Li, Haonan; Wu, Xu; Ribei o, Ma a; San os, B uno; Zheng, Pan
A icle
Deep ein o cemen lea ning app oach o eal- ime
ai po ga e assignmen
Ope a ions Resea ch Pe spec i es
P o ided in Coope a ion wi h:
Else ie
Sugges ed Ci a ion: Li, Haonan; Wu, Xu; Ribei o, Ma a; San os, B uno; Zheng, Pan (2025) : Deep
ein o cemen lea ning app oach o eal- ime ai po ga e assignmen , Ope a ions Resea ch
Pe spec i es, ISSN 2214-7160, Else ie , Ams e dam, Vol. 14, pp. 1-15,
h ps://doi.o g/10.1016/j.o p.2025.100338
This Ve sion is a ailable a :
h ps://hdl.handle.ne /10419/325815
S anda d-Nu zungsbedingungen:
Die Dokumen e au EconS o dü en zu eigenen wissenscha lichen
Zwecken und zum P i a geb auch gespeiche und kopie we den.
Sie dü en die Dokumen e nich ü ö en liche ode komme zielle
Zwecke e iel äl igen, ö en lich auss ellen, ö en lich zugänglich
machen, e eiben ode ande wei ig nu zen.
So e n die Ve asse die Dokumen e un e Open-Con en -Lizenzen
(insbesonde e CC-Lizenzen) zu Ve ügung ges ell haben soll en,
gel en abweichend on diesen Nu zungsbedingungen die in de do
genann en Lizenz gewäh en Nu zungs ech e.
Te ms o use:
Documen s in EconS o may be sa ed and copied o you pe sonal
and schola ly pu poses.
You a e no o copy documen s o public o comme cial pu poses, o
exhibi he documen s publicly, o make hem publicly a ailable on he
in e ne , o o dis ibu e o o he wise use he documen s in public.
I he documen s ha e been made a ailable unde an Open Con en
Licence (especially C ea i e Commons Licences), you may exe cise
u he usage igh s as speci ied in he indica ed licence.
h ps://c ea i ecommons.o g/licenses/by/4.0/
Con en s lis s a ailable a ScienceDi ec
Ope a ions Resea ch Pe spec i es
jou nal homepage: www.else ie .com/loca e/o p
Deep ein o cemen lea ning app oach o eal- ime ai po ga e assignmen
Haonan Lia,∗, Xu Wu b, Ma a Ribei o a, B uno San os a, Pan Zheng c
aDel Uni e si y o Technology, Kluy e weg 1, Del , 2629 HS, Zuid-holland, The Ne he lands
bBeijing Jiao ong Uni e si y, Beijing, China
cCi il A ia ion Managemen Ins i u e o China, Beijing, China
A R T I C L E I N F O
Keywo ds:
Ai po ga e assignmen
Deep ein o cemen lea ning
Asynch onous ad an age ac o -c i ic
Real- ime op imiza ion
A B S T R A C T
Assigning ai c a o ga es is one o he mos impo an daily decision p oblems ha ai po p o essionals ace.
The solu ion o his p oblem has aised a signi ican e o , wi h many esea che s ackling many di e en
a ian s o his p oblem. Howe e , mos exis ing s udies on ga e assignmen con ain only a s a ic pe spec i e
wi hou conside ing possible u u e dis up ions and unce ain ies. We b idge his gap by looking a ga e
assignmen s as a dynamic decision-making p ocess. This pape p esen s he Real- ime Ga e Assignmen P oblem
Solu ion (REGAPS) algo i hm, an inno a i e me hod adep a esol ing p e-assignmen issues and dynamically
op imizing ga e assignmen s in eal- ime a ai po s h ough he in eg a ion o Deep Rein o cemen Lea ning
(DRL). This wo k ep esen s he i s ime ha DRL is used wi h eal ai po da a and a con igu a ion con aining
a la ge numbe o ligh s and ga es. The me hodology combines a ailo ed Ma ko Decision P ocess (MDP)
o mula ion wi h he Asynch onous Ad an age Ac o –C i ic (A3C) a chi ec u e. Mul iple ac o s, such as ligh
schedules, ga e a ailabili y, and passenge walking ime, a e conside ed. An empi ical case s udy demons a es
ha he REGAPS ou pe o ms wo classic deep Q-lea ning algo i hms and a adi ional Gene ic Algo i hm in
e ms o educing passenge walking ime and ap on ga e assignmen . Finally, supplemen a y expe imen s
highligh REGAPS’s adap abili y unde a ious ga e assignmen ules o in e na ional and domes ic ligh s.
The inding demons a es ha no only did REGAPS ou pe o m COVID es ic ions, bu i can also p oduce
conside able bene i s unde o he policies.
1. In oduc ion
One o he mos signi ican cu en discussions in ai po ope a ion
discipline is ga e assignmen , he p ocess o alloca ing ligh s o ai po
ga es. Based on he s a is ical da a acqui ed by IATA [1], in 198 o
354 ai po s su eyed, hey lack he capaci y o ul ill he exis ing
demand, necessi a ing a g ea e deg ee o lexibili y in he assignmen
o ga es. Thus, i is i al o analyze he na u e o he ga e assignmen
p oblem (GAP) and design an e icien and e ec i e app oach o ease
he s ess o ai po coo dina ion. Many esea ch wo ks ha e ackled
ga e assignmen [2]. One key issue wi h mos o he pas s udies is
ha hey we e conduc ed om a s a ic pe spec i e, assuming a s a ic
en i onmen wi hou any unp edic able changes. As a esul , no las -
minu e dis u bances a e aken in o accoun . Howe e , exis ing esea ch
shows ha ligh delays nega i ely a ec he pe o mance o s a ic ga e
assignmen s [3]. Consequen ly, hese solu ions can only be used as
s a egic planning, om which ai po p o essionals mus de ia e in
eal- ime o accommoda e unplanned changes.
In p ac ice, ai po p o essionals assign ga es in wo s ages: i s
in p e-assignmen and hen in eal- ime. In he p e-assignmen phase,
∗Co esponding au ho .
E-mail add ess: [email p o ec ed] (H. Li).
ga es a e assigned in ad ance based on p e ious da a. As plane a el
is so eadily dis up ed, he p e-assignmen plan will almos ce ainly
di e ge om he inal plan. Ope a o s o en handle dis u bances man-
ually, depending on hei expe ience. Some esea che s obse ed he
de ia ion and a emp ed o sol e i [4–6]. These a emp s o mi iga e
he impac o dis up ion by s eng hening he assignmen plan’s obus -
ness o esea ching solu ions o he eassignmen challenge. Howe e ,
cu en sys ems a e incapable o apidly adjus ing he assignmen plan
in esponse o changes in he ligh schedule, and no p e ious wo k has
in es iga ed au oma ic adjus men o he assignmen schedule om a
eal- ime pe spec i e.
This s udy aims o b idge exis ing esea ch gaps by depa ing om a
s a ic iewpoin in ga e assignmen s, by analyzing he ga e assignmen
p ocedu e as a dynamic, eal- ime decision-making p ocess. Thus, ou
main esea ch objec i e can be desc ibed as: To de elop a uni ied decision
suppo sys em o bo h s a ic and dynamic scena ios. In o he wo ds, ou
model can be used a he wo s ages, in p e-assignmen and hen in
eal- ime a e dis up ions occu . Ou solu ion s ands-ou o being a
single model ha can be un a any ime, wi h upda ed in o ma ion,
h ps://doi.o g/10.1016/j.o p.2025.100338
Recei ed 8 Oc obe 2024; Recei ed in e ised o m 23 Decembe 2024; Accep ed 2 Ap il 2025
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
A ailable online 11 Ap il 2025
2214-7160/© 2025 The Au ho s. Published by Else ie L d. This is an open access a icle unde he CC BY license ( h p://c ea i ecommons.o g/licenses/by/4.0/ ).
H. Li e al.
and by being able o p o ide a solu ion in a ma e o milliseconds.
Ou objec i e is o explo e he e icacy o his p omising app oach
in add essing he s ochas ic and dynamic na u e o ga e alloca ion
challenges. We p opose ha by le e aging he sequen ial decision-
making o Deep Rein o cemen Lea ning (DRL), ou amewo k may
e ec i ely manage u gen and equen changes. We an icipa e ha his
app oach will (1) enhance cu en ga e alloca ion me hods by educing
u u e ins ances o undesi able emo e ga e usage, (2) s ike a balance
be ween compe ing objec i es such as minimizing passenge walking
ime and p io i izing ligh schedules, and (3) exhibi excep ional lexi-
bili y, enabling swi execu ion unde a ying ligh con igu a ions wi h
ema kably as un imes.
Ou app oach o he Ga e Assignmen P oblem (GAP) in ol es wo
s ages, each le e aging Ma ko Decision P ocesses (MDP) and DRL
me hodologies. MDP p o ides a da a-based amewo k o au oma -
ically sol ing sequen ial decision-making p oblems. Wi h MDP, he
en i onmen is modeled as a se o s a es and ac ions ha can be
pe o med o con ol he sys em’s s a e. The goal is o con ol he
sys em in such a way ha some pe o mance c i e ia is maximized.
This modeling cons i u es he base o he DRL model. DRL has he
ad an age o no ha ing o ely on a comple e and accu a e model o
he en i onmen . The e o e, i can maximize ewa ds in en i onmen s
whe e he exac dynamics o he en i onmen a e no known. We
employ he Asynch onous Ad an age Ac o -C i ic (A3C) a chi ec u e
o add ess he GAP due o i s unique p ope ies — using mul iple
independen in e ac ing agen s allows o g ea e explo a ion in less
ime. Ini ially, we add ess he GAP om a s a ic pe spec i e, e med
p e-assignmen , whe e ga e assignmen s a e made be o e ope a ional
commencemen . Subsequen ly, we ansi ion o eal- ime assignmen ,
whe e ga e assignmen s a e dynamically adjus ed in esponse o e ol -
ing ope a ional condi ions, including eal- ime ligh schedule changes.
Wi h each app oaching ligh , a decision is made, and we moni o
he esul s be o e adjus ing ou decisions. Le e aging he inhe en
adap abili y o MDP and RL, ou amewo k seamlessly inco po a es
eal- ime adjus men s, ensu ing e icien and esponsi e ga e alloca ion
wi hin ai po ope a ions.
This pape is o ganized as ollows: Sec ion 2 desc ibes pas GAP
esea ch as well as con empo a y DRL applica ions in he anspo a ion
sec o . In Sec ion 3, an MDP o mula ion is p oposed, which is igh ly
ailo ed o he GAP. Sec ion 4 inco po a es he de eloped MDP in o
he A3C a chi ec u e and builds he eal- ime GAP solu ion (REGAPS).
The hypo heses o he implemen ed me hod a e p esen ed in Sec ion 5.
Sec ion 6 conduc s an empi ical case s udy based on eal ai po da a
and e alua es and alida es he model’s pe o mance. No e ha he
speci ic ai po is no disclosed due o p i acy issues. Sec ion 7 dis-
cusses whe he he ini ial hypo heses we e e i ied. Finally, Sec ion 8
concludes his pape .
2. Rela ed wo k
This sec ion co e s he s a e-o - he-a on esea ch on ga e as-
signmen . P e ious wo ks on his opic a e desc ibed in Sec ion 2.1.
Addi ionally, gi en ha DRL is p ominen in his wo k, Sec ion 2.2
desc ibes he la es ad ances in his a ea ou side o GAP. Finally,
Sec ion 2.3 will clea ly de ine he gaps in li e a u e o be co e ed by
his wo k.
2.1. Ga e Assignmen P oblem (GAP)
Resea ch in o GAP has a long his o y in he ope a ions esea ch
discipline due o i s p ac ical impo ance. The e a e mul iple ypes o
app oaches wi h espec o he solu ion me hodologies. These can be
ca ego ized in o ou pa s:
•Expe sys ems design so wa e o simula e he decision-making
p ocess o human expe s. Some s udies ha all in o his ca ego y
a e B azile & Swigge [7], Gosling [8], S iha i & Mu huk ish-
nan [9] and Su & S iha i [10].
•Exac me hods a e one o he mos ocused me hodologies o GAP
mainly modeled as In ege p og amming o Linea P og amming.
Fo ins ance, Ba bic [11] published one o he ea lies s udies
using a b anch and bound algo i hm wi h a lowe bound ha
minimizes he o al walking dis ance o he passenge s. Mangoubi
& Ma haisel [12] p oposed a g eedy heu is ic app oach and a
linea p og amming elaxa ion me hod o minimize he o al
walking dis ance. Bola [13] ocused his objec i e on minimizing
he di e ence be ween he minimum and maximum slack imes,
using a b anch-and-bound algo i hm and a heu is ic app oach
called b anch-and- im. The p ima y limi a ion o exac me hods
is he ex ensi e compu a ion ime. In p ac ical GAPs, he objec i e
is no o ind a globally op imal solu ion bu a he o ob ain a
sa is ac o y solu ion wi hin a easonable ime ame.
•S ochas ic P ocesses and Heu is ic: his is he example o he wo k
om Haghani and Chen [14] ha p oposed a heu is ic algo i hm
o minimize passenge walking dis ance by assigning successi e
ligh s o he same slo . Ding e al. [15] de eloped a g eedy
algo i hm o acqui e an ini ial schedule o he hyb id algo i hms
based on Tabu sea ch and Simula ed Annealing la e on. Recen ly,
Jie Li e al. [16] p esen ed a column gene a ion-based algo-
i hm o sol e ga e assignmen wi h combina ional ga es. Jiang
e al. [17] in oduce a no el app oach o op imizing ai po ga e
assignmen s, add essing ha bo ap on sa e y cons ain s h ough
a wo-phase ma hema ical model. Re inemen s o he b anch-
and-p ice me hod imp o e e iciency and accu acy in sol ing
he p oblem. She e al. [18] add esses he Ai po Ga e Assign-
men P oblem (AGAP) by p oposing a mul i-objec i e in ege
p og amming model and a wo-phase Mon e Ca lo-based NSGA-
II algo i hm. Compu a ional analyses alida e he e icacy o he
app oach in p o iding economical, obus ga e assignmen s. The
d awback o s ochas ic p ocesses lies in hei inhe en high com-
pu a ional complexi y, coupled wi h he challenge o gene a ing
ealis ic scena ios.
•Rein o cemen lea ning (RL) app oaches: hese o e an ad an age
o e s ochas ic p ocesses by no needing o speci y a model.
RL, he subjec o his pape , is model- ee, p o iding a da a-
d i en, lea ning-based amewo k o o mula e and sol e sequen-
ial decision-making p oblems. RL has been widely used in a ia-
ion, wi h applica ions anging om ai line e enue managemen
o ai c a al i ude con ol [19]. Many a ia ion- ela ed scena ios
can be concep ualized as sequen ial decision p oblems, making RL
an ideal ool. In a ious domains, RL has demons a ed p omising
ou comes. Ne e heless, o he bes o he au ho s’ knowledge,
he wo k Yildizi e al. [20] s ands as he pionee ing and sole
a emp o apply RL o ga e assignmen . Thei p elimina y s udy
unco e ed a ime complexi y issue when dealing wi h mo e han
12 ga es. None heless, a a ie y o RL algo i hms and echniques
exis , and de e mining he mos sui able one o each p oblem
equi es ex ensi e examina ion. I is e iden ha u he esea ch
is wa an ed o explo e he po en ial o RL o GAP. To his end,
we p opose a mo e complex RL algo i hm capable o handling a
la ge numbe o ga es.
Despi e all he e o ha has been done o assis on-g ound ope -
a ions, ew o hem a e ac ually implemen ed in ai po s. One o he
majo issues is ha hese app oaches a e mainly designed om a s a ic
pe spec i e. These sol e he P e-assignmen p oblem, which is consid-
e ably di e en om he inal ga e assignmen once pe u ba ions in
eal- ime ope a ions occu . Al hough some solu ions a e o mula ed as
s ochas ic and obus app oaches, and y o sol e GAP om a dynamic
pe spec i e, wi h he limi a ion o heu is ic algo i hms, hese can only
minimize idle ime and ga e con lic s, bu do no ully adjus o u u e
unce ain ies and pe u ba ions.
Fu he mo e, he scale o he p oblem wa an s he u iliza ion o
me hods ha a e no elian on speci ic models and can e ec i ely
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
2
H. Li e al.
inco po a e he empo al dimension o ac ion sequences. Fi s , he ac
ha RL lea ns by di ec aining in he en i onmen ende s i applica-
ble o any ai po con igu a ion. No e ha he decision space o his
p oblem can a y la gely, om he numbe o ligh s o he mul i ude o
ga es, depending on he unique cha ac e is ics o each ai po . Second,
ga e assignmen ep esen s a sequen ial dilemma whe ein he alloca ion
o speci ic ga es a p esen in luences he u u e a ailabili y o ga es o
subsequen ai c a . RL has he abili y o conside u u e s a es when
making decisions. Las ly, o la ge ai po s, eal- ime e alua ion using
con en ional me hods is no a solu ion, gi en hei la ge execu ion
imes.
2.2. Deep Rein o cemen Lea ning (DRL)
To ind he nex -gene a ion solu ion o he GAP, we eso o DRL.
The ounda ion o RL is he MDP, which is molded wi h he ga e
assignmen p ocess. This sec ion explo es DRL wo ks employed in o he
a eas. RL aims o achie e a maximum ewa d h ough ac ions made
by he agen and he in e ac ion ewa d wi h he en i onmen . The RL
amewo k has become p omising due o i s abili y o lea n he dynam-
ics o he en i onmen h ough di ec in e ac ion wi h he en i onmen .
Wi h imp o ed da a a ailabili y and compu a ional powe , RL s udies
in a ia ion now ange om ai line e enue managemen o ai a ic
con ol (ATC) [19].
Recen ly, DRL has eme ged. This coalesces deep lea ning (DL) and
RL oge he and exploi s bo h o hei ad an ages he mos . DL is widely
used o unc ion app oxima ion and alue p edic ion. The e ha e been
many p eceden s ega ding DRL implemen a ion in he anspo a ion
discipline. Lin [21] p oposed a DRL app oach owa d he Elec ic Vehi-
cle Rou ing P oblem wi h Time windows. Yu [22] o e ed a solu ion o
online ehicle ou ing wi h neu al combina o ial op imiza ion and DRL.
The e a e also applica ions on sma anspo a ion sys ems conce ning
he ebalancing p oblem o he Bike-sha ing sys em [21]. Li e al. [23]
p esen a no el heu is ic algo i hm o he Ga e Assignmen P oblem
(GAP) in ai po managemen . Combining abu sea ch wi h ein o ce-
men lea ning, i e icien ly explo es solu ions, ou pe o ming exis ing
me hods in solu ion quali y and compu a ion ime ac oss eal-wo ld
benchma ks. Sui e al. [24] in oduce a ac ical con lic esolu ion
s a egy using Deep Rein o cemen Lea ning o mi iga e he inc easing
isk o ligh con lic s in ai space. Con olle s’ ac ions a e modeled as
a Ma ko Decision P ocess, ained by he Deep Q Ne wo k algo i hm.
Simula ion expe imen s con i m he s a egy’s easibili y and alignmen
wi h eal-wo ld ligh sa e y egula ions. Zhang e al. [25] add ess
he Mul i-T ip Vehicle Rou ing P oblem wi h Time Windows using a
no el Coo dina ed Mul i-agen Hie a chical Deep Rein o cemen Lea n-
ing app oach. By s uc u ing a h ee-laye ed amewo k, his me hod
enhances solu ion quali y and con e gence a es, ou pe o ming a-
di ional heu is ic algo i hms and ein o cemen lea ning echniques.
Resul s indica e signi ican imp o emen s in cos e ec i eness and
ope a ional obus ness, demons a ing he e ec i eness o he p oposed
app oach in anspo a ion scheduling.
DRL has also been employed o add ess dynamic esou ce alloca ion
p oblems. In his con ex , ga e alloca ion can also be concep ualized as
a ype o esou ce, sugges ing ha insigh s om his esea ch domain
may be ele an . Jia Wang e al. [26] p opose an inc emen al ein-
o cemen lea ning amewo k o dynamic esou ce alloca ion, whe e
ask pa e ns a e ex ac ed om la ge-scale da a. They cons uc an
en i onmen model ha enables a lea ning agen o in e he logic o
ask se ice ope a ions and calcula e eedback sco es o each alloca ion
decision. Applica ions o dynamic esou ce alloca ion also ex ends o
he anspo a ion sec o . Fo ins ance, Ying He e al. [27] p opose
a gene al amewo k ha enables as -adap i e esou ce alloca ion in
dynamic ehicula en i onmen s by in eg a ing hie a chical ein o ce-
men lea ning wi h me a-lea ning. This app oach allows he amewo k
o quickly adap o new en i onmen s by ine- uning only he op-le el
mas e ne wo k, while he low-le el sub-ne wo ks con inue o make
op imal esou ce alloca ion decisions. In ano he example, Hongbin
Liang e al. [28] model he esou ce alloca ion p oblem in he In e ne
o Vehicles as a semi-Ma ko decision p ocess, inco po a ing a esou ce
ese a ion s a egy and a seconda y esou ce alloca ion mechanism. A
RL algo i hm is applied o sol e he model.
2.2.1. DRL algo i hms
DRL is also a gene alized e m o a la ge class o algo i hms. One
b anch o DRL is called alue-based app oaches. The algo i hm’s co e
is he alue unc ion ha di ec ly indica es he alue o ewa d o
an ac ion. Deep Q-lea ning Ne wo k (DQN) [29] is a commonly used
DRL model. Howe e , i ends o o e es ima e he Q- alue [30]. Many
a emp s ha e been ca ied ou o sol e his de iciency, and many
a ian s eme ged, such as Double DQN (DDQN) [31], and Dueling
DQN [32]. A second b anch is Policy-based app oaches. The di e ence
om alue-based app oaches is ha a he han di ec ly adjus ing he
ac ion based on he ewa d, i adjus s he p obabili y o choosing a
ce ain ac ion. A commonly used policy-based algo i hm is he REIN-
FORCE algo i hm [33]. The e a e also many o he esea che s who
ha e adop ed DRL in he anspo a ion ield [34–38].
In ecen yea s, ac o -c i ic algo i hms [39] ha e been p oposed o
ake ad an age o he bes p ope ies o alue-based and policy-based
algo i hms. These a e di ided in o wo pa s: (1) gene a ing an ac ion
based on a s a e, and (2) compu ing he Q- alue o he ac ion. The
ac o akes an ac ion based on he gi en s a e, while he c i ic e alua es
he ac ion wi h he alue-based unc ion. Then he ac o adjus s he
p obabili y o he selec ed ac ion based on he c i ic’s e alua ion.
E en hough hese me hods sound p omising, implemen ing hem has
many di icul ies. Ac o -c i ic equi es A i icial Neu al Ne wo k (ANN)
aining bo h in he Ac o and C i ic models, which no doub imposed
g ea di icul y in ne wo k aining. Also, he Q- alue gene a ed by he
c i ic model is unable o ell he ac o how much be e o wo se is he
chosen ac ion. The Asynch onous Ad an age Ac o -C i ic (A3C) [39],
used in his wo k, is one o he solu ions o conque hese d awbacks.
This model asynch onously pa allel ains di e en agen s on mul iple
independen en i onmen s. Each agen will upda e he global ne wo k
asynch onously. This s a egy no only exploi s mos o he compu ing
capabili y o a wo ks a ion bu also g ea ly accele a es he aining
p ocess [40–42]. No e ha he A3C algo i hm was picked o e an
Ad an age Ac o –C i ic (A2C) app oach due o i s asynch onous aspec ,
which has shown as e lea ning imes [43] on a mul i-co e CPU
compu e as used by he au ho s.
2.3. Li e a u e gaps
The e iew o he exis ing li e a u e unde sco es a c i ical gap in
add essing he dynamic na u e o he GAP, which has p edominan ly
been app oached om a s a ic pe spec i e, ocusing on P e-assignmen .
Despi e spo adic e o s o inco po a e s ochas ic and obus me hods,
exis ing solu ions emain hinde ed by heu is ic algo i hms and s uggle
o adap o eal- ime pe u ba ions and unce ain ies. In esponse o
his challenge, ou s udy p oposes a no el app oach ha in eg a es
DRL echniques wi h he MDP amewo k. Speci ically, we employ he
A3C algo i hm, enowned o i s pa allel aining capabili y, enabling
e icien decision-making in ime-sensi i e scena ios.
Wha se s ou app oach apa is i s unique capabili y o seamlessly
add ess bo h p e-assignmen and eal- ime assignmen challenges wi h
minimal modi ica ion. By le e aging he syne gis ic capabili ies o DRL
and MDP, ou amewo k o e s enhanced lexibili y and adap abili y
in op imizing ga e assignmen s ac oss a ious ope a ional scena ios.
Compa ing wi h o he app oaches, DRL has he ollowing ad an ages:
•The p ocess o cus omizing he DRL en i onmen and ewa d
unc ion is s aigh o wa d. Ope a o s can easily design o mod-
i y ga e- ela ed in o ma ion, such as he numbe o ga es, hei
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
3
H. Li e al.
dis ance om secu i y checkpoin s, and he speci ic ligh s as-
signed o each ga e, based on he unique equi emen s o he ai -
po . This lexibili y enables ai po s o ailo he sys em o hei
ope a ional needs wi hou equi ing ad anced echnical expe ise.
•DRL allows us o an icipa e u u e changes and dis up ions while
gene a ing as and e ec i e solu ions o suppo ope a o s in
making decisions unde igh ime cons ain s. Ope a o s can
easily inco po a e changes and dis u bances by upda ing he s a e
space, enabling he sys em o adap swi ly o e ol ing condi-
ions and ensu ing ha decision-making emains esponsi e and
e icien .
•DRL is capable o handling bo h s a ic and dynamic scena ios
simul aneously o swi ching be ween hem as needed by ope a-
o s, p o iding he lexibili y o accommoda e a ying ope a ional
demands. Depending on he ope a o ’s equi emen s, DRL can
gene a e ei he p e-assignmen esul s o eal- ime assignmen
esul s, allowing he sys em o adap o di e en scheduling and
decision-making con ex s e icien ly.
3. Model o mula ion
The GAP poses a signi ican ope a ional challenge in ai po s, e-
ol ing a ound he alloca ion o a i al and depa u e ligh s o sui -
able ga es. The p ima y goal is o op imize ga e assignmen s, aking
in o accoun key ac o s like minimizing passenge walking dis ances,
maximizing ga e u iliza ion, and mi iga ing ope a ional delays. In ou
esea ch, we inco po a e eal- ime ga e assignmen using ein o cemen
lea ning echniques o ully ha ness i s po en ial ad an ages. No e ha
he eal- ime assignmen is applied a e he p e-assignmen , conside -
ing he de ia ions be ween he ac ual and o iginal ligh schedules as
he dis u bance.
This s udy models a ga e assignmen p oblem as a Ma ko decision
p ocess (MDP). We conside ha an ai po has a se ies o nodes 𝐼=
{1,2,…}. Each node is ega ded as a ga e a he ai po o ai c a o
pa k empo a ily and o passenge aligh ing and boa ding. The se o
ga es 𝐼 is classi ied in o 𝐼𝑏 and 𝐼𝑎, whe e 𝐼𝑏 is he subse o nodes in
𝐼 ha a e B idge Ga es and 𝐼𝑎 a e he Ap on ga es (Remo e ga es). A
b idge ga e is a ga e ha is di ec ly connec ed o he e minal wi h a
b idge, while a emo e ga e needs a shu le bus o ans e passenge s
be ween he e minal and he ai c a . We de ine 𝐹= {1,2,…} as he
se o ligh s. The op imiza ion p oblem he e is o ensu e ha e e y
ligh in 𝐹 has a ga e in 𝐼 while minimizing he o al cos .
Sec ion 3.1 p esen s he key assump ions, ollowed by demons a -
ing he s a e ansi ion p ocess in he MDP (Sec ion 3.2) and he
objec i e unc ions (Sec ion 3.4). Finally, Sec ion 3.5 o mula es he
cons ain s imposed on he decision a iables.
3.1. Key assump ions
The ollowing assump ions a e conside ed in his wo k:
Assump ion 1: Simila ype o ligh s occupy he ga e wi h a simila
amoun o ime, and he ai c a will be owed away o eady o
depa u e. Ai c a will no occupy he ga e jus o wai ing pu poses.
The occupa ion ime o he ga e consis s o se e al pa s, as shown in
Fig. 1:
(a) Taxiing/ owing ime.
(b) G ound se ice.
(c) Passenge aligh ing and boa ding.
(d) Idle ime.
Assump ion 2: We assume ha he capaci y o he ai po ap on is
in ini e. When he e is no acan b idge ga e, he ai c a will be
assigned o he Ap on.
Assump ion 3: Conside ing ha he ga e assignmen p ocess occu s
subsequen ly o ob aining he ligh schedule, we assume ha all ligh s
can success ully land wi hou encoun e ing any addi ional in as uc-
u e complica ions. Limi ed unway capaci y, o p io i ies be ween
ai lines, o example, a e no conside ed.
3.2. S a e o mula ion
The RL algo i hm has a disc e e s a e space. Gi en he in as uc u e
and se ings, he GAP is ega ded as an MDP in which each s age
𝑛 e e s o he ligh s in 𝐹. Each s age 𝑛 is cha ac e ized by he
co esponding uple o s a e a iables 𝑠𝑛, and in luenced by he decision
a iables 𝑥𝑛. He ein, we de ine he s a es uple:
𝑠𝑛= (𝑓𝑛, 𝜏𝑛).(1)
Each s a e 𝑛 con ains he a i al imes 𝑓𝑛, which is de e mined in
p e-assignmen scena io and dynamically upda ed in eal- ime assign-
men scena io and he ga e size o he equi ed ligh (medium o la ge),
𝜏𝑛. The ga e sizes conside ed a e as ollows:
•𝜏𝑛= 2: medium size, can accommoda e small o medium ai c a .
•𝜏𝑛= 3: la ge size, can accommoda e small, medium, o la ge
ai c a .
No e ha a la ge ga e can also be used by smalle ai c a . The
dis inc ion be ween small, medium, and la ge ai c a is cla i ied in
ollowing Sec ion 3.3.
Finally, no e ha addi ional in o ma ion could be used o hope ully
p o ide mo e in o ma ion on u u e pe u ba ions. Howe e , i was
decided o keep his simpli ied s a e o mula ion o he ollowing
easons:
•Ensu ing me hod agnos icism wi h espec o ai po layou al-
lows o e sa ile applica ion ac oss a ious ai po s.
•En i onmen al ac o s, such as wea he condi ions, a e delib-
e a ely excluded wi hin his app oach. The in en ion is no o
p edic dis u bances bu , ins ead, o eac o hem wi hin he
decision-making amewo k o he GAP a he ai po .
•Wi h his s a e o mula ion, he algo i hm u ilizes iden ical inpu s
o hose o con en ional me hods employed as baselines. This
alignmen acili a es di ec compa isons,
•Fu u e ligh schedules con ain in o ma ion o he comple e day,
which wa an s a la ge s a e space. Inc easing his s a e would
u he conside ably inc ease he aining ime o he algo i hms.
3.3. Ac ion o mula ion
The e is a se o disc e e ac ions, whe e each one ep esen s a
ga e ha he agen may selec . Fo each s a e, he numbe o possible
ac ions co esponds o he numbe o a ailable ga es. Consequen ly,
his RL model can be un o di e en ai po con igu a ions, wi h
he only modi ica ion being he numbe o ga es. Each ac ion, 𝑥𝑛, is
cha ac e ized as a uple, and on op o he ga e numbe , i has mul iple
a ibu es which ep esen he limi a ions o he ga e:
𝑥𝑛= (𝐼𝑛, 𝜆𝑛, 𝜂𝑛, 𝜔𝑛, 𝜌𝑛).(2)
𝐼𝑛 ep esen s he ga e numbe , which is he ac ual elemen ha agen
chooses, o he elemen s se e as cons ain s. 𝜆𝑛 is he compa ible
ai c a size o his ga e:
•𝜆𝑛= 1: small ai c a (i.e. unde 100 sea s).
•𝜆𝑛= 2: medium ai c a (i.e. be ween 100 and 200 sea s).
•𝜆𝑛= 3: la ge ai c a (i.e. mo e han 200 sea s).
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
4
H. Li e al.
Fig. 1. The se e al phases o he ga e assignmen p ocess.
Fig. 2. S a e ansi ion.
𝜂𝑛 is he occupa ion ime o his ga e. Finally, 𝜔𝑛 is he ea lies
a ailable ime, and i is de e mined as ollows:
𝜔𝑛=
𝑓𝐼𝑛+𝜂𝐼𝑛(3)
whe e
𝑓𝐼𝑛
is he ligh a i al ime he las ime Ga e 𝐼𝑛 was chosen.
𝜂𝐼𝑛
is he ga e occupa ion ime when las ime Ga e 𝐼𝑛 is selec ed. 𝜌𝑛 is
a bina y indica o showing whe he his ga e is an Ap on ga e.
The di e en se ings o 𝑥𝑛 in luence he o e all pe o mance o he
ga e assignmen . I was chosen o de ine all possible ac ions in he o m
o 𝑥𝑛, ins ead o jus he ga e iden i ica ion, as o gi e mo e in o ma ion
o he RL model o make i s decision.
The e minal condi ion o his MDP is when all he inbound ligh s
ge hei ga e assignmen command. Fig. 2 demons a es he s a e an-
si ion p ocess, cha ac e ized by he s a e ansi ion unc ion 𝑝(𝑠𝑛, 𝑥𝑛).
3.4. Objec i e o mula ion
Gi en he cu en ci cums ances and he equi emen s o he ai po
g ound ope a ions, a mul i-objec i e app oach is p e e ed. Based on
he pe o mance selec ion me hod [44], (1) he numbe o he emo e
ga e and (2) passenge walking ime a e applied as he objec i es in his
pape . These objec i es a e hen combined in o a single ewa d ha is
gi en o he RL agen . The weigh o wo objec i es a e de e mined
acco ding o he p io i y lined ou in [44].
The numbe o emo e ga es cos , 𝐿, is deno ed as:
𝐿=
𝑓
∑
𝑛=1
𝜌𝑛(4)
We in e p e Passenge walking ime as an a e age man walking ime
aken om he secu i y check o he boa ding ga e, deno ed in Eq. (5).
𝑡𝑝𝑤𝑡,𝐼𝑛=
𝑑𝑝𝑤𝑡,𝐼𝑛
𝑣,(5)
whe e 𝑡(𝑝𝑤𝑡,𝐼𝑛) is he passenge ’s walking ime om he secu i y check
o he chosen Boa ding Ga e 𝐼𝑛 a S a e 𝑛. Pa ame e 𝑑(𝑝𝑤𝑡,𝐼𝑛) is he
dis ance om secu i y check o he chosen Boa ding ga e 𝐼𝑛 a S a e 𝑛.
In u n, 𝑣 is he a e age human walking eloci y (i.e. 5.43 km/h). The
objec i e is o minimize he o al PWT, deno ed in Eq. (6).
𝑇𝑝𝑤𝑡 =
𝑓
∑
𝑛=1
𝑡𝑝𝑤𝑡,𝐼𝑛(6)
Wi h he wo cos componen s men ioned abo e, he o al cos unc ion
is demons a ed as ollows:
𝐶=𝛼𝐿 + (1 − 𝛼)𝑇𝑝𝑤𝑡,(7)
whe e 𝛼 is a coe icien ha balances he wo objec i es’ ade-o s. We
assume 𝛼= 0.8, as we come up wi h his numbe upon ag eemen wi h
ai po ope a o s. The o e all objec i e unc ion o he GAP is hen
de ined:
𝑚𝑖𝑛[𝐶],(8)
3.5. Cons ain s
GAP is subjec ed o a se o egula o y cons ain s. Cons ain (9)
demons a es he selec ed ga e mus be one o he unc ioning ga es:
𝐼𝑛∈𝐼(9)
No e ha unc ioning means ha he ga e is wo king and can be used.
Cons ain (10) shows he lowe (uppe ) bounds o he ga e occupa ion
ime (𝜂), which is di e en acco ding o Assump ion 1:
𝜂𝑚𝑖𝑛 ≤𝜂𝑛≤𝜂𝑚𝑎𝑥 (10)
Cons ain (11) indica es he ype o he ga e:
𝜌𝑛={1i 𝐼𝑁∈𝐼𝑎,∀𝑛= 1,2,3,…
0o he wise (11)
whe e 𝐼𝑎 indica es Ap on ga es, and 𝐼𝑏 B idge ga es. The e a e also a
se o ope a ional cons ain s ha mus be complied wi h. Cons ain
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
5
H. Li e al.
(12) de ines ha he selec ed ga e mus be a ailable when he ligh is
a i ing:
𝜔𝑛≤𝑓𝑛,(12)
whe e 𝜔𝑛 ep esen s he ea lies a ailable ime o he ga e, and 𝑓𝑛 he
a i al ime o he ligh . Finally, cons ain (13) se s ha he ga e size,
𝜏𝑛, mus be compa ible wi h he ai c a size, 𝜆𝑛:
𝜆𝑛≤𝜏𝑛.(13)
4. Deep ein o cemen lea ning model
DRL is a machine lea ning scheme whe e an agen in e ac s wi h
he en i onmen o e a se ies o ime s eps. A each s a e, he agen
pe o ms an ac ion acco ding o he policy. The en i onmen hen
changes acco ding o he ac ion. A ewa d is gi en o he agen —
his ewa d e alua es how good he en i onmen al change was. The
RL model uses his ewa d o e alua e how good he pe o med ac ion
was. The long- e m ewa d is he accumula i e immedia e ewa d when
he en i onmen eaches i s e minal s a e. RL aims o maximize long-
e m ewa d o e episodes o he aining p ocess. In o he wo ds, RL
a emp s o ind he ac ions ha yield be e ewa ds.
Sec ion 4.1 de ines he componen s o he de eloped RL model:
s a e, ac ions, ewa d, and ac ion shaping. Sec ion 4.2 de ines he RL
algo i hm employed in he wo k, he A3C. Finally, Sec ion 4.3 de ines
how he RL model is used o p oduce a new eal- ime ga e assignmen
based on delays o incoming ai c a .
4.1. Inco po a ion o componen s
The i al componen s o he RL model ha a e used in his pape
a e p esen ed he e:
•S a e: De ined in Sec ion 3.2 as 𝑠𝑛= (𝑓𝑛, 𝜏𝑛). The size o he s a e
space in leng h equals he o al ligh s.
•Ac ion: De ined in Sec ion 3.3 as 𝑥𝑛= (𝐼𝑛, 𝜆𝑛, 𝜂𝑛, 𝜔𝑛, 𝜌𝑛). The size
o he ac ion space equals he numbe o a ailable ga es.
•Rewa d: Since ein o cemen lea ning elies on he accumula ion
o immedia e ewa ds, he o al cos 𝐶 mus be decomposed. The
o al cos comp ises wo componen s: he cos associa ed wi h
assigning ligh s o emo e ga es and he walking ime cos om
secu i y o he boa ding ga e. These can be in e p e ed as he
cos s co esponding o wo ypes o ac ions: assigning a ligh o a
emo e ga e o o a b idge ga e. To e lec his, we designed wo
mu ually exclusi e e ms in he immedia e ewa d unc ion:
𝑟𝑛=𝛼𝜌𝑛+ (1 − 𝛼)(1 − 𝜌𝑛)𝑡𝑝𝑤𝑡,𝐼𝑛(14)
whe e 𝑟𝑛 ep esen s immedia e ewa d a s a e 𝑛, 𝜌𝑛 and necessa y
in o ma ion needed in 𝑡𝑝𝑤𝑡,𝐼𝑛
can be acqui ed in 𝑥𝑛.
In Eq. (14), only one o he wo e ms is applied depending on
𝜌𝑛 in 𝑥𝑛. While adi ional ein o cemen lea ning (RL) ocuses on
sol ing a maximiza ion p oblem, he Ga e Assignmen P oblem
(GAP) is amed as a minimiza ion p oblem. Assigning ligh s o
ap on ga es is mos always undesi able; he e o e, he i s hal
o he ewa d is aken he opposi e. A nega i e ewa d signals o
he agen ha his ac ion is un a o able. Fo ac ions in ol ing
b idge ga es, which adi ionally esul in posi i e ewa ds, we
in e he e m o ensu e he ewa d emains posi i e while
p io i izing lowe walking imes. This adjus men ensu es ha a
lowe walking ime co esponds o a highe ewa d. The inal
immedia e ewa d unc ion is de ined as ollows, inco po a ing
𝛼 o egula e he ela i e impo ance o he wo e ms.
𝑟𝑛= −𝛼𝜌𝑛+ (1 − 𝛼)(1 − 𝜌𝑛)𝑡−1
𝑝𝑤𝑡,𝐼𝑛(15)
•Ac ion Shaping [45–47]: his echnique is used o ain he
agen o choose be e ac ions. The e will be a ious ga es ha
a e una ailable o unsui able o choose om a each s a e. The
chance o selec ing hese aul y ga es should be ze o. We employ
an ac ion mask o il e ou all he impossible ac ions a each
s age, o cing he agen only o choose legal ac ions, conside ably
imp o ing lea ning e iciency. The ac ion mask is implemen ed
h ough a ec o :
Mask𝑎(𝑆𝑡𝑎𝑡𝑒) = [mask𝑎,1(𝑠𝑡𝑎𝑡𝑒),…,mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒)],(16)
whe e mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒) is de ined as ollows:
mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒) = {1i 𝐼𝑁∈𝐼𝑎𝑣𝑎,∀𝑛= 1,2,3,…
0o he wise (17)
whe e 𝐼𝑎𝑣𝑎 ep esen s he subse o ga es ha a e cu en ly unc-
ional, a ailable, and ha e a size ma ching he ai c a and su ice
all he o he cons ain s. Thus, he mask will only enable he RL
agen o pick ga es ha a e unc ional and ma ch he size o he
ligh and mee o he equi emen s.
Finally, he ac ion mask design is adap able and can be easily
cus omized o di e en se ings and cases. We may se he p oba-
bili ies o in alid ac ions o ze o by adding an ex a laye be o e
he ac ion p obabili y calcula ion in he Ac o ne wo k.
4.2. Asynch onous Ad an age Ac o C i ic (A3C)
In gene al, he DRL algo i hm has wo ca ego ies: Value-based and
Policy-Based. The alue-based app oach is a de e minis ic policy, in
which when he model is op imized o he bes , e e y s a e’s co e-
sponding ac ion is de e mined, while Policy-based is s ochas ic. In a
policy-based algo i hm, e e y s a e/ac ion se ies in each episode is
deno ed as T ajec o y (𝜏):
𝜏= {𝑠1, 𝑥1, 𝑠2, 𝑥2,…, 𝑠𝑛, 𝑥𝑛}(18)
The p obabili y o his ajec o y is:
𝑝𝜃(𝜏) = 𝑝(𝑠1)𝜋𝜃(𝑥1|𝑠1)𝑝(𝑠2|𝑠1, 𝑥1)𝜋𝜃(𝑥2|𝑠2)...
=𝑝(𝑠1)𝛱𝑁
𝑛=1𝜋𝜃(𝑥𝑛|𝑠𝑛)𝑝(𝑠𝑛+1|𝑠𝑛, 𝑥𝑛)(19)
The o al ewa d o his ajec o y is:
𝑅(𝜏) =
𝑁
∑
𝑛=1
𝑟𝑛(20)
The goal is o ind he bes policy 𝜋𝜃, ha can op imize he o al
ewa d 𝑅(𝜏).
Howe e , i is di icul o ind a sa is ying policy. To sol e his
d awback, he Ac o -C i ic (AC) amewo k has been pu o wa d. In
he AC algo i hm, he agen is comp ised o an ac o and a c i ic.
The ac o ac s unde he cu en s a e using he cu en policy, hen
he en i onmen will swi ch o a new s a e and e u n he immedia e
ewa d o he c i ic. The c i ic will judge he ac ion based on he ewa d
and e u n i o he ac o o upda ing and calib a ing he policy. A3C is
p oposed based on he AC algo i hm, whe e he di e ence is ha A3C
employs mul iple agen s wo king concu en ly, and asynch onously
ains he A i icial Neu al Ne wo k (ANN), he e o e accele a ing he
aining p ocess while enhancing he con e gence signi ican ly.
He e, we p esen he wo king p ocess o one agen .
Simila o AC algo i hm, The A3C algo i hm has a policy unc ion
𝜋𝜃(𝑥𝑛|𝑠𝑛), and a alue unc ion 𝑉𝜃𝑣(𝑠𝑛),whe e 𝜃, 𝜃𝑣 a e weigh s pa am-
e e s. Bo h policy and alue unc ions a e upda ed a e ce ain s eps
o ac ion. The alue o he discoun ed ewa d, 𝐷𝑅(𝜏), ells he agen
which ac ions a e ewa ding and which a e no .
𝐷𝑅(𝜏) = 𝑟1+𝛾𝑟2+𝛾2𝑅3... =
∞
∑
𝑛=1
𝛾𝑛−1𝑟𝑛(21)
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
6
H. Li e al.
whe e 𝛾(0 < 𝛾 < 1) is he discoun ac o . The g ea e discoun ac o
he mo e he agen is mo e ocused on he u u e ewa d. Thus, he
alue unc ion is de ined as:
𝑉𝜃𝑣(𝑆𝑛) = 𝐸[𝐷𝑅(𝜏)|𝑠=𝑠𝑛, 𝜋] = 𝐸[
∞
∑
𝑛=1
𝛾𝑛−1𝑟𝑛|𝑠=𝑠𝑛, 𝜋](22)
Ra he han use one-s ep Time-Di e ence-e o (TD-e o ) in AC,
we use n-s ep TD-e o o be e upda e he pa ame e s. The upda e
pe o med can be seen as:
∇𝜃𝐽(𝜃) = ∇𝜃𝑙𝑜𝑔𝜋𝜃(𝑥𝑛|𝑠𝑛)𝐴𝜃,𝜃𝑣(𝑠𝑛, 𝑥𝑛)(23)
𝐴𝜃,𝜃𝑣(𝑠𝑛, 𝑥𝑛) = 𝑄(𝑠𝑛, 𝑥𝑛) − 𝑉𝜃𝑣(𝑠𝑛)(24)
whe e 𝑄(𝑠𝑛, 𝑥𝑛) = ∑𝑁
𝑛=1 𝛾𝑛−1𝑟𝑛+𝛾𝑛𝑉𝜃𝑣(𝑠𝑛+1).
To encou age explo a ion and a oid p ema u e con e gence, an
en opy i em is adop ed. The Ac o ne wo k pa ame e upda e is pe -
o med based on he ollowing accumula ed g adien :
𝑑𝜃 ←𝑑𝜃 + ∇𝜃′𝑙𝑜𝑔𝜋𝜃′(𝑥𝑛|𝑠𝑛)𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛) + 𝛽∇𝜃′𝐻(𝜋𝜃′(𝑠𝑛)) (25)
whe e 𝐻(𝜋𝜃′(𝑠𝑛)) is he en opy, 𝛽 con ols he ela i e impo ance o
he en opy and he ewa d. The C i ic-ne wo k pa ame e upda e is
pe o med based on he ollowing accumula ed g adien :
𝑑𝜃𝑣←𝑑𝜃𝑣+
𝜕(𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛))2
𝜕𝜃′
𝑣
(26)
Finally, he p oposed A3C-based Ga e assignmen algo i hm is
demons a ed in Algo i hm 1 [29].
Algo i hm 1 A3C-based Ga e Assignmen Algo i hm
1: Ini ializa ion:
2: Ini ialize global ac o -ne wo k and c i ic ne wo k pa ame e s wi h
𝜃 and 𝜃𝑣
3: Se global Episode Coun e T=0 and h ead episode coun e =0
4: Ini ialize h ead ac o -ne wo k and c i ic ne wo k pa ame e s wi h
𝜃′ and 𝜃′
𝑣
5: Ini ialize 𝑇𝑚𝑎𝑥, 𝑡𝑚𝑎𝑥, 𝑁, 𝛾, 𝛽
6: DO:
7: Se = 1
8: Rese 𝑑𝜃 = 0 and 𝑑𝜃𝑣= 0
9: while 𝑇 < 𝑇𝑚𝑎𝑥 do
10: Re ie e pa ame e s om he global ne wo k: 𝜃′=𝜃, 𝜃′
𝑣=𝜃𝑣
11: Ob ain s a e 𝑠𝑛
12: while 𝑡 < 𝑡𝑚𝑎𝑥 do
13: Choose ac ion 𝑥𝑛 unde he policy 𝜋𝜃(𝑥𝑛|𝑠𝑛)
14: Ge ewa d 𝑟𝑛 and new s a e 𝑠𝑛+1
15: 𝑡=𝑡+ 1
16: end while
17:
𝑄={0, 𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙𝑠𝑡𝑎𝑡𝑒
𝑉𝜃′(𝑠𝑛), 𝑛𝑜𝑛 −𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙𝑠𝑡𝑎𝑡𝑒 (27)
18: while 𝑡=𝑡𝑚𝑎𝑥 do
19: 𝑄=𝑟𝑛+𝛾𝑄
20: Upda e h ead ac o -ne wo k pa ame e :
𝑑𝜃 ←𝑑𝜃 + ∇𝜃′𝑙𝑜𝑔𝜋𝜃′(𝑥𝑛|𝑠𝑛)𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛) + 𝛽∇𝜃′𝐻(𝜋𝜃′(𝑠𝑛)) (28)
21: Upda e h ead c i ic-ne wo k pa ame e :
𝑑𝜃𝑣←𝑑𝜃𝑣+
𝜕(𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛))2
𝜕𝜃′
𝑣
(29)
22: end while
23: Pe o m asynch onous upda e 𝜃 and 𝜃𝑣 using 𝑑𝜃 and 𝑑𝜃𝑣
24: T= T+1
25: end while
The global amewo k o he A3C algo i hm is illus a ed in Fig. 3.
The agen is comp ised o an ac o and a c i ic. The ac o ac s unde
he cu en s a e using he cu en policy. Then he en i onmen will
swi ch o a new s a e and e u n he immedia e ewa d o he c i ic.
The c i ic will judge he ac ion based on he ewa d and e u n i
o he ac o o upda ing and calib a ing he policy. A3C is p oposed
based on he Ac o -C i ic algo i hm, whe e he di e ence is ha A3C
employs mul iple agen s wo king concu en ly and asynch onously
aining he ANN, accele a ing he aining p ocess while enhancing he
con e gence signi ican ly.
4.3. Real- ime Ga e Assignmen P oblem Solu ion (REGAPS)
Wi h he success ul implemen a ion o he A3C-GAP algo i hm,
he Ga e P e-assignmen P oblem has been sol ed. Howe e , he p e-
assignmen schedule always de ia es om he ac ual schedule due o
mul iple causes, such as wea he condi ions and egula ions. Hence,
he p oposed REGAPS algo i hm diag am is demons a ed in Fig. 4. A
Time Ze o, which ma ks he beginning o he day, he model ini ia es
he execu ion o an algo i hm ha ea s ga e assignmen as a p e-
assignmen p oblem, elying on he o iginal ligh schedule. Unde
no mal ci cums ances, he ope a o p oceeds wi h ga e assignmen s pe
he p ede e mined schedule when no delays a e encoun e ed. Howe e ,
upon de ec ing a delay (a Reschedule poin T, Fig. 4), he model
p omp ly upda es he ligh schedule and e-execu es he algo i hm
using he e ised inpu , hus ob aining an upda ed assignmen . The
igu e labeled as Fig. 5 p o ides a de ailed explana ion o how ligh
schedules a e upda ed. No e ha in his pape , we assume ha once
he ga e occupa ion ime (which includes boa ding ime) is comple ed,
he ai c a will ei he be eady o akeo o mo ed o he ap on in
he e en o a delay, as Assump ion 1. The e o e, depa u e delays do
no a ec ga e usage in ou cu en model. I shows ha when delays
o changes in schedules a e iden i ied, only he a i al imes o a ec ed
ligh s a e adjus ed, while he assigned ga es and a i ed ligh s emain
he same. This i e a i e p ocess ensu es he a ainmen o eal- ime
ga e assignmen s, acili a ing hei comple ion in esponse o changing
condi ions.
To sol e he eal- ime assignmen p oblem, he ini ializa ion should
be se as he new ime able o he a i ing ligh s, and he ga e oc-
cupa ion s a es should be se simul aneously. Also, se e al bu e s a e
applied o s o e he bes ac ion, bes ewa d, s a e in o ma ion, and ga e
occupa ion s a us.
5. Hypo heses
The ollowing hypo heses a e es ablished ega ding how he RE-
GAPS algo i hm can imp o e ga e assignmen o e con en ional me h-
ods:
Hypo hesis 1: The REGAPS algo i hm is poised o su pass o he me h-
ods by s a egically conside ing he sequence o ac ions o e ime. No e
ha he REGAPS algo i hm does no possess addi ional in o ma ion
compa ed o o he me hods, and does no p edic dis u bances. How-
e e , we hypo hesize ha i s unique capaci y lies in lea ning om
pas expe iences. The REGAPS algo i hm can disce n which decisions
esul in mo e a o able ou comes. Fo ins ance, i can lea n which ga e
alloca ion pa e ns lead o educed Ap on usage by day’s end.
Hypo hesis 2: The REGAPS algo i hm will p io i ize educing Ap on
ga e usage, as his alue was made pa amoun in he ewa d o mula-
ion (Sec ion 3.4).
Hypo hesis 3: The REGAPS will imp o e upon con en ional me hod
in en i onmen s wi h highe complexi y and unce ain ies. In en i on-
men s wi h lowe complexi y, i is expec ed ha con en ional me hods
al eady pe o m well and he in oduc ion o REGAPS may be no
ep esen and imp o emen .
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
7
H. Li e al.
Fig. 3. The A3C amewo k.
Fig. 4. Diag am o he p oposed REGAPS.
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
8
H. Li e al.
[43] Jang H-C, Huang Y-C, Chiu H-A. A s udy on he e ec i eness o A2C and A3C
ein o cemen lea ning in pa king space sea ch in U ban A eas P oblem. In:
2020 in e na ional con e ence on in o ma ion and communica ion echnology
con e gence. ICTC, 2020, p. 567–71.
[44] Li H, Wu X, Liang Y, Zhang C. A mul is akeholde app oach o he ai po
ga e assignmen p oblem: Applica ion o uzzy heo y o op imal pe o mance
indica o selec ion. Compu In ell Neu osci 2021;2021. Ci ed by: 0; All Open
Access, Gold Open Access, G een Open Access.
[45] Huang S, On añón S. A close look a in alid ac ion masking in policy g adi-
en algo i hms. In: The in e na ional FLAIRS con e ence p oceedings. Vol. 35,
Uni e si y o Flo ida Geo ge A Sma he s Lib a ies; 2022.
[46] Kane is o A, Schelle C, Hau amäki V. Ac ion space shaping in deep e-
in o cemen lea ning. In: 2020 IEEE con e ence on games (coG). 2020, p.
479–86.
[47] Mülle A, Saba elli M. Sa e and psychologically pleasan a ic signal con-
ol wi h ein o cemen lea ning using ac ion masking. In: 2022 IEEE 25 h
in e na ional con e ence on in elligen anspo a ion sys ems. ITSC, 2022, p.
951–8.
[48] Kla M, Gla M, Au ich JC. Pe o mance compa ison o ein o cemen lea ning
and me aheu is ics o ac o y layou planning. CIRP J Manu Sci Technol
2023;45:10–25.
[49] CAAC. Technical guidelines o ai po epidemic p e en ion and con ol. 8 h ed..
2021.
[50] p-ch is . Deep-Rein o cemen -Lea ning-Algo i hms-wi h-PyTo ch: PyTo ch
implemen a ions o deep ein o cemen lea ning algo i hms and en i onmen s.
2024, h ps://gi hub.com/p-ch is /Deep-Rein o cemen -Lea ning-Algo i hms-
wi h-PyTo ch. [Accessed 03 Oc obe 2024].
[51] zhangzhengmin. GA o Ga e assignmen . 2024, h ps://gi hub.com/
zhangzhengmin/–GA. [Accessed 17 Decembe 2024].
[52] Hu X-B, Di Paolo E. An e icien gene ic algo i hm wi h uni o m c osso e o
he mul i-objec i e ai po ga e assignmen p oblem. In: Mul i-objec i e meme ic
algo i hms. Be lin, Heidelbe g: Sp inge Be lin Heidelbe g; 2009, p. 71–89.
[53] Ding C, Bi J, Wang Y. A hyb id gene ic algo i hm based on imi a ion lea ning
o he ai po ga e assignmen p oblem. En opy 2023;25(4).
Ope a ions Resea ch Pe spec i es 14 (2025) 100338
15
