Deep reinforcement learning approach for real-time airport gate assignment

Li, Haonan; Wu, Xu; Ribeiro, Marta; Santos, Bruno; Zheng, Pan Article Deep reinforcement learning approach for real-time airport gate assignment Operations Research Perspectives Provided in Cooperation with: Elsevier Suggested Citation: Li, Haonan; Wu, Xu; Ribeiro, Marta; Santos, Bruno; Zheng, Pan (2025) : Deep reinforcement learning approach for real-time airport gate assignment, Operations Research Perspectives, ISSN 2214-7160, Elsevier, Amsterdam, Vol. 14, pp. 1-15, https://doi.org/10.1016/j.orp.2025.100338 This Version is available at: https://hdl.handle.net/10419/325815 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. https://creativecommons.org/licenses/by/4.0/ Contents lists available at ScienceDirect Operations Research Perspectives journal homepage: www.elsevier.com/locate/orp Deep reinforcement learning approach for real-time airport gate assignment Haonan Lia,∗, Xu Wu b, Marta Ribeiro a, Bruno Santos a, Pan Zheng c aDelft University of Technology, Kluyverweg 1, Delft, 2629 HS, Zuid-holland, The Netherlands bBeijing Jiaotong University, Beijing, China cCivil Aviation Management Institute of China, Beijing, China A R T I C L E I N F O Keywords: Airport gate assignment Deep reinforcement learning Asynchronous advantage actor-critic Real-time optimization A B S T R A C T Assigning aircraft to gates is one of the most important daily decision problems that airport professionals face. The solution to this problem has raised a significant effort, with many researchers tackling many different variants of this problem. However, most existing studies on gate assignment contain only a static perspective without considering possible future disruptions and uncertainties. We bridge this gap by looking at gate assignments as a dynamic decision-making process. This paper presents the Real-time Gate Assignment Problem Solution (REGAPS) algorithm, an innovative method adept at resolving pre-assignment issues and dynamically optimizing gate assignments in real-time at airports through the integration of Deep Reinforcement Learning (DRL). This work represents the first time that DRL is used with real airport data and a configuration containing a large number of flights and gates. The methodology combines a tailored Markov Decision Process (MDP) formulation with the Asynchronous Advantage Actor–Critic (A3C) architecture. Multiple factors, such as flight schedules, gate availability, and passenger walking time, are considered. An empirical case study demonstrates that the REGAPS outperforms two classic deep Q-learning algorithms and a traditional Genetic Algorithm in terms of reducing passenger walking time and apron gate assignment. Finally, supplementary experiments highlight REGAPS’s adaptability under various gate assignment rules for international and domestic flights. The finding demonstrates that not only did REGAPS outperform COVID restrictions, but it can also produce considerable benefits under other policies. 1. Introduction One of the most significant current discussions in airport operation discipline is gate assignment, the process of allocating flights to airport gates. Based on the statistical data acquired by IATA [1], in 198 of 354 airports surveyed, they lack the capacity to fulfill the existing demand, necessitating a greater degree of flexibility in the assignment of gates. Thus, it is vital to analyze the nature of the gate assignment problem (GAP) and design an efficient and effective approach to ease the stress of airport coordination. Many research works have tackled gate assignment [2]. One key issue with most of the past studies is that they were conducted from a static perspective, assuming a static environment without any unpredictable changes. As a result, no lastminute disturbances are taken into account. However, existing research shows that flight delays negatively affect the performance of static gate assignments [3]. Consequently, these solutions can only be used as strategic planning, from which airport professionals must deviate in real-time to accommodate unplanned changes. In practice, airport professionals assign gates in two stages: first in pre-assignment and then in real-time. In the pre-assignment phase, ∗Corresponding author. E-mail address: [email protected] (H. Li). gates are assigned in advance based on previous data. As plane travel is so readily disrupted, the pre-assignment plan will almost certainly diverge from the final plan. Operators often handle disturbances manually, depending on their experience. Some researchers observed the deviation and attempted to solve it [4–6]. These attempts to mitigate the impact of disruption by strengthening the assignment plan’s robustness or researching solutions to the reassignment challenge. However, current systems are incapable of rapidly adjusting the assignment plan in response to changes in the flight schedule, and no previous work has investigated automatic adjustment of the assignment schedule from a real-time perspective. This study aims to bridge existing research gaps by departing from a static viewpoint in gate assignments, by analyzing the gate assignment procedure as a dynamic, real-time decision-making process. Thus, our main research objective can be described as: To develop a unified decision support system for both static and dynamic scenarios. In other words, our model can be used at the two stages, in pre-assignment and then in real-time after disruptions occur. Our solution stands-out for being a single model that can be run at any time, with updated information, https://doi.org/10.1016/j.orp.2025.100338 Received 8 October 2024; Received in revised form 23 December 2024; Accepted 2 April 2025 Operations Research Perspectives 14 (2025) 100338 Available online 11 April 2025 2214-7160/© 2025 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license ( http://creativecommons.org/licenses/by/4.0/ ). H. Li et al. and by being able to provide a solution in a matter of milliseconds. Our objective is to explore the efficacy of this promising approach in addressing the stochastic and dynamic nature of gate allocation challenges. We propose that by leveraging the sequential decisionmaking of Deep Reinforcement Learning (DRL), our framework may effectively manage urgent and frequent changes. We anticipate that this approach will (1) enhance current gate allocation methods by reducing future instances of undesirable remote gate usage, (2) strike a balance between competing objectives such as minimizing passenger walking time and prioritizing flight schedules, and (3) exhibit exceptional flexibility, enabling swift execution under varying flight configurations with remarkably fast run times. Our approach to the Gate Assignment Problem (GAP) involves two stages, each leveraging Markov Decision Processes (MDP) and DRL methodologies. MDP provides a data-based framework for automatically solving sequential decision-making problems. With MDP, the environment is modeled as a set of states and actions that can be performed to control the system’s state. The goal is to control the system in such a way that some performance criteria is maximized. This modeling constitutes the base for the DRL model. DRL has the advantage of not having to rely on a complete and accurate model of the environment. Therefore, it can maximize rewards in environments where the exact dynamics of the environment are not known. We employ the Asynchronous Advantage Actor-Critic (A3C) architecture to address the GAP due to its unique properties — using multiple independent interacting agents allows for greater exploration in less time. Initially, we address the GAP from a static perspective, termed pre-assignment, where gate assignments are made before operational commencement. Subsequently, we transition to real-time assignment, where gate assignments are dynamically adjusted in response to evolving operational conditions, including real-time flight schedule changes. With each approaching flight, a decision is made, and we monitor the results before adjusting our decisions. Leveraging the inherent adaptability of MDP and RL, our framework seamlessly incorporates real-time adjustments, ensuring efficient and responsive gate allocation within airport operations. This paper is organized as follows: Section 2 describes past GAP research as well as contemporary DRL applications in the transportation sector. In Section 3, an MDP formulation is proposed, which is tightly tailored to the GAP. Section 4 incorporates the developed MDP into the A3C architecture and builds the real-time GAP solution (REGAPS). The hypotheses for the implemented method are presented in Section 5. Section 6 conducts an empirical case study based on real airport data and evaluates and validates the model’s performance. Note that the specific airport is not disclosed due to privacy issues. Section 7 discusses whether the initial hypotheses were verified. Finally, Section 8 concludes this paper. 2. Related work This section covers the state-of-the-art on research on gate assignment. Previous works on this topic are described in Section 2.1. Additionally, given that DRL is prominent in this work, Section 2.2 describes the latest advances in this area outside of GAP. Finally, Section 2.3 will clearly define the gaps in literature to be covered by this work. 2.1. Gate Assignment Problem (GAP) Research into GAP has a long history in the operations research discipline due to its practical importance. There are multiple types of approaches with respect to the solution methodologies. These can be categorized into four parts: •Expert systems design software to simulate the decision-making process of human experts. Some studies that fall into this category are Brazile & Swigger [7], Gosling [8], Srihari & Muthukrishnan [9] and Su & Srihari [10]. •Exact methods are one of the most focused methodologies for GAP mainly modeled as Integer programming or Linear Programming. For instance, Barbic [11] published one of the earliest studies using a branch and bound algorithm with a lower bound that minimizes the total walking distance of the passengers. Mangoubi & Mathaisel [12] proposed a greedy heuristic approach and a linear programming relaxation method to minimize the total walking distance. Bolat [13] focused his objective on minimizing the difference between the minimum and maximum slack times, using a branch-and-bound algorithm and a heuristic approach called branch-and-trim. The primary limitation of exact methods is the extensive computation time. In practical GAPs, the objective is not to find a globally optimal solution but rather to obtain a satisfactory solution within a reasonable time frame. •Stochastic Processes and Heuristic: this is the example of the work from Haghani and Chen [14] that proposed a heuristic algorithm to minimize passenger walking distance by assigning successive flights to the same slot. Ding et al. [15] developed a greedy algorithm to acquire an initial schedule for the hybrid algorithms based on Tabu search and Simulated Annealing later on. Recently, Jie Li et al. [16] presented a column generation-based algorithm to solve gate assignment with combinational gates. Jiang et al. [17] introduce a novel approach to optimizing airport gate assignments, addressing harbor apron safety constraints through a two-phase mathematical model. Refinements to the branchand-price method improve efficiency and accuracy in solving the problem. She et al. [18] addresses the Airport Gate Assignment Problem (AGAP) by proposing a multi-objective integer programming model and a two-phase Monte Carlo-based NSGAII algorithm. Computational analyses validate the efficacy of the approach in providing economical, robust gate assignments. The drawback of stochastic processes lies in their inherent high computational complexity, coupled with the challenge of generating realistic scenarios. •Reinforcement learning (RL) approaches: these offer an advantage over stochastic processes by not needing to specify a model. RL, the subject of this paper, is model-free, providing a datadriven, learning-based framework to formulate and solve sequential decision-making problems. RL has been widely used in aviation, with applications ranging from airline revenue management to aircraft altitude control [19]. Many aviation-related scenarios can be conceptualized as sequential decision problems, making RL an ideal tool. In various domains, RL has demonstrated promising outcomes. Nevertheless, to the best of the authors’ knowledge, the work Yildizi et al. [20] stands as the pioneering and sole attempt to apply RL to gate assignment. Their preliminary study uncovered a time complexity issue when dealing with more than 12 gates. Nonetheless, a variety of RL algorithms and techniques exist, and determining the most suitable one for each problem requires extensive examination. It is evident that further research is warranted to explore the potential of RL for GAP. To this end, we propose a more complex RL algorithm capable of handling a larger number of gates. Despite all the effort that has been done to assist on-ground operations, few of them are actually implemented in airports. One of the major issues is that these approaches are mainly designed from a static perspective. These solve the Pre-assignment problem, which is considerably different from the final gate assignment once perturbations in real-time operations occur. Although some solutions are formulated as stochastic and robust approaches, and try to solve GAP from a dynamic perspective, with the limitation of heuristic algorithms, these can only minimize idle time and gate conflicts, but do not fully adjust to future uncertainties and perturbations. Furthermore, the scale of the problem warrants the utilization of methods that are not reliant on specific models and can effectively Operations Research Perspectives 14 (2025) 100338 2 H. Li et al. incorporate the temporal dimension of action sequences. First, the fact that RL learns by direct training in the environment renders it applicable to any airport configuration. Note that the decision space for this problem can vary largely, from the number of flights to the multitude of gates, depending on the unique characteristics of each airport. Second, gate assignment represents a sequential dilemma wherein the allocation of specific gates at present influences the future availability of gates for subsequent aircraft. RL has the ability to consider future states when making decisions. Lastly, for large airports, real-time evaluation using conventional methods is not a solution, given their larger execution times. 2.2. Deep Reinforcement Learning (DRL) To find the next-generation solution for the GAP, we resort to DRL. The foundation of RL is the MDP, which is molded with the gate assignment process. This section explores DRL works employed in other areas. RL aims to achieve a maximum reward through actions made by the agent and the interaction reward with the environment. The RL framework has become promising due to its ability to learn the dynamics of the environment through direct interaction with the environment. With improved data availability and computational power, RL studies in aviation now range from airline revenue management to air traffic control (ATC) [19]. Recently, DRL has emerged. This coalesces deep learning (DL) and RL together and exploits both of their advantages the most. DL is widely used for function approximation and value prediction. There have been many precedents regarding DRL implementation in the transportation discipline. Lin [21] proposed a DRL approach toward the Electric Vehicle Routing Problem with Time windows. Yu [22] offered a solution for online vehicle routing with neural combinatorial optimization and DRL. There are also applications on smart transportation systems concerning the rebalancing problem of the Bike-sharing system [21]. Li et al. [23] present a novel heuristic algorithm for the Gate Assignment Problem (GAP) in airport management. Combining tabu search with reinforcement learning, it efficiently explores solutions, outperforming existing methods in solution quality and computation time across real-world benchmarks. Sui et al. [24] introduce a tactical conflict resolution strategy using Deep Reinforcement Learning to mitigate the increasing risk of flight conflicts in airspace. Controllers’ actions are modeled as a Markov Decision Process, trained by the Deep Q Network algorithm. Simulation experiments confirm the strategy’s feasibility and alignment with real-world flight safety regulations. Zhang et al. [25] address the Multi-Trip Vehicle Routing Problem with Time Windows using a novel Coordinated Multi-agent Hierarchical Deep Reinforcement Learning approach. By structuring a three-layered framework, this method enhances solution quality and convergence rates, outperforming traditional heuristic algorithms and reinforcement learning techniques. Results indicate significant improvements in cost effectiveness and operational robustness, demonstrating the effectiveness of the proposed approach in transportation scheduling. DRL has also been employed to address dynamic resource allocation problems. In this context, gate allocation can also be conceptualized as a type of resource, suggesting that insights from this research domain may be relevant. Jia Wang et al. [26] propose an incremental reinforcement learning framework for dynamic resource allocation, where task patterns are extracted from large-scale data. They construct an environment model that enables a learning agent to infer the logic of task service operations and calculate feedback scores for each allocation decision. Applications of dynamic resource allocation also extends to the transportation sector. For instance, Ying He et al. [27] propose a general framework that enables fast-adaptive resource allocation in dynamic vehicular environments by integrating hierarchical reinforcement learning with meta-learning. This approach allows the framework to quickly adapt to new environments by fine-tuning only the top-level master network, while the low-level sub-networks continue to make optimal resource allocation decisions. In another example, Hongbin Liang et al. [28] model the resource allocation problem in the Internet of Vehicles as a semi-Markov decision process, incorporating a resource reservation strategy and a secondary resource allocation mechanism. A RL algorithm is applied to solve the model. 2.2.1. DRL algorithms DRL is also a generalized term for a large class of algorithms. One branch of DRL is called value-based approaches. The algorithm’s core is the value function that directly indicates the value or reward of an action. Deep Q-learning Network (DQN) [29] is a commonly used DRL model. However, it tends to overestimate the Q-value [30]. Many attempts have been carried out to solve this deficiency, and many variants emerged, such as Double DQN (DDQN) [31], and Dueling DQN [32]. A second branch is Policy-based approaches. The difference from value-based approaches is that rather than directly adjusting the action based on the reward, it adjusts the probability of choosing a certain action. A commonly used policy-based algorithm is the REINFORCE algorithm [33]. There are also many other researchers who have adopted DRL in the transportation field [34–38]. In recent years, actor-critic algorithms [39] have been proposed to take advantage of the best properties of value-based and policy-based algorithms. These are divided into two parts: (1) generating an action based on a state, and (2) computing the Q-value of the action. The actor takes an action based on the given state, while the critic evaluates the action with the value-based function. Then the actor adjusts the probability of the selected action based on the critic’s evaluation. Even though these methods sound promising, implementing them has many difficulties. Actor-critic requires Artificial Neural Network (ANN) training both in the Actor and Critic models, which no doubt imposed great difficulty in network training. Also, the Q-value generated by the critic model is unable to tell the actor how much better or worse is the chosen action. The Asynchronous Advantage Actor-Critic (A3C) [39], used in this work, is one of the solutions to conquer these drawbacks. This model asynchronously parallel trains different agents on multiple independent environments. Each agent will update the global network asynchronously. This strategy not only exploits most of the computing capability of a workstation but also greatly accelerates the training process [40–42]. Note that the A3C algorithm was picked over an Advantage Actor–Critic (A2C) approach due to its asynchronous aspect, which has shown faster learning times [43] on a multi-core CPU computer as used by the authors. 2.3. Literature gaps The review of the existing literature underscores a critical gap in addressing the dynamic nature of the GAP, which has predominantly been approached from a static perspective, focusing on Pre-assignment. Despite sporadic efforts to incorporate stochastic and robust methods, existing solutions remain hindered by heuristic algorithms and struggle to adapt to real-time perturbations and uncertainties. In response to this challenge, our study proposes a novel approach that integrates DRL techniques with the MDP framework. Specifically, we employ the A3C algorithm, renowned for its parallel training capability, enabling efficient decision-making in time-sensitive scenarios. What sets our approach apart is its unique capability to seamlessly address both pre-assignment and real-time assignment challenges with minimal modification. By leveraging the synergistic capabilities of DRL and MDP, our framework offers enhanced flexibility and adaptability in optimizing gate assignments across various operational scenarios. Comparing with other approaches, DRL has the following advantages: •The process of customizing the DRL environment and reward function is straightforward. Operators can easily design or modify gate-related information, such as the number of gates, their Operations Research Perspectives 14 (2025) 100338 3 H. Li et al. distance from security checkpoints, and the specific flights assigned to each gate, based on the unique requirements of the airport. This flexibility enables airports to tailor the system to their operational needs without requiring advanced technical expertise. •DRL allows us to anticipate future changes and disruptions while generating fast and effective solutions to support operators in making decisions under tight time constraints. Operators can easily incorporate changes and disturbances by updating the state space, enabling the system to adapt swiftly to evolving conditions and ensuring that decision-making remains responsive and efficient. •DRL is capable of handling both static and dynamic scenarios simultaneously or switching between them as needed by operators, providing the flexibility to accommodate varying operational demands. Depending on the operator’s requirements, DRL can generate either pre-assignment results or real-time assignment results, allowing the system to adapt to different scheduling and decision-making contexts efficiently. 3. Model formulation The GAP poses a significant operational challenge in airports, revolving around the allocation of arrival and departure flights to suitable gates. The primary goal is to optimize gate assignments, taking into account key factors like minimizing passenger walking distances, maximizing gate utilization, and mitigating operational delays. In our research, we incorporate real-time gate assignment using reinforcement learning techniques to fully harness its potential advantages. Note that the real-time assignment is applied after the pre-assignment, considering the deviations between the actual and original flight schedules as the disturbance. This study models a gate assignment problem as a Markov decision process (MDP). We consider that an airport has a series of nodes 𝐼= {1,2,…}. Each node is regarded as a gate at the airport for aircraft to park temporarily and for passenger alighting and boarding. The set of gates 𝐼 is classified into 𝐼𝑏 and 𝐼𝑎, where 𝐼𝑏 is the subset of nodes in 𝐼 that are Bridge Gates and 𝐼𝑎 are the Apron gates (Remote gates). A bridge gate is a gate that is directly connected to the terminal with a bridge, while a remote gate needs a shuttle bus to transfer passengers between the terminal and the aircraft. We define 𝐹= {1,2,…} as the set of flights. The optimization problem here is to ensure that every flight in 𝐹 has a gate in 𝐼 while minimizing the total cost. Section 3.1 presents the key assumptions, followed by demonstrating the state transition process in the MDP (Section 3.2) and the objective functions (Section 3.4). Finally, Section 3.5 formulates the constraints imposed on the decision variables. 3.1. Key assumptions The following assumptions are considered in this work: Assumption 1: Similar type of flights occupy the gate with a similar amount of time, and the aircraft will be towed away or ready for departure. Aircraft will not occupy the gate just for waiting purposes. The occupation time of the gate consists of several parts, as shown in Fig. 1: (a) Taxiing/towing time. (b) Ground service. (c) Passenger alighting and boarding. (d) Idle time. Assumption 2: We assume that the capacity of the airport apron is infinite. When there is no vacant bridge gate, the aircraft will be assigned to the Apron. Assumption 3: Considering that the gate assignment process occurs subsequently to obtaining the flight schedule, we assume that all flights can successfully land without encountering any additional infrastructure complications. Limited runway capacity, or priorities between airlines, for example, are not considered. 3.2. State formulation The RL algorithm has a discrete state space. Given the infrastructure and settings, the GAP is regarded as an MDP in which each stage 𝑛 refers to the flights in 𝐹. Each stage 𝑛 is characterized by the corresponding tuple of state variables 𝑠𝑛, and influenced by the decision variables 𝑥𝑛. Herein, we define the states tuple: 𝑠𝑛= (𝑓𝑛, 𝜏𝑛).(1) Each state 𝑛 contains the arrival times 𝑓𝑛, which is determined in pre-assignment scenario and dynamically updated in real-time assignment scenario and the gate size of the required flight (medium or large), 𝜏𝑛. The gate sizes considered are as follows: •𝜏𝑛= 2: medium size, can accommodate small or medium aircraft. •𝜏𝑛= 3: large size, can accommodate small, medium, or large aircraft. Note that a larger gate can also be used by smaller aircraft. The distinction between small, medium, and large aircraft is clarified in following Section 3.3. Finally, note that additional information could be used to hopefully provide more information on future perturbations. However, it was decided to keep this simplified state formulation for the following reasons: •Ensuring method agnosticism with respect to airport layout allows for versatile application across various airports. •Environmental factors, such as weather conditions, are deliberately excluded within this approach. The intention is not to predict disturbances but, instead, to react to them within the decision-making framework of the GAP at the airport. •With this state formulation, the algorithm utilizes identical inputs to those of conventional methods employed as baselines. This alignment facilitates direct comparisons, •Future flight schedules contain information for the complete day, which warrants a large state space. Increasing this state would further considerably increase the training time of the algorithms. 3.3. Action formulation There is a set of discrete actions, where each one represents a gate that the agent may select. For each state, the number of possible actions corresponds to the number of available gates. Consequently, this RL model can be run for different airport configurations, with the only modification being the number of gates. Each action, 𝑥𝑛, is characterized as a tuple, and on top of the gate number, it has multiple attributes which represent the limitations of the gate: 𝑥𝑛= (𝐼𝑛, 𝜆𝑛, 𝜂𝑛, 𝜔𝑛, 𝜌𝑛).(2) 𝐼𝑛 represents the gate number, which is the actual element that agent chooses, other elements serve as constraints. 𝜆𝑛 is the compatible aircraft size for this gate: •𝜆𝑛= 1: small aircraft (i.e. under 100 seats). •𝜆𝑛= 2: medium aircraft (i.e. between 100 and 200 seats). •𝜆𝑛= 3: large aircraft (i.e. more than 200 seats). Operations Research Perspectives 14 (2025) 100338 4 H. Li et al. Fig. 1. The several phases of the gate assignment process. Fig. 2. State transition. 𝜂𝑛 is the occupation time for this gate. Finally, 𝜔𝑛 is the earliest available time, and it is determined as follows: 𝜔𝑛= 𝑓𝐼𝑛+𝜂𝐼𝑛(3) where  𝑓𝐼𝑛 is the flight arrival time the last time Gate 𝐼𝑛 was chosen. 𝜂𝐼𝑛 is the gate occupation time when last time Gate 𝐼𝑛 is selected. 𝜌𝑛 is a binary indicator showing whether this gate is an Apron gate. The different settings of 𝑥𝑛 influence the overall performance of the gate assignment. It was chosen to define all possible actions in the form of 𝑥𝑛, instead of just the gate identification, as to give more information to the RL model to make its decision. The terminal condition of this MDP is when all the inbound flights get their gate assignment command. Fig. 2 demonstrates the state transition process, characterized by the state transition function 𝑝(𝑠𝑛, 𝑥𝑛). 3.4. Objective formulation Given the current circumstances and the requirements of the airport ground operations, a multi-objective approach is preferred. Based on the performance selection method [44], (1) the number of the remote gate and (2) passenger walking time are applied as the objectives in this paper. These objectives are then combined into a single reward that is given to the RL agent. The weight of two objectives are determined according to the priority lined out in [44]. The number of remote gates cost, 𝐿, is denoted as: 𝐿= 𝑓 ∑ 𝑛=1 𝜌𝑛(4) We interpret Passenger walking time as an average man walking time taken from the security check to the boarding gate, denoted in Eq. (5). 𝑡𝑝𝑤𝑡,𝐼𝑛= 𝑑𝑝𝑤𝑡,𝐼𝑛 𝑣,(5) where 𝑡(𝑝𝑤𝑡,𝐼𝑛) is the passenger’s walking time from the security check to the chosen Boarding Gate 𝐼𝑛 at State 𝑛. Parameter 𝑑(𝑝𝑤𝑡,𝐼𝑛) is the distance from security check to the chosen Boarding gate 𝐼𝑛 at State 𝑛. In turn, 𝑣 is the average human walking velocity (i.e. 5.43 km/h). The objective is to minimize the total PWT, denoted in Eq. (6). 𝑇𝑝𝑤𝑡 = 𝑓 ∑ 𝑛=1 𝑡𝑝𝑤𝑡,𝐼𝑛(6) With the two cost components mentioned above, the total cost function is demonstrated as follows: 𝐶=𝛼𝐿 + (1 − 𝛼)𝑇𝑝𝑤𝑡,(7) where 𝛼 is a coefficient that balances the two objectives’ trade-offs. We assume 𝛼= 0.8, as we come up with this number upon agreement with airport operators. The overall objective function for the GAP is then defined: 𝑚𝑖𝑛[𝐶],(8) 3.5. Constraints GAP is subjected to a set of regulatory constraints. Constraint (9) demonstrates the selected gate must be one of the functioning gates: 𝐼𝑛∈𝐼(9) Note that functioning means that the gate is working and can be used. Constraint (10) shows the lower (upper) bounds of the gate occupation time (𝜂), which is different according to Assumption 1: 𝜂𝑚𝑖𝑛 ≤𝜂𝑛≤𝜂𝑚𝑎𝑥 (10) Constraint (11) indicates the type of the gate: 𝜌𝑛={1if 𝐼𝑁∈𝐼𝑎,∀𝑛= 1,2,3,… 0otherwise (11) where 𝐼𝑎 indicates Apron gates, and 𝐼𝑏 Bridge gates. There are also a set of operational constraints that must be complied with. Constraint Operations Research Perspectives 14 (2025) 100338 5 H. Li et al. (12) defines that the selected gate must be available when the flight is arriving: 𝜔𝑛≤𝑓𝑛,(12) where 𝜔𝑛 represents the earliest available time of the gate, and 𝑓𝑛 the arrival time of the flight. Finally, constraint (13) sets that the gate size, 𝜏𝑛, must be compatible with the aircraft size, 𝜆𝑛: 𝜆𝑛≤𝜏𝑛.(13) 4. Deep reinforcement learning model DRL is a machine learning scheme where an agent interacts with the environment over a series of time steps. At each state, the agent performs an action according to the policy. The environment then changes according to the action. A reward is given to the agent — this reward evaluates how good the environmental change was. The RL model uses this reward to evaluate how good the performed action was. The long-term reward is the accumulative immediate reward when the environment reaches its terminal state. RL aims to maximize longterm reward over episodes of the training process. In other words, RL attempts to find the actions that yield better rewards. Section 4.1 defines the components of the developed RL model: state, actions, reward, and action shaping. Section 4.2 defines the RL algorithm employed in the work, the A3C. Finally, Section 4.3 defines how the RL model is used to produce a new real-time gate assignment based on delays of incoming aircraft. 4.1. Incorporation of components The vital components of the RL model that are used in this paper are presented here: •State: Defined in Section 3.2 as 𝑠𝑛= (𝑓𝑛, 𝜏𝑛). The size of the state space in length equals the total flights. •Action: Defined in Section 3.3 as 𝑥𝑛= (𝐼𝑛, 𝜆𝑛, 𝜂𝑛, 𝜔𝑛, 𝜌𝑛). The size of the action space equals the number of available gates. •Reward: Since reinforcement learning relies on the accumulation of immediate rewards, the total cost 𝐶 must be decomposed. The total cost comprises two components: the cost associated with assigning flights to remote gates and the walking time cost from security to the boarding gate. These can be interpreted as the costs corresponding to two types of actions: assigning a flight to a remote gate or to a bridge gate. To reflect this, we designed two mutually exclusive terms in the immediate reward function: 𝑟𝑛=𝛼𝜌𝑛+ (1 − 𝛼)(1 − 𝜌𝑛)𝑡𝑝𝑤𝑡,𝐼𝑛(14) where 𝑟𝑛 represents immediate reward at state 𝑛, 𝜌𝑛 and necessary information needed in 𝑡𝑝𝑤𝑡,𝐼𝑛 can be acquired in 𝑥𝑛. In Eq. (14), only one of the two terms is applied depending on 𝜌𝑛 in 𝑥𝑛. While traditional reinforcement learning (RL) focuses on solving a maximization problem, the Gate Assignment Problem (GAP) is framed as a minimization problem. Assigning flights to apron gates is most always undesirable; therefore, the first half of the reward is taken the opposite. A negative reward signals to the agent that this action is unfavorable. For actions involving bridge gates, which traditionally result in positive rewards, we invert the term to ensure the reward remains positive while prioritizing lower walking times. This adjustment ensures that a lower walking time corresponds to a higher reward. The final immediate reward function is defined as follows, incorporating 𝛼 to regulate the relative importance of the two terms. 𝑟𝑛= −𝛼𝜌𝑛+ (1 − 𝛼)(1 − 𝜌𝑛)𝑡−1 𝑝𝑤𝑡,𝐼𝑛(15) •Action Shaping [45–47]: this technique is used to train the agent to choose better actions. There will be various gates that are unavailable or unsuitable to choose from at each state. The chance of selecting these faulty gates should be zero. We employ an action mask to filter out all the impossible actions at each stage, forcing the agent only to choose legal actions, considerably improving learning efficiency. The action mask is implemented through a vector: Mask𝑎(𝑆𝑡𝑎𝑡𝑒) = [mask𝑎,1(𝑠𝑡𝑎𝑡𝑒),…,mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒)],(16) where mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒) is defined as follows: mask𝑎,𝑛(𝑠𝑡𝑎𝑡𝑒) = {1if 𝐼𝑁∈𝐼𝑎𝑣𝑎,∀𝑛= 1,2,3,… 0otherwise (17) where 𝐼𝑎𝑣𝑎 represents the subset of gates that are currently functional, available, and have a size matching the aircraft and suffice all the other constraints. Thus, the mask will only enable the RL agent to pick gates that are functional and match the size of the flight and meet other requirements. Finally, the action mask design is adaptable and can be easily customized to different settings and cases. We may set the probabilities of invalid actions to zero by adding an extra layer before the action probability calculation in the Actor network. 4.2. Asynchronous Advantage Actor Critic (A3C) In general, the DRL algorithm has two categories: Value-based and Policy-Based. The value-based approach is a deterministic policy, in which when the model is optimized to the best, every state’s corresponding action is determined, while Policy-based is stochastic. In a policy-based algorithm, every state/action series in each episode is denoted as Trajectory (𝜏): 𝜏= {𝑠1, 𝑥1, 𝑠2, 𝑥2,…, 𝑠𝑛, 𝑥𝑛}(18) The probability of this trajectory is: 𝑝𝜃(𝜏) = 𝑝(𝑠1)𝜋𝜃(𝑥1|𝑠1)𝑝(𝑠2|𝑠1, 𝑥1)𝜋𝜃(𝑥2|𝑠2)... =𝑝(𝑠1)𝛱𝑁 𝑛=1𝜋𝜃(𝑥𝑛|𝑠𝑛)𝑝(𝑠𝑛+1|𝑠𝑛, 𝑥𝑛)(19) The total reward of this trajectory is: 𝑅(𝜏) = 𝑁 ∑ 𝑛=1 𝑟𝑛(20) The goal is to find the best policy 𝜋𝜃, that can optimize the total reward 𝑅(𝜏). However, it is difficult to find a satisfying policy. To solve this drawback, the Actor-Critic (AC) framework has been put forward. In the AC algorithm, the agent is comprised of an actor and a critic. The actor acts under the current state using the current policy, then the environment will switch to a new state and return the immediate reward to the critic. The critic will judge the action based on the reward and return it to the actor for updating and calibrating the policy. A3C is proposed based on the AC algorithm, where the difference is that A3C employs multiple agents working concurrently, and asynchronously trains the Artificial Neural Network (ANN), therefore accelerating the training process while enhancing the convergence significantly. Here, we present the working process of one agent. Similar to AC algorithm, The A3C algorithm has a policy function 𝜋𝜃(𝑥𝑛|𝑠𝑛), and a value function 𝑉𝜃𝑣(𝑠𝑛),where 𝜃, 𝜃𝑣 are weights parameters. Both policy and value functions are updated after certain steps of action. The value of the discounted reward, 𝐷𝑅(𝜏), tells the agent which actions are rewarding and which are not. 𝐷𝑅(𝜏) = 𝑟1+𝛾𝑟2+𝛾2𝑅3... = ∞ ∑ 𝑛=1 𝛾𝑛−1𝑟𝑛(21) Operations Research Perspectives 14 (2025) 100338 6 H. Li et al. where 𝛾(0 < 𝛾 < 1) is the discount factor. The greater discount factor the more the agent is more focused on the future reward. Thus, the value function is defined as: 𝑉𝜃𝑣(𝑆𝑛) = 𝐸[𝐷𝑅(𝜏)|𝑠=𝑠𝑛, 𝜋] = 𝐸[ ∞ ∑ 𝑛=1 𝛾𝑛−1𝑟𝑛|𝑠=𝑠𝑛, 𝜋](22) Rather than use one-step Time-Difference-error(TD-error) in AC, we use n-step TD-error to better update the parameters. The update performed can be seen as: ∇𝜃𝐽(𝜃) = ∇𝜃𝑙𝑜𝑔𝜋𝜃(𝑥𝑛|𝑠𝑛)𝐴𝜃,𝜃𝑣(𝑠𝑛, 𝑥𝑛)(23) 𝐴𝜃,𝜃𝑣(𝑠𝑛, 𝑥𝑛) = 𝑄(𝑠𝑛, 𝑥𝑛) − 𝑉𝜃𝑣(𝑠𝑛)(24) where 𝑄(𝑠𝑛, 𝑥𝑛) = ∑𝑁 𝑛=1 𝛾𝑛−1𝑟𝑛+𝛾𝑛𝑉𝜃𝑣(𝑠𝑛+1). To encourage exploration and avoid premature convergence, an entropy item is adopted. The Actor network parameter update is performed based on the following accumulated gradient: 𝑑𝜃 ←𝑑𝜃 + ∇𝜃′𝑙𝑜𝑔𝜋𝜃′(𝑥𝑛|𝑠𝑛)𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛) + 𝛽∇𝜃′𝐻(𝜋𝜃′(𝑠𝑛)) (25) where 𝐻(𝜋𝜃′(𝑠𝑛)) is the entropy, 𝛽 controls the relative importance of the entropy and the reward. The Critic-network parameter update is performed based on the following accumulated gradient: 𝑑𝜃𝑣←𝑑𝜃𝑣+ 𝜕(𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛))2 𝜕𝜃′ 𝑣 (26) Finally, the proposed A3C-based Gate assignment algorithm is demonstrated in Algorithm 1 [29]. Algorithm 1 A3C-based Gate Assignment Algorithm 1: Initialization: 2: Initialize global actor-network and critic network parameters with 𝜃 and 𝜃𝑣 3: Set global Episode Counter T=0 and thread episode counter t=0 4: Initialize thread actor-network and critic network parameters with 𝜃′ and 𝜃′ 𝑣 5: Initialize 𝑇𝑚𝑎𝑥, 𝑡𝑚𝑎𝑥, 𝑁, 𝛾, 𝛽 6: DO: 7: Set t = 1 8: Reset 𝑑𝜃 = 0 and 𝑑𝜃𝑣= 0 9: while 𝑇 < 𝑇𝑚𝑎𝑥 do 10: Retrieve parameters from the global network: 𝜃′=𝜃, 𝜃′ 𝑣=𝜃𝑣 11: Obtain state 𝑠𝑛 12: while 𝑡 < 𝑡𝑚𝑎𝑥 do 13: Choose action 𝑥𝑛 under the policy 𝜋𝜃(𝑥𝑛|𝑠𝑛) 14: Get reward 𝑟𝑛 and new state 𝑠𝑛+1 15: 𝑡=𝑡+ 1 16: end while 17: 𝑄={0, 𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙𝑠𝑡𝑎𝑡𝑒 𝑉𝜃′(𝑠𝑛), 𝑛𝑜𝑛 −𝑡𝑒𝑟𝑚𝑖𝑛𝑎𝑙𝑠𝑡𝑎𝑡𝑒 (27) 18: while 𝑡=𝑡𝑚𝑎𝑥 do 19: 𝑄=𝑟𝑛+𝛾𝑄 20: Update thread actor-network parameter: 𝑑𝜃 ←𝑑𝜃 + ∇𝜃′𝑙𝑜𝑔𝜋𝜃′(𝑥𝑛|𝑠𝑛)𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛) + 𝛽∇𝜃′𝐻(𝜋𝜃′(𝑠𝑛)) (28) 21: Update thread critic-network parameter: 𝑑𝜃𝑣←𝑑𝜃𝑣+ 𝜕(𝐴𝜃′,𝜃𝑣(𝑠𝑛, 𝑥𝑛))2 𝜕𝜃′ 𝑣 (29) 22: end while 23: Perform asynchronous update 𝜃 and 𝜃𝑣 using 𝑑𝜃 and 𝑑𝜃𝑣 24: T= T+1 25: end while The global framework of the A3C algorithm is illustrated in Fig. 3. The agent is comprised of an actor and a critic. The actor acts under the current state using the current policy. Then the environment will switch to a new state and return the immediate reward to the critic. The critic will judge the action based on the reward and return it to the actor for updating and calibrating the policy. A3C is proposed based on the Actor-Critic algorithm, where the difference is that A3C employs multiple agents working concurrently and asynchronously training the ANN, accelerating the training process while enhancing the convergence significantly. 4.3. Real-time Gate Assignment Problem Solution (REGAPS) With the successful implementation of the A3C-GAP algorithm, the Gate Pre-assignment Problem has been solved. However, the preassignment schedule always deviates from the actual schedule due to multiple causes, such as weather conditions and regulations. Hence, the proposed REGAPS algorithm diagram is demonstrated in Fig. 4. At Time Zero, which marks the beginning of the day, the model initiates the execution of an algorithm that treats gate assignment as a preassignment problem, relying on the original flight schedule. Under normal circumstances, the operator proceeds with gate assignments per the predetermined schedule when no delays are encountered. However, upon detecting a delay (at Reschedule point T, Fig. 4), the model promptly updates the flight schedule and re-executes the algorithm using the revised input, thus obtaining an updated assignment. The figure labeled as Fig. 5 provides a detailed explanation of how flight schedules are updated. Note that in this paper, we assume that once the gate occupation time (which includes boarding time) is completed, the aircraft will either be ready for takeoff or moved to the apron in the event of a delay, as Assumption 1. Therefore, departure delays do not affect gate usage in our current model. It shows that when delays or changes in schedules are identified, only the arrival times of affected flights are adjusted, while the assigned gates and arrived flights remain the same. This iterative process ensures the attainment of real-time gate assignments, facilitating their completion in response to changing conditions. To solve the real-time assignment problem, the initialization should be set as the new timetable of the arriving flights, and the gate occupation states should be set simultaneously. Also, several buffers are applied to store the best action, best reward, state information, and gate occupation status. 5. Hypotheses The following hypotheses are established regarding how the REGAPS algorithm can improve gate assignment over conventional methods: Hypothesis 1: The REGAPS algorithm is poised to surpass other methods by strategically considering the sequence of actions over time. Note that the REGAPS algorithm does not possess additional information compared to other methods, and does not predict disturbances. However, we hypothesize that its unique capacity lies in learning from past experiences. The REGAPS algorithm can discern which decisions result in more favorable outcomes. For instance, it can learn which gate allocation patterns lead to reduced Apron usage by day’s end. Hypothesis 2: The REGAPS algorithm will prioritize reducing Apron gate usage, as this value was made paramount in the reward formulation (Section 3.4). Hypothesis 3: The REGAPS will improve upon conventional method in environments with higher complexity and uncertainties. In environments with lower complexity, it is expected that conventional methods already perform well and the introduction of REGAPS may be not represent and improvement. Operations Research Perspectives 14 (2025) 100338 7 H. Li et al. Fig. 3. The A3C framework. Fig. 4. Diagram of the proposed REGAPS. Operations Research Perspectives 14 (2025) 100338 8 H. Li et al. [43] Jang H-C, Huang Y-C, Chiu H-A. A study on the effectiveness of A2C and A3C reinforcement learning in parking space search in Urban Areas Problem. In: 2020 international conference on information and communication technology convergence. ICTC, 2020, p. 567–71. [44] Li H, Wu X, Liang Y, Zhang C. A multistakeholder approach to the airport gate assignment problem: Application of fuzzy theory for optimal performance indicator selection. Comput Intell Neurosci 2021;2021. Cited by: 0; All Open Access, Gold Open Access, Green Open Access. [45] Huang S, Ontañón S. A closer look at invalid action masking in policy gradient algorithms. In: The international FLAIRS conference proceedings. Vol. 35, University of Florida George A Smathers Libraries; 2022. [46] Kanervisto A, Scheller C, Hautamäki V. Action space shaping in deep reinforcement learning. In: 2020 IEEE conference on games (coG). 2020, p. 479–86. [47] Müller A, Sabatelli M. Safe and psychologically pleasant traffic signal control with reinforcement learning using action masking. In: 2022 IEEE 25th international conference on intelligent transportation systems. ITSC, 2022, p. 951–8. [48] Klar M, Glatt M, Aurich JC. Performance comparison of reinforcement learning and metaheuristics for factory layout planning. CIRP J Manuf Sci Technol 2023;45:10–25. [49] CAAC. Technical guidelines for airport epidemic prevention and control. 8th ed.. 2021. [50] p-christ. Deep-Reinforcement-Learning-Algorithms-with-PyTorch: PyTorch implementations of deep reinforcement learning algorithms and environments. 2024, https://github.com/p-christ/Deep-Reinforcement-Learning-Algorithmswith-PyTorch. [Accessed 03 October 2024]. [51] zhangzhengmin. GA for Gate assignment. 2024, https://github.com/ zhangzhengmin/–GA. [Accessed 17 December 2024]. [52] Hu X-B, Di Paolo E. An efficient genetic algorithm with uniform crossover for the multi-objective airport gate assignment problem. In: Multi-objective memetic algorithms. Berlin, Heidelberg: Springer Berlin Heidelberg; 2009, p. 71–89. [53] Ding C, Bi J, Wang Y. A hybrid genetic algorithm based on imitation learning for the airport gate assignment problem. Entropy 2023;25(4). Operations Research Perspectives 14 (2025) 100338 15