Testing Evolutionary and Reinforcement Learning Approaches to Traffic Flow Optimization in SUMO

Testing Evolutionary and Reinforcement Learning Approaches to Traffic Flow Optimization in SUMO Dominic Domingo1,†, Aryan Bandi2,†, Arya Kunisetty3,†, Ahan Banerjee4,†, George Flint5,*, Kevin Zhu6,* 1Mater Dei Catholic High School, San Diego, CA, USA 2Westlake High School, Austin, TX, USA 3American High School, Fremont, CA, USA 4Thomas Jefferson High School for Science and Technology, Alexandria, VA, USA 5University of California, Berkeley, Berkeley, CA, USA 6Algoverse Academy, Los Angeles, CA, USA † denotes equal contribution to the work. * denotes senior authorship. Abstract Urban road systems require intricate planning to ensure safe and efficient transportation. Effective road systems help reduce traffic congestion, maximize throughput, and minimize collisions. Previous applications of machine learning algorithms to this topic have largely been focused on prediction, not optimization. In this work, we train and evaluate evolutionary and reinforcement learning models on this optimization problem by interfacing with the Simulation of Urban Mobility (SUMO) package. SUMO offers a framework to model and evaluate traffic dynamics, allowing users to configure parameters to explore the effect of various configurations on evaluation metrics. Models set values for traffic signal timings, speed limits, and designated lane access across a road network. They are evaluated on metrics for vehicle throughput, total waiting time, total travel time, speed variation, and crash frequency. We observe performance improvements compared to configurations that estimate the corresponding real-world networks, indicating evolutionary and reinforcement learning approaches might be well suited for this task, despite sparse application thus far. Introduction Traffic congestion is a significant challenge for urban areas worldwide, leading to longer travel times, higher fuel consumption, increased emissions, and elevated stress levels among commuters. The repercussions go beyond mere inconvenience; congestion also contributes to economic losses and environmental harm [2]. While many studies examine individual factors influencing traffic congestion, such as Traffic Signal Control (TSC) or Vehicle Routing Problem (VRP), they overlook the interaction between multiple factors at once, at scale [1, 6]. We used an evolutionary algorithm for broad optimization and detailed reinforcement learning to cover all aspects of traffic flow optimization. Our primary approach to this problem is the application of an evolutionary algorithm. Our evolutionary algorithms are efficient models to solve complex problems [3], iteratively evolving configurations based Copyright © 2025, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. on performance metrics. In our experiment, the evolutionary algorithm optimized factors such as speed limits and traffic signal timings. The simulation was then run with these configurations, and evaluated with a set of metrics measuring efficiency and safety. In this work, we offer a more extensive optimization and analysis by applying evolutionary and reinforcement algorithms to configure parameters influencing traffic flow in the Simulation of Urban Mobility (SUMO; 10) and assess their performance. Methodology Overview This study employs two computational approaches to optimize traffic configurations: an evolutionary algorithm, and an actor-critic method [7]. The generation algorithm generates diverse solutions helping us generate various solutions in this highly dynamic system. On the contrary, the actorcritic approach uses reinforcement learning to generate policies by continuously learning based on feedback from the environment, ensuring real-time responsiveness to changing traffic conditions and gaining a general sense of positive and negative changes. A snapshot of each of 10 cities was exported from OpenStreetMaps, an open-source geospatial database. These networks are complete with speed limits and traffic signals as shown in the real world, ensuring the baseline simulation is accurate to real life. These files are then converted into a simulation-ready format using the SUMO netconvert tool. The simulations were configured to have 1000 vehicles total with randomly generated routes. The vehicles would incrementally spawn throughout the 1800-step (30 sim-minutes) simulation, ensuring that there is a constant flow of traffic at all times. Simulations were run on the following cities due to their high population sizes, heavily congested traffic, and different styles of road architecture such as grid and concentric models. For our simulations, we utilize data from Los Angeles, San Francisco, and Boston in the United States; Rio de Janeiro, Brazil; Lisbon, Portugal; Monaco; Mumbai, India; Manila, Philippines; Johannesburg, South Africa; Shanghai, China to analyze global urban trends and patterns. Parameters The following SUMO parameters were configured before running the simulations: •Speed limits. These values were randomly changed from 15 to 50 miles per hour in multiples of 5. Balancing speed limits is essential to maintaining the proper flow of traffic. The value must be high enough to keep up with the travel volume demands at peak traffic conditions, while also being low enough to minimize safety risk. •Traffic signals. Timings were randomized from 10 to 60 seconds per signal light cycle. While low values lower the idle time of vehicles, it also could lead to choppy starting/stopping due to uncoordinated lights. •Lane accessibility. Lane restrictions were added, causing certain lanes to have carpool or high occupancy vehicle access only. This allows for some passengers to have a ”fast lane”, which could potentially lower travel time and waiting time for some people, although if it is underutilized it will worsen traffic congestion. Evaluation Metrics The following evaluation metrics were used to evaluate the performance of the configurations: •Total travel time: The average time taken per vehicle to complete its intended route. •Total waiting time The total amount of time each vehicle is currently idle at a traffic intersection. •Vehicle throughput: How many vehicles can enter or exit the system; this was divided by 1800 to contextualize it by seconds. •Speed variation: Speed variation represents how much each car changes its speed relative to the maximum speed of the lane. Evolutionary Algorithm Approach We use an evolutionary algorithm to optimize traffic configurations within the SUMO simulation environment. The genetic algorithm starts with the creation of an initial population of 50. Within these populations are chromosomes which are comprised of configurations for speed changes, traffic light duration, and HOV lane creation. This function ensures diversity in the population by randomly selecting edge IDs and traffic light IDs from the SUMO network configuration. Afterwards, the next generation is created by selecting the best-performing configurations, combining some configurations, and adding new chromosomes to others. Fitness evaluation is performed with the fitness function which assesses how well each chromosome performs relative to a baseline simulation. The fitness score is calculated by averaging the percent change difference for each metric compared to the baseline. Figure 1: Normalized evaluation metrics (average fitness, average waiting time, average speed variations, and vehicle throughput). Preliminary Experiments We are currently working on implementing the actor-critic approach, but we have run some initial experiments with the evolutionary algorithm approach, and we have found that city networks improve by as much as 16% compared to the baseline map without any changes. We expect results to improve even more with hyperparameter tuning. Results Evaluating the evolutionary algorithms on Shanghai, Lisbon, and Boston, we found that speed variations initially skyrocketed, but as the generations went by the metric returned, resulting in a small increase. Travel time, waiting time, and throughput experienced large amounts of variability due to the mutations of each chromosome, but converged towards a decrease in overall value in later generations. Vehicle Throughput showed a steady declining pattern in Boston and Lisbon, but increased in Shanghai. In the future, we will evaluate and compare the actor-critic model with these results. We will also experiment with different values of mutation and combination of chromosomes within the genetic algorithm to maximize the output and exploration of the solution space. References [1] L. Da, C. Chu, W. Zhang, and H. Wei, “CityFlowER: An Efficient and Realistic Traffic Simulator with Embedded Machine Learning Models,” arXiv preprint, arXiv:2402.06127, 2024. [Online]. Available: https:// arxiv.org/abs/2402.06127 [2] M. A. Fattah, S. R. Morshed, and A.-A. Kafy, “Insights into the socio-economic impacts of traffic congestion in the port and industrial areas of Chittagong city, Bangladesh,” Transportation Engineering, vol. 9, p. 100122, 2022. doi: 10.1016/j.treng.2022.100122 [3] B. Galv´ an, D. Greiner, J. P´ eriaux, M. Sefrioui, and G. Winter, “Parallel Evolutionary Computation for Solving Complex CFD Optimization Problems: A Review and Some Nozzle Applications,” in Parallel Computational Fluid Dynamics: New Frontiers and MultiDisciplinary Applications. Elsevier, 2003, pp. 573– 604. doi: 10.1016/B978-044450680-1/50072-3 [4] J. D. Wiegand and C. Y. D. Yang, “Traffic Simulation Runs: How Many Needed?,” Public Roads, vol. 74, no. 4, pp. 30–35, 2011. [Online]. Available: https://www.fhwa.dot.gov/publications/publicroads/ 11janfeb/04.cfm [5] K. Kasaura, S. Miura, T. Kozuno, R. Yonetani, K. Hoshino, and Y. Hosoe, “Benchmarking ActorCritic Deep Reinforcement Learning Algorithms for Robotics Control with Action Constraints,” IEEE Robotics and Automation Letters, vol. 8, no. 8, pp. 4449–4456, 2023. doi: 10.1109/LRA.2023.3284378 [6] D. Kikuta, H. Ikeuchi, K. Tajiri, and Y. Nakano, “RouteExplainer: An Explanation Framework for Vehicle Routing Problem,” arXiv preprint, arXiv:2403.03585, 2024. [Online]. Available: https://arxiv.org/abs/2403.03585 [7] V. R. Konda and J. N. Tsitsiklis, “Actor-Critic Algorithms,” in Advances in Neural Information Processing Systems 12, 2000, pp. 1008–1014. [8] Federal Highway Administration, “Step 3 – Detailed System Planning and Design,” 2014. [Online]. Available: https://ops.fhwa.dot.gov/publications/ fhwahop14008/step3.htm [Accessed: 3-Jul-2025]. [9] Federal Highway Administration, Traffic Congestion and Reliability: Trends and Advanced Strategies for Congestion Mitigation, U.S. Department of Transportation, Tech. Rep. FHWA-HOP-05-065, 2005. [Online]. Available: https://ops.fhwa.dot.gov/congestion report/ [10] P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Fl¨ otter¨ od, R. Hilbrich, L. L¨ ucken, J. Rummel, P. Wagner, and E. Wießner, “Microscopic Traffic Simulation using SUMO,” in The 21st IEEE International Conference on Intelligent Transportation Systems, 2018. [Online]. Available: https://elib.dlr.de/ 124092/