Variational model-based Deep Reinforcement Learning for Non-Homogeneous Patrolling aquatic environments with multiple unmanned surface vehicles

Abstract

This paper addresses the challenge of Non-Homogeneous Patrolling for Autonomous Surface Vehicles in non-homogeneous importance water environments with a dissimilar biological monitorization criterion. Traditional monitoring methods fail, especially in expansive areas such as Lake Ypacaraíin Paraguay. The proposed solution employs a cooperative Deep Reinforcement Learning framework, specifically a multi-agent version of the Double Deep Q-Learning algorithm based on safe-consensus decision making. This framework optimizes adaptive policies for such vehicles by simultaneously modeling the environment and patrolling high-importance zones. The incorporation of a Variational Auto-Encoder based on the U-Network architecture directly addresses the non-observability of the environment by predicting biological importance from partial observations. The methodology is validated in a realistic algae bloom contamination scenario, demonstrating superior performance and computational efficiency compared to traditional approaches like Gaussian Processes and K-Nearest-Neighbors. The Deep Reinforcement Learning framework, coupled with the Variational Auto-Encoder model, showcases flexibility and efficiency in addressing multi-agent cooperation and long-term objective optimization for water quality monitoring. The results reveal significant improvements, with the proposed model exceeding well-founded approaches with a 30% faster minimization of the patrolling score compared to these methods.