Variational model-based Deep Reinforcement Learning for Non-Homogeneous Patrolling aquatic environments with multiple unmanned surface vehicles

Contents lists available at ScienceDirect Expert Systems With Applications journal homepage: www.elsevier.com/locate/eswa Variational model-based Deep Reinforcement Learning for Non-Homogeneous Patrolling aquatic environments with multiple unmanned surface vehicles Samuel Yanes Luis a,∗, Nicola Basilico b, Michele Antonazzi b, Daniel Gutiérrez-Reina a, Sergio Toral Marín a aDepartment of Electronics Engineering, University of Sevilla, Camino Avenida de Los Descubrimientos s/n, Sevilla, 41005, Spain bDepartment of Computer Science, University of Milan, Via Celoria 18, Milano, 20133, Italy ARTICLE INFO Keywords: Deep Reinforcement Learning Environmental patrolling Multi-agent path planning Model-based decision making ABSTRACT This paper addresses the challenge of Non-Homogeneous Patrolling for Autonomous Surface Vehicles in nonhomogeneous importance water environments with a dissimilar biological monitorization criterion. Traditional monitoring methods fail, especially in expansive areas such as Lake Ypacaraíin Paraguay. The proposed solution employs a cooperative Deep Reinforcement Learning framework, specifically a multi-agent version of the Double Deep Q-Learning algorithm based on safe-consensus decision making. This framework optimizes adaptive policies for such vehicles by simultaneously modeling the environment and patrolling high-importance zones. The incorporation of a Variational Auto-Encoder based on the U-Network architecture directly addresses the non-observability of the environment by predicting biological importance from partial observations. The methodology is validated in a realistic algae bloom contamination scenario, demonstrating superior performance and computational efficiency compared to traditional approaches like Gaussian Processes and K-Nearest-Neighbors. The Deep Reinforcement Learning framework, coupled with the Variational Auto-Encoder model, showcases flexibility and efficiency in addressing multi-agent cooperation and long-term objective optimization for water quality monitoring. The results reveal significant improvements, with the proposed model exceeding well-founded approaches with a 30% faster minimization of the patrolling score compared to these methods. 1. Introduction Lakes, rivers, and shores play a vital role in the ecosystems of the Earth and are critical components of our planet’s water cycle. The biological state of such resources is constantly changing due to various natural and human-caused factors: rainfall patterns, temperature fluctuations, and human activities such as agriculture and urbanization (Baron, Poff, Angermeier, et al.,2002). Monitoring water for pollutants and disease-causing organisms is essential to understand and manage the impact on health. However, the size of these resources makes it a challenging task. This has been a usual problem in places such as Mar Menor (Spain) or Lake Ypacaraí(Paraguay). Traditional manual monitoring methods are limited in their ability to cover large areas and collect comprehensive data on changing conditions of water resources (Arzamendia, Gutierrez et al.,2019). The use of Autonomous Surface Vehicles (ASVs) for monitoring offers a unique opportunity to model water resources in real time with high resolution (SánchezGarcía et al.,2018). With the use of advanced water quality sensors ∗Corresponding author. E-mail address: [email protected] (S. Yanes Luis). and an appropriate coordinated monitoring policy, ASVs can continuously collect data on various physical and chemical properties of water, providing a more comprehensive understanding of the risks these environments suffer. However, effective monitoring of water resources using ASVs requires the optimization of these adaptive policies that take into account the unique challenges of real biophysical environments. These policies must decide sequentially where to take samples of the water quality parameters (WQP), at the same time considering different objectives: (i) obstacle avoidance, (ii) efficient coordination between vehicles, and (iii) integration of data gathered by multiple vehicles into a comprehensive picture of the water resource conditions. The use of multiple agents also has the problem of scalability, since its complexity increases with the number of independent mobile measurement stations, represented by the ASVs. By far, the solution to cooperative information gathering with multiple agents remains an open challenge that must be addressed https://doi.org/10.1016/j.eswa.2025.126483 Received 29 March 2024; Received in revised form 24 October 2024; Accepted 7 January 2025 Expert Systems With Applications 270 (2025) 126483 Available online 13 January 2025 0957-4174/© 2025 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license ( http://creativecommons.org/licenses/bync-nd/4.0/ ). S. Yanes Luis et al. from multiple perspectives. In the particular case of water quality monitoring, this task required to be continuous. Persistently monitoring an environment involves revisiting zones in which the information could become obsolete or there is a higher risk of biological danger. This problem has been commonly formulated as the Patrolling problem (Chevaleyre,2004;Yanes, Reina, & Marín,2020), which is a particular case of Path Planning problems. The Patrolling problem consists of continuously sampling the zones of higher interest with a cyclic temporal criterion (Chevaleyre,2004). In general, those zones that have been unvisited the longest are the most important zones to cover. Among environmental applications, such as the problem of monitoring water quality, seen in Yanes et al. (2020) or in the surveillance of wildfires (Julian & Kochenderfer,2018), it is convenient to define a dissimilar importance between zones for monitoring idleness. In these scenarios, there are zones of higher biological importance that need to be covered more persistently, such as zones of high-risk biological activity in algae blooms or clusters of high turbidity in waters. This problem can be formulated as the Non-Homogeneous Patrolling Problem (NHPP). NHPP is defined as the sequential optimization problem of finding the best route 𝜓∶= [𝑝0,…, 𝑝𝑇]that covers the maximum information possible 𝜔at the same time it maximizes the coverage of zones that have remained unvisited for longer 𝑤. An important aspect addressed in this article is the imposition that the importance criterion 𝜔is not known a priori and is discovered as the fleet explores the navigation space. A poor model of importance distribution implies inefficient patrolling as the important zones remain unknown and therefore unvisited for longer. This aspect configures the problem into a Partially Observable Markov Decision Process (POMDP), where the fleet of agents has limited vision of the surroundings. This indicates that agents face the dual challenge of continuously modeling a changing environment while patrolling critical zones. In the context of a non-observable NHPP, when information importance is unknown in advance, it is proven that optimal solutions via off-line path planners are infeasible. Therefore, two modules are required: one to address the environment’s non-stationarity and another to develop an adaptive policy that can respond to these changes. Patrolling also involves cooperation between agents to avoid measurement redundancies and to share the navigation space. This paper proposes to use Deep Reinforcement Learning (DRL) as a methodology to obtain dynamic policies for the ASVs to infer the importance model 𝜔and to patrol efficiently. Previous approaches that used DRL (Yanes et al.,2020;Yanes, Reina, Marín & Toral,2021;Yanes Luis, GutiérrezReina, & Toral Marín,2023) for WQP patrolling have shown promising results in generating optimized tailored patrolling policies, allowing ASVs to make decisions on the fly based on real-time interactions with the environment. Nevertheless, there is a clear uncovered aspect of these methods when modeling and patrolling must be performed at the same time, moreover, when it involves multiple vehicles. In this sense, this paper proposes a multi-agent version of the Double Deep Q-Learning algorithm (DDQL) from the DRL methodology (Mnih, Kavukcuoglu, Silver, et al.,2015), to deal with different water quality monitoring tasks with this particular partial observability condition. In DDQL, a deep neural network 𝑄(𝑠, 𝑎)is defined to represent the future discounted reward given an observation 𝑠and a possible action 𝑎∈𝐴for the agent to choose. This network will be trained in a model-free manner, using only simulated interactions of the agents with the environment. The original approach from Yanes, GutiérrezReina, and Toral Marín (2021) is modified to train multiple agents using one single 𝑄function. For multiple cooperative agents, a unique function 𝑄is defined. This function receives an egocentric observation 𝑜𝑗, where every agent 𝑗differentiates itself from other agents visually. With regard to the training scenario, a realistic water contamination simulator is proposed, based on the dynamics of the sparse algae blooms (Qiu, Ren, Li, Tao, & Zhou,2021). This scenario generator introduces stochasticity to obtain infinite possible contamination cases Fig. 1. Diagram of the proposed framework. subjected to the natural limits of the algae phenomenon. Deep Q-Learning and other reinforcement learning methods base their estimation on observations of the environment. As mentioned above, without proper observation, it is harder to infer the optimal actions for agents to move (Yanes et al.,2020). To address importance modeling, an enhanced model is proposed online based on a Deep Variational Auto-encoder, to infer the biologic importance map 𝜔given partial observations like in an Inverse Problem Estimation (Yi, Guo, Fan, Hamann, & Wang,2020). The architecture of this module is made up of a variant encoder–decoder based on the popular UNet network (Ronneberger, Fischer, & Brox,2015). Our architecture implements a variational version of this UNet similar to Wang et al. (2020) with the novelty of a Gaussian operation in the channel for stochasticity generation. The input of the model is visually constructed by all the measurements taken, which results in a constant size input, constant size inference time, much more efficient than other classical approaches like the Gaussian process (GPs). See Fig. 1for a complete overview of the proposed framework. In summary, this paper proposes the following: •A realistic scenario of algae bloom contamination for water resources based on a stochastic physical model. •A Deep Reinforcement Learning framework to obtain Deep patrolling policies by means of an appropriate NHPP formulation, reward, and state. •A Variational Auto-Encoder model to enhance the observability of the agents and to predict the model of the environment with partial observations. This paper is organized as follows. In Section 2previous approaches are discussed. In Section 3, the NHPP is presented and the movement and the final optimization objective is explained. In Section 4, the DRL methodology is presented within the network architectures. The importance model based on the VAE-UNet is also presented in Section 4. In Section 5, we present and discuss the simulations and the results. Finally, in Section 6, the conclusions and future work are explained. 2. Related works The use of autonomous surface vehicles has gained relevance in recent years due to advances in battery autonomy and, above all, to the capacity for remote computing and sensing (Sánchez-García et al., 2018). These vehicles are particularly convenient because they can be used to obtain a status of water resource quality combined with good Expert Systems With Applications 270 (2025) 126483 2 S. Yanes Luis et al. GNSS localization capability, maneuverability, and autonomy. Those vehicles dedicated to biological conservation are usually equipped with modules such as water quality sensors, bathymetry, and spectral cameras, for environmental data acquisition. These sensors will define the observation capability in monitoring tasks and observability within the patrol optimization problem. Multiple previous works have addressed the issue of acquiring environmental water quality information. Depending on the final objective and conditions of the patrol, we can highlight different types of path planning: (I) When the objective is to find an accurate model of one or several parameters of water quality, we usually refer to Informative Path Planning (IPP) (Popovic, Vidal-Calleja, Hitz, et al.,2020). (II) When planning requires continuous monitoring due to the need for temporal monitoring, we refer to Informative Patrolling (Yanes et al., 2020). In Peralta, Reina, Marín, Gregor, and Arzamendia (2021) the use of Bayesian optimization and Gaussian processes (GP) is proposed for optimal sampling with different agents. The search space for each vehicle is equally distributed using a Voronoi tessellation. For the policy, the Expected Improvement acquisition function with regularization based on the sample distance is used. This limits the vehicles in the travel distance before sampling. With this formulation, the aim is to obtain the highest precision with a minimum number of samples. However, no cyclic criterion could be easily incorporated into this framework, which is absolutely necessary for the patrolling task. It can be seen how the Particle Swarm Optimization (PSO) method is used to find the maximum contamination levels (Jara Ten Kathen, Peralta, Johnson, Jurado Flores, & Gutiérrez Reina,2024). In this work, each vehicle behaves as a particle in a continuous search space. In Kathen, Flores, and Reina (2021), a modification of the classic PSO algorithm is proposed that encourages exploration using a predictive uncertainty of GP as a surrogate model. The main problem with applying these methods for patrolling is that they are not intended for continuous monitoring. The optimization of Gaussian processes has (𝑁3)complexity with the number of samples, which is not ideal for large amounts of information from patrolling paths. Moreover, GPs involve imposing how information correlates by choosing a kernel and hyperparameter boundaries (Kathen, Peralta, Johnson, Jurado Flores, & Gutiérrez Reina,2023). In our proposal, the use of Deep policies trained by a DRL kernel allows the behavior to be optimized without any previous heuristic in a model-free scheme. In addition, the VAE mode will exhibit a much more efficient prediction with less inference time, as it incorporates qualified information from the environment during offline training. Regarding the use of DRL with patrolling missions, some previous works have explored this possibility (Piciarelli & Foresti,2019; Yanes et al.,2020;Yanes, Reina et al.,2021;Yanes Luis et al.,2023). In Piciarelli and Foresti (2019), the use of flying autonomous vehicles is proposed for the first time for the resolution of nonhomogeneous patrolling. This work posits the use of spectral cameras as a form of information acquisition with a single agent. However, the adopted scenario does not appear to be based on any real phenomena, is tailored for only one agent, and the navigation space is assumed to be convex. This condition allows traveling from one point to another in space without obstacles, and all actions are valid. In this paper, a mechanism is implemented to avoid risky or directly forbidden actions, as was done in Yanes Luis et al. (2023). This mechanism has been shown to allow for more homogeneous and direct learning, since the avoidance of known obstacles is no longer part of learning. The use of DDQL for NHPP patrolling is also discussed in Yanes, Gutiérrez-Reina et al. (2021). The conclusions of this work allow us to determine that DDQL is capable of optimizing large state–action policies for this task. In addition, a detailed study of the sensitivity of DDQL hyperparameters is carried out in Yanes et al. (2020). Finally, in Yanes, Reina et al. (2021), a methodology for the multi-agent case is proposed. Nevertheless, the agents have all the information of the problem a priori, in a fully-observable NHPP, which could be seen as unrealistic. There, it is proposed to use a neural network whose last layer has an output for each agent. In contrast, a shared weight policy is proposed as in Yanes, Reina et al. (2021), however, with this architecture, it is not necessary to modify its size depending on the agents. Instead, it incorporates an egocentric observation for each agent, in which we separate the observation of the agent that observes from the other agents that are observed. Other works related to patrolling put the focus on pure fleet-movement coordination, such as Sun, Li, Chen, Dong, and Wang (2024). In this work, a related subtask of patrolling is addressed for cooperative defense using ASVs. Behavioral cloning is proposed to enhance the performance of a DRL-based policy. This idea explores the fact that there are very competitive algorithms that DRL can learn from. The DRL has been used in other tasks beyond patrolling. It is common to find that reinforcement learning algorithms are used to design motion controllers for obstacle avoidance or tracking. An example of DRL application in surface vehicles is in Song, Gan, Yao, Zang, and Qu (2023). In this work, the use of an Actor Critic algorithm is proposed to solve the tracking problem. Here, the use of a realistic model of the environment is proposed: dynamic model of waves, inertias and associated hydrodynamic phenomena. However, the work limits its scope to low-level target tracking, without any model of environment perception or exogenous variables (biological or human). Other works, such as Sawada, Sato, and Majima (2021), propose a similar task but incorporating LSTM recurrent networks in the algorithm. This is a particularly interesting mechanism when, in the absence of an explicit dynamic model, past state information needs to be incorporated in decision making. This work realistically models the obstacles in the environment using proximity sensors and uses this LSTM mechanism to implicitly estimate how the obstacles and the rest of the fleet will move. In the latter two works, continuous actions have been used because the control task imposes continuous control signals. To reduce the dimension of the decision space, in Zhang, Wang, Bi, and Huang (2024), it is proposed to use a Soft-Actor Critic algorithm with 11 discrete motion actions for the control of a water vehicle. In this work the model of the environment is static, and obstacle avoidance is again delegated to the policy through implicit modeling of the environment. However, the work proposes a hierarchical training mechanism to separate the tasks of obstacle detection, use of tides to promote motion, and tracking. With respect to the environment information model, we explore the use of reconstruction techniques based on variational encoders (Kohl et al.,2019;Yi et al.,2020). In Kohl et al. (2019), a VAE network is proposed for image segmentation tasks, which translates a visual input into semantic maps using a stochastic kernel operator. This work also proposes processing the prior and posterior distributions of visual input using independent CNNs. In Yi et al. (2020), a VAE framework is proposed for the reconstruction of partially occluded solar radiation images. We make use of the proposed loss functions proposed in Yi et al. (2020) and the prior/posterior architecture in Kohl et al. (2019) but with variations to adapt it to our particular application process. These kind of techniques enable the possibility to obtain a deep regressor able to invert the dynamics of the hidden information. This is different from approaches such as Kathen et al. (2021) and Peralta et al. (2021) where the model has no other knowledge of the environment rather than the samples taken on every mission. 3. Problem statement In this section, we discuss the Non-Homogeneous Patrolling Problem (NHPP). We also enumerate the assumptions taken in the particular case of the biological monitorization of Lake Ypacaraí, which will serve as our reference real-world scenario. In the Static Non-Homogeneous Patrolling Problem (NHPP), the environment is modeled as a grid map, represented by a connected graph 𝐺= (𝑉 , 𝐸 , 𝜔). Each node 𝑣∈𝑉corresponds to a location a Expert Systems With Applications 270 (2025) 126483 3 S. Yanes Luis et al. robot can reach, while each edge (𝑢, 𝑣) ∈𝐸represents an unobstructed and shortest path between adjacent locations. Node adjacency is defined under the assumption that the grid is 8-connected. Thus, 𝐺is formally depicted as an 8-connected grid graph, with holes indicating the presence of obstacles. In our reference scenario, the nodes represent areas within the navigable waters where robots can travel to collect measurements. Time is treated as discrete in this model, and moving along any edge of the graph – whether horizontal, vertical, or diagonal – requires 1temporal unit. The function 𝜔∶𝑉→R≥0assigns to each node its information gain, namely a value that, in the scope of the global monitoring task, quantifies the importance of a measurement taken in the area associated with the node. The higher 𝜔(𝑣)the more valuable a measurement gathered at 𝑣. Clearly, 𝜔captures a key aspect for the efficiency of the monitoring task, contingent upon the specific environment in which such a task is carried out. This function is commonly employed to characterize the spatial biological interest across the area to be surveyed. For instance, in Kathen et al. (2021) and Yanes et al. (2020), this relevance is expressed through a contamination index within the range of [0,1], which denotes the level of anomaly in water quality. In this work, we define the information gain for a node 𝑣as 𝜔(𝑣) =𝜔0(𝑣) +𝜔1(𝑣). The component 𝜔0(𝑣)is independent of the measurement taken at node 𝑣. Essentially, 𝜔0represents the intrinsic significance of 𝑣, stemming solely from its presence in the environment. The other component, 𝜔1(𝑣), is directly proportional to the informational value at 𝑣and thus varies based on the measurement a robot acquires there. Notably, 𝜔1 is initially unknown and can only be determined by having a robot performing a measurement at the related location. Being related to real measurements taken in the environment, this value typically exhibits some degree of spatial correlation. Given a set of agents 𝐽= {1,2,…, 𝑚}, 𝑚≥1, tasked with deployment in the environment for monitoring purposes. Addressing the NHPP involves determining a collection of paths 𝜓= {𝜓1, 𝜓2,…, 𝜓𝑚}, assigning one distinct path to each ASV. In particular, the path for robot 𝑗is a sequence of nodes 𝜓𝑗= (𝑣1, 𝑣2,…, 𝑣𝑇)that the agent is scheduled to visit, where each consecutive pair of nodes (𝑣𝑖, 𝑣𝑖+1)is an element of the edge set 𝐸. This implies that an agent 𝑗conducts a measurement at every node it visits along its path, specifically at time step 𝑡it will perform a measurement at node 𝜓𝑗(𝑡). Given the practical constraints of real-world monitoring applications, it is imperative that the entire monitoring mission concludes within a predefined time limit 𝑇. When a measurement is taken at node 𝑣, a robot uncovers 𝜔1(𝑣)and designates a group of nodes around 𝑣, within a certain maximum distance, as monitored at the current time. This set of nodes is referred to as the neighborhood of 𝑣, defined as (𝑣) = {𝑞∈𝑉∣𝑑(𝑣, 𝑞)≤𝜅}. Here, 𝑑(𝑣, 𝑞) represents the shortest time required to traverse from 𝑣to 𝑞across the graph 𝐺, and 𝜅is a parameter specifying the range. In this context, a candidate solution 𝜓yields the function 𝑀𝜓(𝑡) = {𝑣∈ ∪𝑗∈𝐽(𝜓𝑗(𝑡))}, indicating the set of nodes monitored by the team of robots at a given time 𝑡. Through this function, a solution 𝜓generates an idleness profile for each node in the environment. The idleness at a node, usually defined as the number of time steps since its last visit, can be interpreted as a constant cost 𝜏incurred for every time step the node remains unvisited. Formally, we denote the idleness of node 𝑣at time step 𝑡as 𝑤(𝑣, 𝑡), where 𝑤(𝑣, 0) = 0and 𝑤(𝑣, 𝑡+ 1) ={min{𝑊𝑀 𝐴𝑋 , 𝑤(𝑣, 𝑡) +𝜏}if 𝑣∉𝑀𝜓(𝑡) 0otherwise.(1) where 𝑊𝑀 𝐴𝑋 denotes the maximum idleness expected in a node, usually 1 if the idleness is min–max normalized as in Yanes et al. (2020). An optimal solution to NHPP can be defined as a set of paths 𝜋∗ that minimizes an environment-cumulative weighted idleness over the mission horizon 𝑇. That is, 𝜋∗= ar g min 𝜋∑ 𝑣∈𝑉 𝑤(𝑣, 𝑇)𝜔(𝑣)(2) 4. Methodology In this Section, we will explain the multi-agent Reinforcement Learning framework and the proposed algorithm to optimize the fleet policy. We will also address the Variational Deep scheme and how it is trained to invert the observations into the complete state of the environment. 4.1. DDQL framework Deep Reinforcement Learning algorithms use a neural network to represent a deep policy 𝜋(𝑠∣𝜃). The deep policy is trained by trial and error through the agent interaction with the environment. In DRL algorithms, the cornerstone of optimization is the generation of experiences < 𝑠𝑡, 𝑎𝑡, 𝑠𝑡+1, 𝑟𝑡>composed by the state 𝑠, actions 𝑎(which represents any feasible movements in the aforementioned graph 𝐺for an agent, and the generated rewards 𝑟that result from this movement. DRL algorithms seek to adjust the policy parameters to maximize the cumulative future reward 𝑅=∑𝑇 𝑡=0 𝛾𝑡𝑟𝑡over a control horizon 𝑇and given a discount factor 𝛾that balances the importance of short-term rewards. One of the most widely used algorithms in scenarios with discrete actions 𝑎∈𝐴 ⊆R|𝐴|, is Double Deep Q-Learning (DDQL). This algorithm, first proposed in Hasselt, Guez, and Silver (2016), attempts to estimate the state–action function 𝑄(𝑠, 𝑎), which represents the cumulative and discounted reward, given a state 𝑠and each action 𝑎: 𝑄(𝑠𝑡, 𝑎) = 𝑇 ∑ 𝑘=𝑡 𝛾 𝑟(𝑠𝑘, 𝑎)(3) The core of DQL lies in minimizing the Bellman error, which quantifies the discrepancy between the current estimation of the action-value function and a target value based on the Bellman equation. Given a transition (𝑠𝑡, 𝑎𝑡, 𝑟𝑡, 𝑠𝑡+1), the Bellman error is defined as follows: 𝛿=𝑟𝑡+𝛾⋅𝑄(𝑠𝑡+1,ar g max 𝑎′𝑄(𝑠𝑡+1, 𝑎′;𝜃); 𝜃−) −𝑄(𝑠𝑡, 𝑎𝑡;𝜃)(4) where 𝛾is the discount factor that weights the importance of future rewards, 𝑄(𝑠𝑡, 𝑎𝑡;𝜃)represents the current estimation of the Q-value for state 𝑠𝑡and action 𝑎𝑡using the online network with parameters 𝜃, max𝑎′𝑄(𝑠𝑡+1, 𝑎′;𝜃)represents the maximum Q-value over all possible actions in the next state 𝑠𝑡+1, and where 𝑄(𝑠𝑡+1,ar g max𝑎′𝑄(𝑠𝑡+1, 𝑎′;𝜃); 𝜃−) represents the Q-value of the action selected by the online network but evaluated using the target network parameters 𝜃−. Note that DDQL employs two separate neural networks: the online network and the target network. The target network is updated every 𝑁𝑢𝑝𝑑 𝑎𝑡𝑒 training step, which updates the targets to the already seen data. Once the optimal 𝑄-value function is reached, the optimal action is given by: 𝑎∗= ar g max 𝑎′𝑄(𝑠, 𝑎′)(5) 4.2. The 𝜖-greedy exploratory policy In order to explore the state–action space, it is important to take actions different from those indicated by 𝑄. An 𝜖-greedy policy is employed to achieve this balance between exploration and exploitation of 𝑄. At each time step, the agent selects the action with the highest Qvalue with a probability of (1 −𝜖)(exploitation), and with a probability of 𝜖, selects a random action (exploration). The value of 𝜖determines the degree of randomness in the policy and is annealed over the training time to gradually shift the agents’ behavior from exploration to exploitation as learning progresses. 4.3. Prioritized experience replay For this algorithm to converge, it is necessary to employ a replay memory buffer that stores the agent’s experiences during interactions Expert Systems With Applications 270 (2025) 126483 4 S. Yanes Luis et al. Fig. 2. Example of the observation of an agent 𝑗. with the environment. By randomly sampling mini-batches of experiences from the replay memory, we decorrelate the training data and mitigate potential issues caused by temporal correlations in the sequences of experiences. To improve the better learning of experiences with higher temporal errors 𝛿, we implement a Prioritized Experience Replay (PER) like in Schaul, Quan, Antonoglou, and Silver (2015). PER assigns priorities, denoted by 𝑝𝑖, to experiences based on their absolute Bellman errors 𝛿𝑖, which measure the significance of a transition. The priority 𝑝𝑖of experience 𝑖is computed as: 𝑝𝑖=|𝛿𝑖|(6) The higher the absolute Bellman error, the higher the priority, indicating that the experience is more informative for learning. 4.4. Safe DDQL for multiple agents Due to the priorities of Eq. (6), observations become interchangeable between agents and can be stored in the same experience buffer to indistinctly train the shared policy. The reward function, which is the same for all agents, also allows multiple agents to be trained with a single neural network for the same cooperative objective. The policy simply maps each agent’s egocentric observation into its 𝑄 values, and each agent takes the highest valued action independently. This approach is halfway between Independent Q-Learning where each agent has its own neural network and centralized Q-learning (Yanes et al.,2020), where agents’ actions are decided jointly with a single global observation. A direct benefit of this strategy is that the fleet size can change, and the policy can continue to be used. In addition, this architecture allows for the setting up of a serverless scheme, where each agent can carry the policy on its local hardware. Another aspect related to the problem scaling is that the neural network architecture does not change when the number of agents changes. Algorithm 1reports the proposed Multi-agent DDQL Algorithm. With respect to obstacle avoidance, the DRL is able to effectively learn the boundaries of the map (Yanes et al.,2020), the actions that generate an agent-scenario collision can be deterministically calculated (Yanes Luis et al.,2023). It is different with the collision between agents since, by deciding the actions simultaneously, two agents may decide to move to the same place. This is a particular NPhard subproblem of Multi-agent Path Finding (Stern et al.,2019). In Algorithm 1Safe Multi-agent Double Deep Q-Learning Algorithm 1: Initialize replay memory 𝐷to capacity |𝐷| 2: Initialize target Q-network 𝑄′with weights 𝜽′=𝜽 3: Initialize policy network 𝑄with weights 𝜽 4: for episode = 1to 𝐸𝑚𝑎𝑥 do 5: Reset environment 6: Get initial observation 𝑜0=(𝑠0) 7: for timestep = 1to 𝑇do 8: 𝑝∼𝑈(0,1) 9: if 𝑝 < 𝜖then 10: 𝒂𝑗←𝑆 𝑎𝑓 𝑒𝐶 𝑜𝑛𝑠𝑒𝑛𝑠𝑢𝑠(𝑈(0,1),…, 𝑈(0,1)) 11: else 12: 𝒂𝑗←𝑆 𝑎𝑓 𝑒𝐶 𝑜𝑛𝑠𝑒𝑛𝑠𝑢𝑠(𝑄(𝑜0, 𝑎),…, 𝑄(𝑜|𝐴|, 𝑎)) 13: end if 14: Execute action 𝒂𝑗 15: Observe rewards 𝑟𝑗and new observations 𝑜𝑡+1 𝑗 16: Store every transition (𝑜𝑡 𝑗,𝒂𝑡 𝑗, 𝑟𝑡 𝑗, 𝑜𝑡+1 𝑗)in 𝐷 17: Sample random batch 𝐵of (𝑜𝑗,𝒂𝑗, 𝑟𝑗, 𝑜𝑗+1)from 𝐷 18: Set 𝑦𝑗=𝑟𝑗+𝛾 𝑄′(𝑠𝑗+1,ar g max𝒂𝑄(𝑠𝑗+1,𝒂;𝜽); 𝜽′) 19: Update weights by minimizing the loss: (𝜽) =1 𝐵 𝐵 ∑ 𝑗=1 (𝑦𝑗−𝑄(𝑠𝑗,𝒂𝑗;𝜽))2 20: if 𝑚𝑜𝑑(𝑡, 𝑀) == 0then 21: 𝜃′←𝜃 ⊳Target weights update. 22: end if 23: end for 24: 𝜖←min(𝜖𝑚𝑖𝑛, 𝜖−𝑑 𝜖). 25: end for this paper, it is proposed to overcome these situations by means of a heuristic based on the conditional decision 𝑎𝑗=𝜋(𝑜𝑗|𝑎𝑗−)of an action taken by an agent 𝑎𝑗, depending on the actions of the other agents 𝑎𝑗−. First, we order the agents according to the largest estimated value of 𝑄. The agent with the highest value in 𝑄, which is under the assumption of good estimation the most promising action, follows the 𝜖-greedy policy free of fixed obstacles without taking into account the movements of the rest of the agents. The other agents also take an action considering the new position of the previous agent. Those values of 𝑄that place the agent in collision are deterministically censored with a value of −∞. A new action, already safe, is taken by that agent and the next agent does the same. When all actions have been decided, the movements are processed, and measurements are taken. In this way, this heuristic relies on the optimism of the agents to take precedence. In the event that a random action is taken, following the 𝜖-greedy policy, again only safe actions are considered and no collision is produced. We provide the pseudo-code of the whole Safe Dueling DQL algorithm in Algorithm 1 and the particular subroutine for consensus in Algorithm 2. 4.5. Neural network The shared policy is implemented by a neural network composed of an initial Convolutional Neural Network (CNN), a fully connected 3Layer Perceptron, and a final two-headed Dueling scheme (see Fig. 3). The CNN, formed by 3 convolutional layers with ReLU activations, extracts spatial features from the state 𝑜𝑡 𝑗, which is modeled as a singlechannel image (see Fig. 2). The observation channels correspond to: (a) the navigation map, the information model 𝜔, (c) the idleness map 𝑤(𝑣, 𝑡), (d) the position of the agent 𝑗, and (e) the position of the other agents 𝑗−. The resulting embedding is processed by 3 linear layers with ReLU and then fed into a Dueling Deep Q-Network (DQN) like in Wang, Freitas, and Lanctot (2015). A Dueling network augments the traditional DQN architecture with two streams for value estimation. The Expert Systems With Applications 270 (2025) 126483 5 S. Yanes Luis et al. Fig. 3. Dueling Deep Q-Network architecture with a convolutional encoder. Algorithm 2Safe Consensus Algorithm Require: Positions 𝑃𝑡=𝒑𝑡 1,𝒑𝑡 2,…,𝒑𝑡 𝑁of 𝑁agents at time 𝑡 Require: Values 𝑄= {𝑄1, 𝑄2,…, 𝑄|𝐴|}that weight each agent’s action. 1: Initialize empty set of future positions 𝑃𝑡+1 ∶= ∅ 2: Obtain order of agents’ actions in decreasing order of their 𝑄values: 𝑗1, 𝑗2,…, 𝑗𝑁, such that max 𝑄𝑗1≥max 𝑄𝑗2≥…≥max 𝑄𝑗𝑁. 3: for each agent 𝑗in order of actions do 4: Select greedy safe action 𝒂𝑗= ar g max 𝒂∈𝐴𝑄𝑗(𝑎) subjected to: ‖(𝒑𝑡 𝑗+𝒂𝑗) −𝒑′‖2≤𝑑𝑠𝑎𝑓 𝑒∀𝒑′∈𝑃𝑡+1 5: 𝑃𝑡+1 ←𝑃𝑡+1 ∪ (𝒑𝑡 𝑗+𝒂𝑗)⊳Update next fleet positions. 6: 𝐴𝑠𝑒𝑙 𝑒𝑐 𝑡𝑒𝑑 ←𝐴𝑠𝑒𝑙 𝑒𝑐 𝑡𝑒𝑑 ∪𝒂𝑗⊳Update consensus actions. 7: end for 8: return 𝐴𝑠𝑒𝑙 𝑒𝑐 𝑡𝑒𝑑 first stream computes the state value function 𝑉(𝑜𝑗), representing the expected cumulative reward given an observation in state 𝑜𝑗and following the current policy. The second stream calculates the action advantage function 𝐴(𝑜𝑗, 𝑎), which measures how much better taking action 𝑎given an observation 𝑜𝑗is compared to the average actionvalue in that state. The two steams are combined using the following equation: 𝑄(𝑜𝑡 𝑗, 𝑎) =𝑉(𝑜𝑡 𝑗) +𝐴(𝑜𝑡 𝑗, 𝑎) −1 |𝐴|∑ 𝑎′ 𝐴(𝑜𝑡 𝑗, 𝑎′)(7) This separation improves the representation and approximation of the 𝑄function and is particularly useful for tasks with large action spaces or complex environments, leading to more stable and accurate policy learning in DRL (Wang et al.,2015). The output of the layer will be, as stated in Section 5.2, the estimation of the 𝑄-values for each possible Cartesian movement in the graph 𝐺(𝑉 , 𝐸)for an ASV 𝑗in the current observed state 𝑂𝑡 𝑗, with respect to the future discounted reward. 4.6. Reward function The reward function is fundamental in all DRL algorithms (Sutton & Barto,2018). The reward must consider in a quantitative manner the optimality of an action in a particular state. In this paper, the reward will be individually evaluated for the exact action. The reward function will be designed using the definitions of optimality of the NHPP in Eq. (2). With each new sample, every agent will receive the sum of idleness 𝜔collected in its new position multiplied by the term of importance of the information given the importance model 𝜔. To address redundancy in the measurement, we define a redundancy factor 𝜌(𝑣)as the number of ASVs that cover a particular node 𝑣𝑗 in a particular instant. A node is considered covered if any agent is in such position 𝑣𝑖that, 𝑣𝑗is a neighbor 𝑣𝑗∈(𝑣𝑖). Thus, the reward associated with overlapping areas of different agents will be weighted depending on how many agents cover this particular position 𝜌(𝑣). Consequently, this will decrease the individual reward for taking measurements with multiple agents in the same place. The final reward for an action 𝑎𝑡 𝑗will be: 𝑟(𝑠𝑡, 𝑎𝑡 𝑗) =∑ 𝑣∈(𝑣) 𝑤(𝑣, 𝑡) ×𝜔(𝑣, 𝑡) 𝜌(𝑣)(8) 4.7. Information importance model The information model is composed of a convolutionaldeconvolutional neural network based on the UNet architecture (Ronneberger et al.,2015). This architecture is composed of a fully convolutional neural network (CNN) that combines both contracting and expanding paths. The contracting path consists of 4 convolutional and max-pooling layers. This process enables the network to learn highlevel features from the entire input image. Following this, there is the deconvolutional path, which consists of 4 convolutional and upsampling layers, which increases the spatial resolution of the feature maps. In the central part of the U, the contraction and expansion paths are connected through residual connections. The input of the network consists of two channels: i) a pre-processed model of importance 𝑌𝑡 constructed by a composition of the samples taken by the agents so far in the locations in which they were taken, and (ii) a binary visit mask 𝑀𝑡, with those places that have been sampled represented by a 1, and 0 when this place has not been sampled yet. This input will be queried from the UNet architecture to produce an estimate of the importance model at that time 𝜔𝑡. In this paper, this original architecture is expanded to implement a variational version of the UNet, as it was used in Yi et al. (2020) for image reconstruction. Two distinct convolutional networks are implemented, one to estimate the prior probabilistic Gaussian distribution (𝜇 , 𝜎), and the other to estimate the posterior distribution (𝜇 , 𝜎), with 𝜇 , 𝜎∈R𝑁. The prior network (used in inference) receives partial observations 𝑌𝑡and is trained to minimize the Kullback–Leibler (KL) distance with respect to the output of the posterior network that also receives the ground truth of the information importance 𝜔. With this probabilistic distribution, we generate a set of convolutional filters, each sampled from every distribution of (𝜇 , 𝜎). The resulting stochastic filters are added to the final output layer of the UNet. The entire architecture is trained by minimizing the loss composed of a weighted sum of the following terms: •Reconstruction loss: the Mean Squared Error (MSE) between the generated model and the real ground truth. This term is related to the capacity of the network to invert the observation and provide an estimate of the complete map: L𝑟𝑒𝑐 𝑜𝑛𝑠 =𝑀 𝑆 𝐸(𝜔, 𝜔)(9) •KL loss: The Kullback–Leibler divergence between the prior and posterior distributions. By minimizing this term, the network learns the inner distribution of the data. L𝐾 𝐿=𝐾 𝐿((𝜇 , 𝜎),(𝜇 , 𝜎))(10) Expert Systems With Applications 270 (2025) 126483 6 S. Yanes Luis et al. Fig. 4. VAE-UNet training architecture for training and inference. •Perceptual loss: The difference between the high-level feature map 𝜉of the output and the ground truth from a pre-trained deep convolutional model like 𝑉 𝐺 𝐺16 (Gatys, Ecker, & Bethge,2015). L𝑝𝑒𝑟𝑐 𝑒𝑝𝑡𝑢𝑎𝑙 =𝑀 𝑆 𝐸(𝜉(𝜔), 𝜉(𝜔))(11) The final loss will be: L=𝛽0L𝑟𝑒𝑐 𝑜𝑛𝑠 +𝛽1L𝐾 𝐿+𝛽2L𝑝𝑒𝑟𝑐 𝑒𝑝𝑡𝑢𝑎𝑙 (12) with 𝛽1and 𝛽2being parameters to weigh the importance of each term in the minimization of total loss. These weights must be chosen to minimize the final prediction error between the final prediction of importance and the ground truth. A complete diagram of the model and its submodules is shown in Fig. 4. 5. Training and results 5.1. Environment simulation In this section, we first present the training results of both the Deep Model and the Deep Policy. Secondly, we present the results of the training of the Deep Model and we analyze its performance compared to other regression methods. Third, we show the results of the DRL training and discuss different swarm behaviors. Finally, we compare our approach with other state-of-the-art approaches in the literature. All simulations and training have been carried out on a server running Ubuntu 20.04, equipped with an Intel Dual Xeon Gold 5220R CPU2.20 GHz, 192Gb of RAM and two GPUs: Nvidia Quadro A4000 48 GB and Nvidia RTX 3090 25 GB. All simulations were conducted with Lake Ypacaraías the baseline scenario with 4 vehicles, as it happened in previous approaches to monitoring contamination in large water resources (Yanes et al.,2020; Yanes, Reina et al.,2021;Yanes Luis et al.,2023). The movementrelated parameters (𝑑𝑚𝑎𝑥,𝑑𝑚𝑜𝑣𝑒 and 𝑁) has been selected based on Yanes et al. (2020). Their values has been selected by considering realistic values in terms of battery life of real ASV prototypes. The value of the forgetting factor 𝜏is chosen as a user parameter, which imposes the effective time in which the information is valid for the user. This parameter is extrinsic to the problem and can be modified according to the desire for more or less monitoring rate. Finally, the simulation Fig. 5. Two examples of WQP distributions with both benchmarks in the southern part of Lake Ypacaraí: (a) Shekel benchmark, and (b) Algae Bloom Benchmark. constant parameter 𝛥𝑡 is simply a value for the integration of the laws of motion of each moving element. It must be chosen small enough to avoid integration errors. Two different benchmarks are used to represent the information importance 𝜔. On the one hand, a Shekel benchmark function1is used, similarly to many other previous works on monitoring water quality (Kathen et al.,2021;Peralta et al.,2021). This standard WQPmodel function treats the contamination in a similar way to the data collected in the Mar Menor Data Server,2which is a particular case of contaminated water quality resource. The WQPs have a smooth distribution over the waters with peaks and valleys. On the other hand, we present an Algae Bloom simulator based on a simplified diffusion model of blue–green algae bacteria. Up to 6 random algae blooms can appear in any part of the water. We treat these algae bloom bacteria as particles with random speeds 𝑣𝑟for each one, to model the diffusion effect of contaminants on the surface of the waters. In addition, two speeds related to wind speed 𝑣𝑤and water currents 𝑣𝑐are introduced. These three components are weighted to compose the final speed. The position of every particle is updated depending on these speeds by computing the discrete integral with a fixed time step 𝛥𝑡. Finally, to model the effects of the shores, the physical boundaries 1https://www.sfu.ca/~ssurjano/shekel.html 2https://marmenor.upct.es/maps/Transparency Expert Systems With Applications 270 (2025) 126483 7 S. Yanes Luis et al. Table 1 Summary of the simulation and benchmark parameters. Parameter Value Number of agents (𝑁)4 Number of actions |𝐴|8 Max. distance (𝑑𝑚𝑎𝑥) 32 𝑘𝑚 Movement distance (𝑑𝑚𝑜𝑣𝑒) 160 𝑚 Forgetting factor (𝜏)0.01 Simulator time constant (𝛥𝑡)0.05 Table 2 Summary of the training parameters of the Deep VAE. Parameter Value Learning rate (𝑙 𝑟) 1 ⋅10−3 Batch Size 64 Max. Epochs 30 Discount factor (𝛾)0.99 Training dataset size (Shekel) 3000 ground truths Training dataset size (Algae Bloom) 3000 ground truths Validation dataset size (Algae Bloom) 100 ground truths Validation dataset size (Algae Bloom) 100 ground truths Fig. 6. Example of two inputs for the VAE-UNet taken in a random timestep for both benchmarks. In the left, the first channel with the measurement mask. In the middle, the measured values in their positions. In the right, the respective ground truths. of the navigable zones exert a pushing-back force to the particles. The Algae Bloom simulations will be termed static when the blooms are simulated for a random period and then considered, in general terms, stationary (see Tables 1and 2). 5.2. Importance model training In order to train the VAE Importance Model, a training set of 3000 functions from both the WQP simulator and from the static Algae Bloom simulator are generated. In addition, a validation set and a test set of 100 ground truths of each are also simulated for performance comparison (see Fig. 5for an example of both benchmarks). This data is generated with different random seeds and different starting conditions such as initial Algae Blooms, maxima-minima positions, currents, and wind conditions. Consequently, no single ground truth sampled will be found simultaneously in the training and validation set. An example of the input and ground truth for both benchmarks can be seen in Fig. 6. Thus, each ground truth is paired with the random path generated for each vehicle using a Non-Redundant Path Planner (NRPP), as explained in Yanes Luis et al. (2023). This path planner consists of a random coverage heuristic used in exploratory scenarios. Each vehicle selects an obstacle-free trajectory, and when the shore is reached, another random direction is selected. This random path selection is used to reduce the bias of the Importance Model with respect to the type of path planner used in the application. As path planning also plays a role in the quality of the estimation, it is important to decouple the Table 3 Best obtained loss weights and its ANOVA significance for the hyperparametrization of both ground truths. Parameter Best ANOVA signf. Algae Bloom benchmark Learning Rate 2.8 × 10−3 0.71 Recons. Loss (𝛽0) 6.402 0.08 KL Loss (𝛽1) 4.561 0.16 Percep. Loss (𝛽2) 4.18 0.03 WQP benchmark Learning Rate 1.4 × 10−3 0.54 Reconstruction Loss (𝛽0) 3.42 0.29 KL Loss (𝛽1) 1.84 0.08 Perceptual Loss (𝛽2) 4.23 0.08 importance inference task from the patrolling task to avoid cross bias. The VAE is trained for 50 epochs on each benchmark. The weight loss (𝛽0, 𝛽1, 𝛽2)from Eq. (12) are optimized using a Tree-structured Parzen Estimator (TPE) sampler (Bergstra, Bardenet, Bengio, & Kégl, 2011), by minimizing the Mean Squared Error (MSE) of the predictions over the validation datasets. The most significant term for both benchmarks is the reconstruction loss. In Table 3we present all the hyperparameters related to the training of the model and the estimated importance given its Analysis of Variance (ANOVA) test. For performance evaluation, we will analyze 3 metrics: (I) the Root Mean Squared Error (RMSE) that will measure the average residuals between the estimated model and the ground truth at each timestep. (II) A weighted RMSE that will ponder the error of every zone with its importance, so the error in highly important will weight more. (III) The cost of computing every method in seconds to evaluate the aptitude of the model to work in a time-constrained application. We compare our method with other approaches for 2D regression: 1. Myopic naive model: This is a naive model used as a baseline. This model is updated only in the vicinity 𝜅of every measurement with the most updated information. 2. Radial k-Nearest Neighbors (R-kNN): A 2D k-Nearest Neighbors algorithm. Every position in 𝜔 is updated with the nearest sampled value within a radius 𝑟𝐾 𝑁 𝑁. 3. Gaussian Process (GP): Taken directly from Peralta et al. (2021). This model uses a Radial Basis Kernel to model the correlation between samples. The model output will be directly the mean of the GP. In Fig. 7a-c the RMSE and the weighted RMSE are represented with respect to the number of measurement steps. Each step involves one measurement per vehicle. It can be seen that the proposed model is able to predict the real importance map with higher accuracy in all cases. In the WQP benchmark, with enough distance budget, both the GPs and VAE-UNet algorithms are able to find low errors. However, the proposed methodology can reduce the error earlier in this first scenario by a 28% and a 30% in the first third and second third, respectively, with respect to the second-best method (GP) (see Table 5 and Fig. 7a). In the Algae Bloom scenario, the improvement is higher, with a 35% and 29% in the first and two thirds (see Table 4and Fig. 7b). This improved performance has been observed to be related to the lack of smoothness in algae blooms. The GPs seem to overfit the hyperparameters of the RBF kernel to the low-bandwidth features of the blooms. The VAE algorithm is able to deal, at the same time and more efficiently, with smoothness predictions in valley zones and noisy highimportance areas. The main benefit of the proposed algorithm with respect to the patrolling task, inferred from the results in Fig. 7, is the faster convergence of the proposed algorithm. With battery restrictions, a faster convergence in the model provides the path planner with a better understanding of hidden 𝜔to deal with intensification much more efficiently. To ensure that these results are statistically significant, a Wilcoxon Ranked test (Wilcoxon,1945) has been used to validate the Expert Systems With Applications 270 (2025) 126483 8 S. Yanes Luis et al. Fig. 7. Comparison of the average RMSE and Weighted RMSE between different regression models with 100 sampled ground truths from (a) WQP function, (b) static Algae Bloom benchmark, and (c) dynamic Algae Bloom benchmark simulator. Table 4 Average RMSE and its Confidence Interval values for each model in the Algae Bloom benchmark. Error 33% Error 66% Error 100% 𝜇CI (95%) 𝜇CI (95%) 𝜇CI (95%) rKNN 0.117 ±0.012 0.086 ±0.009 0.071 ±0.008 Myopic 0.137 ±0.012 0.115 ±0.011 0.102 ±0.011 GP 0.107 ±0.013 0.078 ±0.010 0.065 ±0.008 VAE-UNet 0.077 ±0.011 0.055 ±0.007 0.047 ±0.005 Table 5 Average RMSE and its Confidence Interval values for each model in the WQP benchmark. Error 33% Error 66% Error 100% 𝜇CI (95%) 𝜇CI (95%) 𝜇CI (95%) rKNN 0.163 ±0.017 0.102 ±0.012 0.065 ±0.010 Myopic 0.175 ±0.017 0.120 ±0.012 0.084 ±0.010 GP 0.056 ±0.010 0.024 ±0.003 0.015 ±0.002 VAE-UNet 0.036 ±0.007 0.017 ±0.001 0.013 ±0.001 Table 6 Average RMSE and its Confidence Interval values for each model in the dynamic Algae Bloom benchmark. Error 33% Error 66% Error 100% 𝜇CI (95%) 𝜇CI (95%) 𝜇CI (95%) rKNN 0.125 ±0.012 0.100 ±0.009 0.092 ±0.007 Myopic 0.144 ±0.012 0.124 ±0.011 0.113 ±0.009 GP 0.115 ±0.012 0.091 ±0.009 0.087 ±0.008 VAE-UNet 0.086 ±0.009 0.072 ±0.006 0.076 ±0.005 comparisons between the proposed method and others with 𝑝 <0.05. When introducing dynamics in the Algae Bloom simulation, the proposed VAE architecture is still able to give better predictions even when the training dataset is static. Since in this scenario, new measurement substitute old ones in the visual input, the VAE network can still process estimations without much loss of generalization. It is observed a 20% improvement (with a𝑝 <0.05 according to the Wilcoxon Ranked test), over 100 episodes across the measurement campaign (see Fig. 7(b)) with respect to the GPs. It is seen that GPs suffer from loss of generalization when two samples too close are acquired in different times. The other algorithms, although they can come up with a comprehensive image of the contamination at the end of a exploratory mission, cannot capture the realistic image of this benchmark (see Table 6). Fig. 8. Inference times for each model under test. In Fig. 8it can be observed the probabilistic distribution of the different computation times for every model under test. The computation times are represented there for 100 different multi-agent paths and for the three benchmarks. It can be observed that the second-best algorithm on average in terms of the importance model error needs one order of magnitude more time per computation than the VAE-UNet proposed algorithm. The computation time is similar for the rKNN algorithm, with a 10% slower time. The faster approach is obviously the Myopic approach, which obtains the worst performance in the model accuracy. This result can be interpreted as the metric robustness of the proposed information model to be used in real-time scenarios with more strict time-critic requirements. It is important to note that the computation time of the VAE-UNet is constant regardless of the number of samples 𝑀. This is not true for the rKNN benchmark and (especially) for the GPs, which suffer from a complexity of (𝑀). Then, the computation times are expected to increase significantly with larger paths in a continuous monitorization task. This fact supports the decision to use alternative methods such as the one proposed here with VAE-UNet for the continuous patrolling task, provided that there is some kind of information or prior on how the information to be monitored behaves. 5.3. Deep policy training Once the VAE-UNet model is trained, each information model is embedded in the DRL framework for training. Two different trainings are conducted for the two benchmarks, with a total training budget of 20.000 episodes each. During training, every mission is conducted in a completely new sample of the benchmark, not used in the model Expert Systems With Applications 270 (2025) 126483 9