Decoupling Patrolling Tasks for Water Quality Monitoring: A Multi-Agent Deep Reinforcement Learning Approach

Received 15 April 2024, accepted 13 May 2024, date of publication 21 May 2024, date of current version 4 June 2024. Digital Object Identifier 10.1109/ACCESS.2024.3403790 Decoupling Patrolling Tasks for Water Quality Monitoring: A Multi-Agent Deep Reinforcement Learning Approach DAME SECK DIOP , SAMUEL YANES LUIS , MANUEL PERALES ESTEVE , SERGIO L. TORAL MARÍN , AND DANIEL GUTIÉRREZ REINA Department of Electronic Engineering, Technical School of Engineering of Seville, 41092 Seville, Spain Corresponding author: Dame Seck Diop ([email protected]) This work was partially funded by the following projects: TED2021-131326B-C21 of the Spanish Ministry of Science MCIN/AEI/10.13039/501100011033, and the European Union with ‘‘NextGenerationEU/PRTR’’ funds. ABSTRACT This study proposes the use of an Autonomous Surface Vehicle (ASV) fleet with water quality sensors for efficient patrolling to monitor water resource pollution. This is formulated as a Patrolling Problem, which consists of planning and executing efficient routes to continuously monitor a given area. When patrolling Lake Ypacaraí with ASVs, the scenario transforms into a Partially Observable Markov Game (POMG) due to unknown pollution levels. Given the computational complexity, a Multi-Agent Deep Reinforcement Learning (MADRL) approach is adopted, with a common policy for homogeneous agents. A consensus algorithm assists in collision avoidance and coordination. The work introduces exploration and reinforcement phases to the patrolling problem. The Exploration Phase aims at homogeneous map coverage, while the Intensification Phase prioritizes high polluted areas. The innovative introduction of a transition variable, ν, efficiently controls the transition from exploration to intensification. Results demonstrate the superiority of the method, which outperforms a Single-Phase (trained on a single task) Deep Q-Network (DQN) by an average of 17% on the intensification task. The proposed multitask learning approach with parameter sharing, coupled with DQN training, outperforms Task-Specific DQN (two DQNs trained on separate tasks) by 6% in exploration and 13% in intensification. It also outperforms the heuristic-based Lawn Mower Path Planner (LMPP) and Random Wanderer Path Planner (RWPP) algorithms, by 35% and 20% on average respectively. Additionally, it outperforms a Particle Swarm Optimization-based Path Planner (PSOPP) by an average of 26%. The algorithm demonstrates adaptability in unforeseen scenarios, giving users flexibility in configuration. INDEX TERMS Patrollingproblem,multi-agent deepreinforcement learning,multi-task deepreinforcement learning environmental monitoring, exploration and intensification phases, partially observable Markov games, autonomous surface vehicles. I. INTRODUCTION Water resources, such as lakes and ponds, are vital to life, providing habitat for wildlife and a source of drinking water. Lake Ypacaraí in Paraguay, which spans over 60 km2, The associate editor coordinating the review of this manuscript and approving it for publication was Jianxiang Xi . is facing increasing pollution due to waste discharges, lack of sewage systems, and tourism. This has resulted in eutrophication, excessive nutrients [1], and cyanobacteria blooms that deplete dissolved oxygen, posing a threat to aquatic life and water quality. Efficient monitoring of Lake Ypacaraí’s biological status, especially for dynamic blue-green algae blooms, necessitates periodic Water Quality (WQ) sampling. VOLUME 12, 2024 2024 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ 75559 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach WQ data, such as pH, dissolved oxygen, temperature, and water turbidity, can be used to create a physicochemical model that provides up-to-date information on the water’s biological status. This can aid the decision-making process for monitoring pollution in Lake Ypacaraí more efficiently. Also, certain areas of the lake are more polluted than others [2],[3],[4], and therefore require more frequent monitoring. Manual sampling is impractical due to the lake’s vast size (60 km2) and the potential health risks for biologists in contaminated environments. Furthermore, installing a fixed sensor grid is not an optimal solution because it limits the ability to vary sampling locations and may be more costly due to frequent battery replacements compared to humanconducted missions [5]. In [2] and [3], the deployment of Autonomous Surface Vehicles (ASVs) equipped with WQ sensors for continuous monitoring instead of manual sampling (Figure 1) was proposed. ASVs are capable of operating autonomously, making them more efficient and suitable for tasks performed in hazardous environments, such as polluted areas. In addition, they offer the advantage of hourly sampling, in contrast to the limited frequency of manual sampling (once per day). By deploying a fleet of ASVs, simultaneous measurements can be taken at various locations on the map, yielding a more complete and detailed physicochemical model. The main challenge in the multi-ASV paradigm is to develop an effective coordination strategy that enables the ASVs to independently sample the lake with collision-free routes while meeting a low redundancy criterion. Physical constraints such as battery limitations and non-navigable zones must also be considered, as well as avoiding collisions between ASVs. To address these issues, the continuous monitoring of the lake is modeled as the Patrolling Problem. The objective of this task is to find an efficient route or strategy to cover important zones of the lake while also revisiting those that have not been sampled in a long time. When monitoring Lake Ypacaraí with ASVs, the scenario becomes a Partially Observable Markov Game (POMG) due to the unknown pollution levels at the beginning of the episode. Additionally, experiments in this work focused only on blue-green algae bloom contamination, which is dynamic, chaotic, and changes in size over time. In this work, the patrolling problem is addressed in a decoupled way, dividing it into two phases: exploration and intensification. During the exploration phase, ASVs are required to visit all areas of the map and taking pollution measurements. At this stage, homogeneous patrolling is carried out, as the relative importance of the areas is not taken into account so the lake is uniformly covered. After identifying the most significant areas of contamination, which are characterized by blue-green algae blooms, the intensification phase follows. In this stage, non-homogeneous patrolling is carried out, with a focus on visiting the most relevant areas more frequently. This approach poses a challenge, as it implies that the same agent must adapt to the changing phases and make optimal decisions in each of them. The proposed approach involves employing a variable named νto regulate a smooth transition between the Exploration and Intensification phases. This variable establishes the probability of choosing an exploratory or an intensifying policy action. This transition mechanism, based on ν, is an innovative scientific contribution since, to the best of our knowledge, it has not been proposed in the literature before. Given the extensive number of routes and the number of ASVs required for continuous monitoring of large bodies of water, the complexity of this problem becomes NP-hard. As suggested by [3],[6], and [7], a Deep Reinforcement Learning (DRL) approach is recommended for dealing with this problem’s high dimensionality. DRL methods, like Deep Q-Learning (DQL), operate without a model of the environment (model-free), and uses a Convolutional Neural Network (CNN) to approximate the Q-function, which can effectively handle high-dimensional state spaces [8]. In this multi-ASV system, each vehicle is an agent in a cooperative Multi-Agent DRL (MADRL) environment. To deal with scalability issues, a single neural network, designed and trained based on the study of [6], is shared by all agents. Each agent can share the Deep Policy, as they are interchangeable in terms of the observation and action capabilities. As the patrolling missions have been decoupled into two phases, the agents must learn both tasks, resulting in Multitask Multiagent DRL (MT-MADRL). Our approach is based on the neural network having two heads, one for each task’s Q-function. Therefore, despite any number of agents, there is only one network and one Q-function to optimize for each task. This is due to the agents pursuing the same collective reward and acting in a purely cooperative manner. As a result, each individual reward can be modeled with a single network. Concerning the nonnavigable terrain constraints, Censoring-DQL [5] is applied to takes the deterministic information of the environment to neglect the actions that would violate those constraints. Furthermore, if a possible collision between agents is about to occur, a consensus algorithm proposed in [9] is used to avoid it. Therefore, the main contributions of this article are: •A framework for monitoring partially observable dynamic scenarios with a two-phase approach: Exploration and Intensification. The proposed approach employs a smooth transition mechanism between the phases. •The application of Multitask Multiagent Deep Reinforcement Learning using Deep Q-Learning and parametersharing techniques for environmental monitoring. •A comparison between the proposed framework and other algorithms based on heuristics. This paper is structured as follows: Section II is a review of the literature on Multiagent and Multitask Deep Reinforcement Learning with applications to environmental 75560 VOLUME 12, 2024 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach FIGURE 1. ASV prototype designed for water resources exploration. monitoring. Section III presents the Multiagent Patrolling Problem, provides specific scenario details, and outlines key assumptions. Section IV details the approach utilized to tackle the Multiagent Patrolling Problem, including the use of DRL approaches, the Exploration versus Intensification phase strategy, reward function design principles, and state space representation. In Section V, the paper presents key metrics and simulation results to verify the effectiveness of the proposed methodology. Finally, Section VI summarizes the findings and proposes potential avenues for future research. II. RELATED WORK The use of ASVs in aquatic environments has gained significant attention in recent years [10]. ASVs have a wide range ofpotential applications,including environmental monitoring for early warning of pollution [3],[6] [7], bathymetric surveying for navigation safety [11],[12], and emergency response tasks such as oil spill tracking [13]. Artificial Intelligence approaches, such as Bayesian Optimization (BO) [14], Genetic Algorithms (GA) [15] and DRL [3], have been applied to mitigate the NP-hard complexity of many applications involving a large number of possible trajectories. In [14], it was proposed a mission planning method based on BO, which defines the movements of the ASV with the aim of minimizing the uncertainty of the contamination distribution of an aquatic environment. Although this approach efficiently obtains a physicochemical model for low data regimes, the authors do not consider that water quality parameters can change during a mission. Our proposal addresses the dynamic behavior of algae blooms. Additionally, in [7], the authors demonstrated that scalability issues are better addressed with DRL than with GA approaches, which motivated this work to use DQL to solve the Patrolling Problem. The Deep Q-Network (DQN) algorithm [8], which enabled games on the Atari 2600 console to be played at an expert human level, catapulted the popularity of DQL in 2015. In this work, Rainbow DQN [16] is used, which was introduced as a combination of several improvements, including DDQN [17], DuelingDQN [18], and Prioritized Buffer Replay [19]. Regarding solving non-homogeneous patrolling with DQL, in [20] a relevance map is used to represent areas with different coverage requirements. The policy could only make decisions based on the data of the relevance map within the camera’s field of view. In contrast, our proposal takes point sample measurements while creating a model of the relevant zones (polluted areas). This model is then used as an input image for the neural network. This methodology of environmental representation was previously employed in [3], where the authors created a pollution map for Lake Ypacaraí and generated an image to highlight the areas with the highest pollution levels. This work differs from ours in that they did not have a dynamic pollution map and also did not use a collision avoidance mechanism. Using multiple autonomous vehicles as sensing nodes [6], [21],[22] leads to increased coverage and faster data collection. The extension of DRL to the multi-agent case (MADRL) enhances the performance of the single-agent case but brings new inherent challenges such as computational complexity, nonstationarity, partial observability and credit assignment [23]. In cooperative multiagent environments, parameter sharing (PS) [24] has been shown to be effective when agents are homogeneous (they share the same set of skills). The PS approach, which our proposal uses, involves sharing the parameters of a single policy among all agents, which are trained with the experiences of all agents simultaneously. This enables an efficient training process that can scale up to an arbitrary number of agents, thereby reducing the computational complexity. Previous studies, such as [25], have utilized PS in MADRL, where the network input includes information on the relative position and velocity of other agents. While the proposed approach is egocentric, our work diverges as it employs a single buffer to store experiences while they use a separate buffer for each agent, limiting the scalability of their algorithm. In contrast, other works like [26], do not use replay memory. Instead, they combine PS with Deep Recurrent Q-Network (DRQN) [27] to address partial observability. They approximate the Q-function with a recurrent neural network (RNN) that can maintain an internal state and aggregate observations over time. However, DuelingDQN architecture is proposed as a faster and easier-to-train alternative to RNN in highdimensional spaces, as noted by [24]. Additionally, the partial observability of our proposed environment is mitigated by implementing a first phase of homogeneous patrolling, as the pollution distribution along the lake at the beginning of the episode is unknown. In the context of multi-agent patrolling of environmental missions, [6] and [28] proposed a scheme that employs Deep Q-Learning with a Convolutional Neural Network as a shared fleet policy and utilizes a global visual VOLUME 12, 2024 75561 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach state.In [6],a decoupledfinal layerwasproposed forN agents with |A| possible actions. This layer consists of an individual fully-connected layer for every agent, resulting in |A|×N neurons in the last layer. In contrast, our proposal only has 2x|A| neurons in the last layer for any number of agents. In fully cooperative environments that employ joint reward signals, agents face credit assignment challenges when determining the impact of their actions on team performance. To tackle the credit assignment problem, [28] proposed a decoupled reward in which each agent receives a reward only for their individual contributions, without any additional considerations. They demonstrated that their approach is effective in addressing the credit assignment problem, which motivates this work to employ it. The works mentioned above only address possible collisions of the agents that are explicitly penalized in the reward function. In contrast, our proposal includes collision avoidance mechanisms to guarantee that no action will cause any collisions. In this study, the agents learns to optimize two tasks, therefore two policies. The first part of the episode involves making decisions based on a policy optimized only for the exploration task, while the second part involves taking actions following an intensification policy. In the literature, the multitask paradigm in DQL has been addressed with Multitask Deep Q-Network (MDQN) [29],[30],[31]. A network is trained as a vanilla DQN, but with a separate output layer (head) for each task. In [29], it was concluded that, given sufficiently related tasks, shared hidden layers in the MDQN can efficiently learn a shared feature representation and therefore perform well across tasks. In our case, the exploration and intensification phases are similar, as the agents coordinate their efforts to reduce the weighted average idleness on a single map. The conclusion of [29] and other successful single-agent applications of MDQN [30],[31] motivated us to use this approach. Extending to the multi-agent paradigm, works [32] and [33] have applied knowledge transfer to MT-MADRL by training task-specific neural networks (teachers) first and then applying a knowledge distillation algorithm to train a policy that performs well across tasks. Our proposal differs in that it aims to learn a policy for each task, rather than one policy for multiple tasks. On the other hand, the authors in [34] proposed an approach that allows agents to have multiple policies for different tasks. However, their algorithm is based on the deep deterministic gradient algorithm (DDPG), which is a DRL algorithm for continuous action space environments. Unfortunately, this algorithm cannot be applied to our proposed discrete action space environment. Furthermore, to the best of our knowledge, no approaches to MT-MADRL utilize DQL. This research aims to fill this gap. III. PRELIMINARIES A. THE PATROLLING PROBLEM The main task of the patrolling problem is to identify the ideal strategy for visiting relevant areas of an environment FIGURE 2. Example of a graph G with vertices Vkand edges Ek−j, for clarity all the edges names are not displayed. Either of the two agents (in V7and V9can move to eight different directions, indicated by the red arrows. Nodes in red indicates the possible locations where a collision could occur if both agents are headed to it. in order to monitor them in a periodic way. To represent a continuous surface, the terrain skeletonization technique [35] can be used. This means that the environment to be covered can be represented as an undirected graph G(V,E,W) (see Figure 2), V=1,2, . . . ..nis the set of vertices (also called nodes) of the graph and Eis the set of edges of G. With this formulation, different types of problems can be represented, depending on the importance of the costs associated with the edges. In the proposed scenario, the cost is determined by the distance between nodes. Every graph node also has an idleness, denoted by Wkfor the node k∈V, which indicates the amount of time since the node was last visited by an agent. In this paper, the objective is to reduce the average idleness of the graph and, consequently, of each individual node. In the patrolling problem, when all areas are considered equally important and there are no specific or relevant areas, it is referred to as the homogeneous patrolling problem. However, if there are relevant areas, they should be patrolled more frequently based on the importance assigned to each zone, which is known as the non-homogeneous patrolling problem. Furthermore, Lake Ypacaraí contains regions with high pollution concentration, such as blue-green algae blooms. The behavior of these blooms is dynamic and often chaotic, causing them to frequently change in size over time. Therefore, the set of dynamic weights for the graph nodes, denoted as I(t), indicates the importance of each node based on its level of contamination, with the more contaminated ones having greater importance. Therefore, the non-homogeneous patrolling problem proves to be the best fit for the Lake Ypacaraí case [3] and can be formulated as finding a policy π that minimizes the average idleness weighted by the set of weights I(t) given a number of time steps: G(V,E,W,I,t)−→ π(E)|min 1 |V| |V| X k=1 Wk×Ik(t) (1) The patrolling problem is naturally suited to be shared by multiple agents in space and time since they can patrol 75562 VOLUME 12, 2024 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach FIGURE 3. Discretized map of Lake Ypacaraí. Blue cells denote visitable water surfaces while brown cells denote illegal, non-navigable areas. different areas simultaneously. Our fleet of ASVs shares a homogeneous architecture and shares its information about the environment, so the idleness to minimize is the shared idleness. This means that, all agents collectively reset the estimate of a node’s idleness based on the visits made by all agents. B. SCENARIO AND ASSUMPTIONS The scenario definition is essential in patrolling problems as it establishes the ASV or agent’s movement abilities, as well as the real constraints and boundary conditions. Our target scenario design is based on the following assumptions: 1) LAKE MAP The map of Lake Ypacaraí has been divided into a grid map (see Figure 3), where each cell corresponds to a node and the distance traveled between adjacent cells corresponds to the weight of an edge. Given the total surface area of the lake at 60 km2, each square cell represents 290m×290 m of the lake’s area. Cells that cannot be occupied (brown cells in Figure 3), such as those outside the navigable surface of the lake or those representing land space, have a null value in the grid map. However, obstacles within the lake are not taken into consideration during the simulation, as the ASV obstacle sensors (Lidar +Camera) possess the ability to avoid them with a local reactive trajectory planner. 2) LAKE’S CONTAMINATION As the ASVs explore the lake and collect samples, they build a map of the lake’s pollution, which is also part of the scenario. Our experiments focused solely on pollution caused by blue-green algal blooms. The bloom’s behavior is dynamic and chaotic, and its size changes over time, so it is modeled using a brownian movement. In this model, the concentration of algae is represented by discrete particles that can move within the lake space. Using the position of each particle, a particle map P(t) is constructed following the same discretization as the grid map. Each position or cell of the map indicates the number of particles within it. To convert the particle map P(t) to a scalar contamination field I(t), a Gaussian filter is applied to the image of the particles on the grid. This converts the particle map to a particle density map, which is ultimately referred to as the contamination map (see Figure 4). 3) VEHICLE MOVEMENT Once the defined number of directions for the agent has been established, the angles for each direction are calculated, evenly spaced within an angle interval of [0,2π]. To ensure the ASV’s motion capabilities are realistically represented, eight different cardinal directions (N, E, S, W and NE, SE, NW, SW) (red arrows in Figure 2) is assumed at each step. Since all ASVs have the same capabilities, they are assumed to move synchronously. For simplicity, each step of the simulation is a movement in one direction. Illegal actions are movement into a cell that is simultaneously desired by another agent (red nodes in Figure 2) or movement into a cell that forms part of non-navigable zones (obstacles or shores). Actions leading to non-navigable zones are masked to prevent them from being taken. This ensures that only actions allowing movement within the graph are available. The battery life of ASVs is a significant constraint when deployed in real-world applications. In our study, the maximum distance that drones can travel if they start with a full battery is determined. This is the condition in which the mission is terminated. 4) COMMUNICATION In this scenario, communication between agents regarding fleet position and pollution measurements is assumed. They jointly create an importance map (the set of weights I(t)) based on contamination measurements. The model determines the importance of a cell based on its most recent contamination measurement. To ensure an interest in covering them, all cells within the navigable zone of the lake have a minimum importance value. Furthermore, having knowledge about the fleet position is crucial for each agent to avoid collisions and ensure safe and efficient behavior. IV. METHODOLOGY A. MULTITASK REINFORCEMENT LEARNING Reinforcement Learning (RL) [36] is an Artificial Intelligence approach where an agent learns by interacting with the environment. The Markov Decision Process (MDP) mathematical framework is used to formalize RL problems, such as patrolling Lake Ypacaraí. A MDP is denoted by a tuple consisting of five elements: (S,A,P,R, γ ). Where Sis the state space, Ais the action space, P:S×A×S−→ [0,1] is the transition probability, R:S×A−→ R+is the reward function, γ∈[0,1) is the discount factor and it represents the agent’s preference for immediate rewards over future rewards. At each time step t, the agent observes a state st∈Sand selects an action at∈Abased on the policy π(at|st). The environment produces a reward of rtbased on VOLUME 12, 2024 75563 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach FIGURE 4. Example of the evolution of a contamination map as a result of blue-green algae in Lake Ypacaraí. (a) is the initial map, (b) is the map after 30 steps, (c) after 60 steps, (d) after 100 steps. the reward function rt=R(st,at) and progresses to the next state st+1through the transition probability P(st+1|st,at). The agent’s long-term policy return (J(π)) is defined as the expected discounted accumulated rewards: J(π)=Eat∼π"∞ X t=0 γtrt#(2) Therefore, the agent must learn an optimal behavior policy π:S−→ P(A) that optimizes the expected performance J(π). In partially-observable environments, agents receive observations of the state rather than observing it directly. These domains are formalized as Partially Observable Markov Decision Processes (POMDPs), which are defined as (S,A,P,R, , O, γ ). In addition to the elements shared with MDPs, POMDPs include the discrete set of observations = {o1,...,oM}and the observation function O:S×A× −→ [0,1]. After choosing an action aat timestep t, the agent observes ot+1∈with probability P(ot+1|st+1,at)= O(ot+1,st+1,a). In Multitask Reinforcement Learning, there is a collection of M tasks T= {Ti}N i=0. Each task Tihas a unique MDP (POMDP if the environment is partiallyobservable) Mi=(S,A,Pi,Ri, γ ) that shares the same stateaction space, but the transition probability Piand the reward function Ridiffer across tasks. The goal is to simultaneously learn multiple tasks and exploit their similarity to improve performance relative to single-task learning. Each task has a maximization objective given, so the goal is to learn a policy that maximizes the total expected returns over the tasks. B. DEEP Q-LEARNING The state-action value function, Qπ(s,a), represents a table of Q values that describe the utility of being in a specific state sand taking a particular action awhile following a given policy π(s). Deep Q-Learning is a widely used algorithm in reinforcement learning (RL). Instead of iteratively updating the Q-values in a table (Q-Learning), it uses a neural network to estimate the state-action, which can learn to map the states directly to their corresponding actions while estimating the corresponding Q-values using parameterized weights (θ). This is useful for handling high-dimensional state spaces in Q-learning. For a given state s, action a, received reward r, and next state s′, the Q-value update in DQL is expressed as: Q(s,a;θ)=Q(s,a;θ) +α∗(r+γ∗max aQ(s′,a;θ)−Q(s,a;θ)) (3) The learning rate (0 < α ≥1) determines how quickly Q(s,a) is updated with new data during each iteration. The discount factor γ∈(0,1] adjusts the importance of rewards over time. In addition, DQL commonly uses the epsilon-greedy strategy to balance exploration and exploitation. This strategy is applied during action selection to maintain a balance between exploiting current knowledge and exploring new possibilities. This is done by selecting the action with the highest estimated Q-value with a probability of (1 −ϵ) for exploitation, or selecting a random action with a probability of ϵfor exploration. In their work [8], authors present two important methods for improving stability and efficiency in reinforcement learning: Experience Replay and Target Network. Experience Replay consist of storing past experiences (s,a,r,s′) in a memory buffer. This allowed the agent to learn from different situations and avoiding problems such as catastrophic forgetting and correlated observations. The Target Network is a duplicate of neural network parameters for action selection; it updates less frequently than the main network, reducing correlations and enhancing stability. Additionally, to tackle overestimation of action values, Double Deep Q-Network (DDQN) [17] was proposed. This approach uses the main network for action selection and the target network for value estimation, mitigating the overestimation problem. Furthermore, [18] proposed the dueling architecture, a neural network framework that separates the estimation of the State-Value Function V(s) and the Advantage Function A(s, a). In this dueling architecture, two streams representing V(s) and A(s, a) share a common feature learning module 75564 VOLUME 12, 2024 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach and are then combined through a unique aggregation layer to obtain an estimate of the state-action value function Q using the following general formula: Q(s,a;θ)=V(s;θ′)+ A(s,a;θ′′)−1 |A|X a′ A(s,a′;θ′′)! (4) where θ′and θ′′ are the separate parameter sets used to estimate Vand Arespectively. C. PROPOSED DECOUPLED METHOD The pollution levels and, consequently, the set of weights I(t) are unknown at the beginning of the mission. However, to effectively achieve Equation 1, the group of agents must first acquire sufficient knowledge of I. For this reason, the mission is divided into two phases: exploration and intensification. In the exploration phase, ASVs patrol the area homogeneously with the aim of minimizing average idleness, without considering the importance (the set I(t)) of specific zones. So, our main goal is to achieve homogeneous coverage by searching for a joint policy 5= {π1, π2, . . . , πN}of N agents that minimizes the average idleness value throughout the map. G(V,E,W)−→ 5(E)|min 1 |V| |V| X k=1 Wk(5) Once contamination information has been gathered, our goal is to implement a strategic plan to ensure thorough and intensive coverage of the highly polluted zones during the intensification phase. Our primary objective is to develop a joint policy, denoted by 5= {π1, π2, . . . , πN}, for N agents with the aim of minimizing the average idleness idleness weighted by the set of weights I(t) throughout the map. G(V,E,W,I,t)−→ π(E)|min 1 |V| |V| X k=1 Wk×Ik(t) (6) In this case, the exploration and intensification phases are similar since the agents coordinate their efforts to decrease average idleness on a single map. Conversely, exploring has a homogeneous coverage while intensification is heterogeneous, making our problem inherently multiobjective. If the agents only learned the task of intensification, they would need to explore to gather a better model of the contamination at the beginning of the mission and then decide where to intensify. To speed up the learning process and increase the efficiency of sampling, the study relaxes the requirement that agents must use the same policy for both exploration and intensification, acknowledging the conflicting nature of these goals. The proposed method uses a variable, ν, to regulate a smooth transition between the exploration and intensification phases. This variable determines the probability of choosing an exploration or intensification policy action. For the sake of clarity, let Qerepresent the head of our network that estimates the Q-values of the exploration policy, and Qirepresent the head of our network that estimates the Q-values of the intensification policy. The resultant policy, πν, is defined as: πν(s)=   arg max a Qe(s,a)with probability ν arg max a Qi(s,a)with probability 1−ν(7) Thus, the patrolling phases and the transition phase can be defined flexibly: •ν=1: Exploration phase. •ν=0: Intensification phase. •0< ν > 1: Transition phase, during which there is a (ν∗100)% probability of choosing an exploratory action, and therefore νgradually decreases to smoothly transition to the intensification phase. D. COLLISION AVOIDANCE MECHANISMS As mentioned above, the Censoring-DQL [5] algorithm is employed to ensure deterministic computation that can address actions leading to nonnavigable zones. In this algorithm, invalid actions of an agent ithat would lead to nonnavigable zones are identified using the information in the lake map, which is known a priori. Then, a censoring function η(s,ai)∈R|Ai|(see Equation 8) is computed. η(s,ai)=1if aiis valid −∞ if aiis not valid (8) Once the invalid actions are detected, the observation oiis processed by the Q-network and the final censored Q values (for each task) Qi Care obtained. Qi C(oi,ai)=η(s,ai)◦Qi(oi,ai) (9) This ensures that only actions allowing movement within the graph are available. On the other hand, simultaneous actions within the navigable zone may cause conflicts, resulting in collisions between agents. To address this issue, the SafeConsensus [9] algorithm is utilized. The algorithm sorts agents based on the highest joint value of Q, with the highest-Q agent taking action without considering other agents. The agents that follow consider the new position of the previous one, censoring Q values that lead to collisions with −∞. This heuristic is based on conditional decision-making and relies on agent optimism to prioritize actions. E. MULTITASK MULTIAGENT DEEP Q-NETWORK In this study, the Parameter-Sharing Multiagent MDQN (PSMA-MDQN) is presented as an extension of the MDQN to the multiagent paradigm. The DDQN method in combination with the dueling architecture (Dueling DQN) is used to develop a Dense Convolutional Neural Network with two parallel terminations, referred to as ‘‘heads’’, each corresponding to a specific task: exploration and intensification. The output layer and parameters of each head are separated, allowing the PSMA-MDQN to learn and VOLUME 12, 2024 75565 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach FIGURE 5. The PSMA-MDQN architecture proposed here utilizes a shared Feature Extractor that captures common features from the input state. Each task is assigned an individual head consisting of a three-layer dense neural network and then a dueling architecture with separate Advantage (A(s,a)) and State (V(s)) value heads. The final Q-values of each task, Qe(s,a) for exploration and Qi(s,a) for intensification, are obtained by combining its correspondent A(s,a) and V(s) streams. All activation layers correspond to the ReLU function. optimize these two tasks independently. Each head in the PSMA-MDQN has its own loss function, and the weights of each head are adjusted independently during training. The shared block, also known as the Feature Extractor, extracts useful common features for both tasks [29]. These shared features are utilized by each individual head to generate its corresponding output (see Figure 5). This approach is feasible because each agent operates with the same set of actions and is subject to the same constraints, making them homologous in both actions and observations. So, this work benefits from the homogeneity of the agents to train on a single network a policy π1=π2=. . . =πNthat serves multiple agents without increasing the network parameters with respect to a single agent. Since the objective is to learn more than one policy, our proposal only has M× |A|neurons in the last layer for any number of agents and for M tasks. Furthermore, as validated in [6] and [9], the number of agents in this scheme does not impact on the stability in the learning process. In Algorithm 1, the pseudocode for the approaches is presented. F. STATE REPRESENTATION The state represents the environment in which the agent operates and the information accessible for decision-making purposes. In our simulation environment, the agents have partial information about the lake’s contamination, as they do not know the contamination levels of the cells they have never visited. Therefore, the environment is partially observable as the agents only have an observation of the dynamic set of weights I(t). To enable the use of a single policy network for all agents, an egocentric observation formulation is employed. This means that the relative position of other agents is also observed, ensuring that each agent has a unique observation. Thus, there is a distinction between the shared and individual elements of the observations. Our proposal is a state representation composed of 4-channel images minmax-normalized, this way every pixel value of the state is within [0, 1]. 1) SHARED ELEMENTS These common elements of the state ensure that all agents have access to the same information about the environment and can coordinate their actions effectively (see Figure 6). •Idleness map: An image containing idleness values for each cell, which is generated based on fleet visits (see Figure 6a). All cells within the navigable zone are set to the maximum value 1 initially, denoting that none of the cells has been visited. When an agent visits a cell, its idleness resets to zero. If a cell remains unvisited over time, its idleness value increases, indicating the need for revisitation. •Importance map: This map updates the relative importance of visited cells as new measurements are taken (see Figure 6b). All cells within the navigable zone are initially set to the minimum importance value, indicating an unknown but not null interest. When an agent visits a cell, its relative importance is updated based on the level of contamination. The more polluted a cell is, the more importance it has. 2) INDIVIDUAL ELEMENTS Each agent’s state will be differentiated by his position and therefore it is allowed to use one single policy network for all of them (see Figure 6): •Agent position: This binary image has a zero value for all cells, except for those cells that are covered by the ASV detection area (white cells in Figure 6c). This image provides specific information about the agent’s current location and allows him to make decisions based on his own position. •Other agent’s position: It is a binary image where all cells have a value of zero, except for cells corresponding to the location and detection area of other agents in the environment (white cells in Figure 6d). This image enables each agent to have knowledge of the other 75566 VOLUME 12, 2024 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach FIGURE 6. 4-channel egocentric observation of an agent, in this case, the agent 0. agents’ position and avoid collisions or coordinate their actions appropriately. G. REWARD FUNCTIONS To guide the agents towards optimal behavior, it is necessary to design a reward function that motivates the agents to achieve the following goals: 1) During the first phase of the patrolling, the agent fleet should be fully exploratory, visiting the entire map in a coordinated fashion. 2) After gathering data on the lake during the initial phase, the agents should move on to an intensification phase that takes into account the importance of each zone. 3) Penalize agents who take measurements in the same cell on the map. In both phases of the process, an idleness matrix (W) is defined such that, Wt(x,y) is the idleness at the position (cell) (x,y) at time t. Moreover, the measurements taken by agent i at time tare restricted to an area of detection (ωi t) of radius r, defined by the agent’s sensors (see Figure 7). In this notation, cells measured by an individual agent i(within the area ωi t) are denoted as (xi,yi). To penalize that more than one agent is taking a measurement in the same cell, the Redundancy Mask (RM) stores the number of measurements in each cell (see Figure 7). In each step t, every agent i=1,2,...,Nperforms a single measurement within the detection mask, thus the calculation of RM is as follows: RMt= N X i ωi t(10) When multiple agents take measurements in the same cell, they share the reward received for measuring in that cell. To achieve this, the value of the reward is divided by RM matrix. As a result, the agents should distribute themselves and maintain a safe distance from each other to improve coverage and maximize the use of available information. FIGURE 7. This image illustrates the detection areas ωof the ASVs and the values of the redundancy mask RM in overlapping ωregions. Additionally, the total reward colLected is normalized by dividing it by r. This enables a fair comparison of the rewards regardless of the size of the detection radius used. Although rewards are calculated using global matrices (where Wand RM are updated with data from the entire fleet), only information within the ASV’s detection area is considered to reward this particular ASV. During the exploratory phase, the objective is to visit homogeneously the entire zones to minimize the average idleness of the map. Therefore, visits to cells with high idleness should be incentivized. Given (xi,yi), the cells to be measured as a result of the action ai, the Exploration Reward (ERi) received by agent iis calculated as follows: ERi t=Wt(xi,yi) r×RMt(xi,yi)(11) During the Intensification phase, the agents should focus on the most relevant zones. Thus, an importance matrix (I) is defined such that, It(x,y) is the relative importance at the position (x,y) at time t. The Intensification Reward (IRi) received by agent iis calculated as the Equation 12. By weighting the relative importance of the cells to be VOLUME 12, 2024 75567 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach •The decoupled method introduced in this study addresses the constraint of agents having to use a single policy for both initial exploration and subsequent intensification, thereby taking into account the inherent conflict between these objectives. This method accelerates the learning process. This is demonstrated by the fact that when entering the intensification phase, our algorithm reduces the IGWI by 17% more than the Single-Phase DQN. This performance superiority is maintained throughout the rest of the episode. •Parameter sharing is more efficient for related multitask learning in our case study. Our algorithm showed a 6% lower reduction in IGI during the Exploration Phase, a 13% lower reduction in IGWI, and a 7% lower reduction in AGWI compared to the Task-Specific DQN, despite the latter using 1.96 times as many neural network parameters. •The algorithm developed in this study outperforms heuristic-based approaches such as LMPP, RWPP and PSOPP. On average, it achieves a 44% lower AGWI compared to LMPP and a 31% lower AGWI compared to RWPP and PSOPP by the end of the episode. Additionally, the algorithm demonstrates a learned coordination strategy, resulting in a 47% reduction in IGI compared to LMPP, a 34% and 39% lower IGI than RWPP and PSOPP respectively by the conclusion of the first phase. The algorithm’s superior speed covering the map is evident, with a 48% advantage over RWPP, a substantial 130% lead over LMPP and a 58% more PV than PSOPP by the end of the exploration phase (step 30). •The comparison of the proposed algorithm and LMPP highlights the superior performance of the former. This is mainly due to LMPP’s exhaustive approach to ensuring complete coverage by strictly following parallel paths, leading to excessive redundancy.Additionally, in non-convex scenarios, LMPP would face challenges in escaping corners, which would impede its proper operation. As for RWPP, it provides fast homogeneous map coverage, but lacks coordination based on weighted idleness, thereby lacking effective intensification. Additionally, its performance also would decrease in nonconvex settings. •Regarding PSOPP, when particles initiate exploration from nearby locations, they tend to exhibit similar behaviors. In the Exploration Phase, each particle moves towards the nearest cell with the highest idleness, resulting in a dispersion of paths when encountering obstacles such as the border of the lake. However, in the Intensification Phase, where importance is concentrated, particles tend to congregate and move collectively. This behavior is especially problematic in environments with multiple contamination peaks. In such cases, particles tend to move in groups from one peak to another, making the task extremely inefficient. In contrast, our algorithm provides agent allocation strategies that are particularly suitable for achieving homogeneous and non-homogeneous coverage across the entire map, even in complex and dynamic environments, e.g., when contamination peaks are dispersed. •Our algorithm has learned to perform two tasks independently, and the policies can be used arbitrarily. •Our algorithm allows for smooth transitions between phases or the use of a single phase. This provides users with the flexibility to configure the algorithm according to their needs. VI. CONCLUSION In the context of a dynamic Partially Observable Markov Game (POMG), such as the challenging Lake Ypacaraí patrolling scenario with multiple ASVs, our approach divides the patrolling task strategically into two distinct phases: the Exploration Phase and the Intensification Phase. The aim of the Exploration Phase is to cover the map homogeneously, while the objective of the Intensification Phase is to intensify the coverage in the most polluted areas. Additionally, a novel scientific approach have been introduced to ensure a smooth transition between the two phases. To tackle the computational complexity of the problem, a Dueling DQN has been trained with two heads, one dedicated to estimating the Q-function for the Exploration Phase and the other for the Intensification Phase. The policy is shared across all agents since they are homogeneous and the input state formulation is egocentric. The results indicate that the decoupled method introduced in our study is effective. This method frees agents from the constraint of using a single policy for both exploration and intensification, which accelerates the learning process. Our algorithm consistently outperforms the Single-Phase DQN, a policy trained on a single intensification task, with the same architecture as ours but utilizing a single Dueling DQN head. Furthermore, our multitask learning approach with parameter sharing is more efficient than Task-Specific DQN. The latter uses two single Dueling DQN with the same architecture as ours,buteach dedicatedto adifferenttaskand trainedwithin a decoupled phase setting. Our approach achieves better results despite using approximately half as many neural network parameters. By changing the values of ν−intervals during evaluation to ones that the agents had not trained with, it is demonstrated that the algorithm had acquired the ability to perform the two tasks autonomously, even in scenarios not encountered during training. This provides the flexibility for users to configure the values of ν−intervals according to their requirements. In future lines of research, shifting from predefined task durations to training the network with multiple objectives emerges as a promising way to achieve optimal performance configurations. This shift allows for a more nuanced consideration of user preferences, including exploration and intensification requirements, as well as energy efficiency. The proposed approach is to solve a multi-objective optimization problem to identify a Pareto front which 75574 VOLUME 12, 2024 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach contains non-dominated policies. These policies are solutions where improving one objective cannot be achieved without compromising the performance of another objective. The exploration of the Pareto front, achieved by training with varying objective weightings, offers opportunities to discover versatile and adaptive patrolling strategies tailored to diverse user needs and environmental dynamics. Additionally, other environmental monitoring tasks, such as bathymetric surveys and trash detections could be added to the proposed framework. Moreover, including battery level as another decision variable in trajectory design considerations offers a promising area of research. This addition has the potential to improve the efficiency and sustainability of autonomous systems operating in dynamic environments. REFERENCES [1] G. López Moreira M. et al., ‘‘Eutrophication, research and management history of the shallow Ypacaraí lake (Paraguay),’’ Sustainability, vol. 10, no. 7, p. 2426, Jul. 2018. [Online]. Available: https://www.mdpi.com/2071-1050/10/7/2426 [2] M. Arzamendia, D. Gutierrez, S. Toral, D. Gregor, E. Asimakopoulou, and N. Bessis, ‘‘Intelligent online learning strategy for an autonomous surface vehicle in lake environments using evolutionary computation,’’ IEEE Intell. Transp. Syst. Mag., vol. 11, no. 4, pp. 110–125, Winter. 2019. [3] S. Y. Luis, D. G. Reina, and S. L. T. Marín, ‘‘A deep reinforcement learning approach for the patrolling problem of water resources through autonomous surface vehicles: The Ypacarai lake case,’’ IEEE Access, vol. 8, pp. 204076–204093, 2020. [4] M. Arzamendia, D. Gregor, D. G. Reina, and S. L. Toral, ‘‘An evolutionary approach to constrained path planning of an autonomous surface vehicle for maximizing the covered area of Ypacarai lake,’’ Soft Comput., vol. 23, no. 5, pp. 1723–1734, Mar. 2019. [5] S. Y. Luis, D. Gutiérrez-Reina, and S. T. Marín, ‘‘Censored deep reinforcement patrolling with information criterion for monitoring large water resources using autonomous surface vehicles,’’ Appl. Soft Comput., vol. 132, Jan. 2023, Art. no. 109874. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1568494622009231 [6] S. Y. Luis, D. G. Reina, and S. L. T. Marin, ‘‘A multiagent deep reinforcement learning approach for path planning in autonomous surface vehicles: The Ypacaraí lake patrolling case,’’ IEEE Access, vol. 9, pp. 17084–17099, 2021. [7] S. Yanes Luis, D. Gutiérrez-Reina, and S. T. Marín, ‘‘A dimensional comparison between evolutionary algorithm and deep reinforcement learning methodologies for autonomous surface vehicles with water quality sensors,’’ Sensors, vol. 21, no. 8, p. 2862, Apr. 2021. [Online]. Available: https://www.mdpi.com/1424-8220/21/8/2862 [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, and G. Ostrovski, ‘‘Human-level control through deep reinforcement learning,’’ Nature, vol. 518, no. 7540, pp. 529–533, 7540. [9] S. Yanes Luis, D. Shutin, J. Marchal Gómez, D. Gutiérrez Reina, and S. Toral Marín, ‘‘Deep reinforcement multiagent learning framework for information gathering with local Gaussian processes for water monitoring,’’ Adv. Intell. Syst., Apr. 2024, Art. no. 2300850. [10] J. Sánchez-García, J. M. García-Campos, M. Arzamendia, D. G. Reina, S. L. Toral, and D. Gregor, ‘‘A survey on unmanned aerial and aquatic vehicle multi-hop networks: Wireless communications, evaluation tools and applications,’’ Comput. Commun., vol. 119, pp. 43–65, Apr. 2018. [11] C. Specht, E. Świtalski, and M. Specht, ‘‘Application of an autonomous/unmanned survey vessel (ASV/USV) in bathymetric measurements,’’ Polish Maritime Res., vol. 24, no. 3, pp. 36–44, Sep. 2017. [12] J.-H. Hyun, D.-H. Lee, and J.-C. Lee, ‘‘Bathymetric surveying using autonomous surface vehicle for shallow-water area with exposed and partially exposed rocks,’’ Sensors Mater., vol. 35, no. 9, p. 3451, 2023. [13] S. S. Rathour, N. Tanabae, N. Kato, H. Senga, Y. Hirai, M. Yoshie, and T. Tanaka, ‘‘Control algorithm for oil spill tracking using ASV with onboard oil detecting sensor,’’ in Proc. ISOPE Int. Ocean Polar Eng. Conf., 2015, p. 1. [14] F. P. Samaniego, D. G. Reina, S. L. T. Marín, M. Arzamendia, and D. O. Gregor, ‘‘A Bayesian optimization approach for water resources monitoring through an autonomous surface vehicle: The Ypacarai lake case study,’’ IEEE Access, vol. 9, pp. 9163–9179, 2021. [15] M. J. T. Kathen, F. P. Samaniego, I. J. Flores, and D. G. Reina, ‘‘AquaHetPSO: An informative path planner for a fleet of autonomous surface vehicles with heterogeneous sensing capabilities based on multi-objective PSO,’’ IEEE Access, vol. 11, pp. 110943–110966, 2023. [16] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver, ‘‘Rainbow: Combining improvements in deep reinforcement learning,’’ Proc. AAAI Conf. Artif. Intell., vol. 32, no. 1, Apr. 2018. [17] H. Van Hasselt, A. Guez, and D. Silver, ‘‘Deep reinforcement learning with double Q-learning,’’ in Proc. AAAI Conf. Artif. Intell., 2016, vol. 30, no. 1, pp. 2094–2100. [18] Z. Wang, N. D. Freitas, and M. Lanctot, ‘‘Dueling network architectures for deep reinforcement learning,’’ in Proc. Int. Conf. Int. Conf. Mach. Learn., vol. 48, Jun. 2016, pp. 1995–2003. [19] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, ‘‘Prioritized experience replay,’’ 2015, arXiv:1511.05952. [20] C. Piciarelli and G. L. Foresti, ‘‘Drone patrolling with reinforcement learning,’’ in Proc. 13th Int. Conf. Distrib. Smart Cameras, Sep. 2019, pp. 1–6. [21] X. Yu, C. Wang, L. Xu, C. Wu, Z. Wang, Y. He, and W. Wang, ‘‘When connected and automated vehicles meet mobile crowdsensing: A perception and transmission framework in the metaverse,’’ IEEE Veh. Technol. Mag., vol. 18, no. 4, pp. 22–34, Dec. 2023. [22] Z. Wang, L. Tian, L. Lin, J. Xie, W. Wu, and Y. Tong, ‘‘Data collection system of IoT based on the coordination of drones and unmanned surface vehicle,’’ J. Adv. Transp., vol. 2023, pp. 1–21, Feb. 2023. [23] A. Wong, T. Bäck, A. V. Kononova, and A. Plaat, ‘‘Deep multiagent reinforcement learning: Challenges and directions,’’ Artif. Intell. Rev., vol. 56, no. 6, pp. 5023–5056, Jun. 2023. [24] J. K. Gupta, M. Egorov, and M. Kochenderfer, ‘‘Cooperative multi-agent control using deep reinforcement learning,’’ in Proc. Auto. Agents Multiagent Syst., AAMAS Workshops, São Paulo, Brazil. Cham, Switzerland: Springer, May 2017, pp. 66–83. [25] D. Chen, M. R. Hajidavalloo, Z. Li, K. Chen, Y. Wang, L. Jiang, and Y. Wang, ‘‘Deep multi-agent reinforcement learning for highway on-ramp merging in mixed traffic,’’ IEEE Trans. Intell. Transp. Syst., vol. 29, pp. 1–16, 2023. [26] J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson, ‘‘Learning to communicate with deep multi-agent reinforcement learning,’’ 2016, arXiv:1605.06676. [27] M. Hausknecht and P. Stone, ‘‘Deep recurrent Q-learning for partially observable MDPs,’’ in Proc. AAAI Fall Symp. Ser., 2015, pp. 1–9. [28] S. Y. Luis, M. P. Esteve, D. Gutiérrez Reina, and S. T. Marín, ‘‘Deep reinforcement learning applied to multi-agent informative path planning in environmental missions,’’ in Mobile Robot: Motion Control and Path Planning (Studies in Computational Intelligence), vol. 1090. Cham, Switzerland: Springer, 2023, pp. 31–61. [Online]. Available: https://link.springer.com/10.1007/978-3-031-26564-8_2 [29] L. T. Liu, U. Dogan, and K. Hofmann, ‘‘Decoding multitask DQN in the world of minecraft,’’ in Proc. 13th Eur. Workshop Reinforcement Learn., 2016, pp. 1–11. [30] C. D’Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters, ‘‘Sharing knowledge in multi-task deep reinforcement learning,’’ in Proc. 8th Int. Conf. Learn. Represent., Addis Ababa, Ethiopia, Apr. 2020, pp. 1–11. [31] S. Kai, B. Wang, D. Chen, J. Hao, H. Zhang, and W. Liu, ‘‘A multi-task reinforcement learning approach for navigating unsignalized intersections,’’ in Proc. IEEE Intell. Vehicles Symp. (IV), 2020, pp. 1583–1588. [32] S. Omidshafiei, J. Pazis, C. Amato, J. P. How, and J. Vian, ‘‘Deep decentralized multi-task multi-agent reinforcement learning under partial observability,’’ in Proc. Int. Conf. Mach. Learn., 2017, pp. 2681–2690. [33] Y. Mai, Y. Zang, Q. Yin, W. Ni, and K. Huang, ‘‘Deep multi-task multi-agent reinforcement learning with knowledge transfer,’’ IEEE Trans. Games, pp. 1–11, 2023, doi: 10.1109/TG.2023.3316697. VOLUME 12, 2024 75575 D. S. Diop et al.: Decoupling Patrolling Tasks for WQ Monitoring: A MADRL Approach [34] K. Hamadeh, J. El Zini, J. Hajar, and M. Awad, ‘‘MTMA-DDPG: A deep deterministic policy gradient reinforcement learning for multi-task multiagent environments,’’ in Proc. IFIP Int. Conf. Artif. Intell. Appl. Innov. Cham, Switzerland: Springer, 2022, pp. 270–281. [35] A. Machado, G. Ramalho, J.-D. Zucker, and A. Drogoul, ‘‘Multi-agent patrolling: An empirical analysis of alternative architectures,’’ in Proc. Int. Workshop Multi-Agent Syst. Agent-Based Simul. New York, NY, USA: Springer, 2002, pp. 155–170. [36] R. S. Sutton and A. G. Barto, Introduction to Reinforcement Learning. Cambridge, MA, USA: MIT Press, 1998. [37] D. Precup, R. S. Sutton, and S. Dasgupta, ‘‘Off-policy temporaldifference learning with function approximation,’’ in Proc. 7th. Int. Conf. Mach. Learn., 2001, pp. 417–424. [38] J. Kennedy and R. Eberhart, ‘‘Particle swarm optimization,’’ in Proc. ICNN Int. Conf. Neural Netw., vol. 4, 1995, pp. 1942–1948. [39] M. Carolina Jara Ten Kathen, I. Jurado Flores, and D. Gutierrez Reina, ‘‘A comparison of PSO-based informative path planners for autonomous surface vehicles for water resource monitoring,’’ in Proc. 7th Int. Conf. Mach. Learn. Technol. (ICMLT). New York, NY, USA: Association for Computing Machinery, Mar. 2022, pp. 271–276, doi: 10.1145/3529399.3529442. DAME SECK DIOP was born in Louga, Senegal, in 2000. He received the M.S. degree in electronics, robotics and automation engineering from the University of Seville, Spain, in 2023, where he is currently pursuing the Ph.D. degree in automatic, electronic, and telecommunication engineering with the Department of Electronic Engineering. His research interests include reinforcement learning, multi-objective optimization, machine learning for robotics applications, autonomous vehicles control, and artificial vision. SAMUEL YANES LUIS was born in Tenerife, Spain, in 1997. He is currently pursuing the Ph.D. degree with the University of Seville. He is also a Pre-Doctoral Researcher and a Trainee Professor with the Department of Electronic Engineering, University of Seville. He has been a Visiting Researcher with the Institute of Navigation and Communications, German Aerospace Center, and the Department of Computer Science, State University of Milan. His research interests include the development of optimization algorithms for decision problems with autonomous vehicles, the estimation of environmental models using machine learning and deep learning, and the development of aquatic surface robots for hydrological monitorization. MANUEL PERALES ESTEVE received the Ing. Industrial and Doctor Ing. Industrial degrees from the University of Seville, Seville, Spain, in 1995 and 2002, respectively. He joined the Department of Electronic Engineering, University of Seville, in 1996, as a Researcher and has been an Assistant Professor, since 1998. He is currently an Associate Professor with the Department of Electronic Engineering. His current research interests include active power filters, modulation techniques, power systems, and medical applications of digital systems. SERGIO L. TORAL MARÍN was born in Rabat, Morocco, in 1972. He received the M.S. and Ph.D. degrees in electrical and electronic engineering from the University of Seville (US), Spain, in 1995 and 1999, respectively. He is currently a Full Professor with the Department of Electronic Engineering, US. He is actually the author or coauthor of 95 articles in major international peer-reviewed journals (with JCR impact factor) and of 111 articles in well-established international conferences and workshops. His research interests include the cooperative deployment of autonomous vehicles, intelligent transportation systems, embedded systems and artificial intelligence, and machine learning and deep learning algorithms. DANIEL GUTIÉRREZ REINA received the B.E. degree (Hons.) in electronic engineering, the M.S. degree in electronics and telecommunications, and the Ph.D. degree (Hons.) in electronic engineering from the University of Seville, Seville, Spain, in 2009, 2011, and 2015, respectively. He was an Assistant Professor with Loyola University, from October 2018 to April 2019. He has been a Visitor Researcher with Liverpool John Moores University, U.K.; the Free University of Berlin, Germany; the Colorado School of Mines, USA; and Leeds Beckett University, U.K. He is currently an Associate Professor with the Departamento de Ingeniería Electrónica, Universidad de Sevilla. He has published about 60 articles in JCR journals with impact factor. His current research interests include the application of meta-heuristic, machine learning, and deep algorithms to solve monitoring problems using autonomous systems. He is part of the editorial board of several journals, such as International Journal of Distributed Sensor Networks (SAGE), Electronics (MDPI), and Future Internet (MDPI), organizing numerous SI for these journals. 75576 VOLUME 12, 2024