VAT4MAV: Visual Active Tracking for Micro Aerial Vehicles

VAT4MAV: Visual Active Tracking for Micro Aerial Vehicles Alessandro Rocco Denarda Alberto Dionigi Gabriele Costante Abstract—Visual active tracking has become an increasingly prominent topic in robotics because of its central role in applications such as assistive robotics and surveillance. Unlike passive tracking, active methods tightly couple perception and control: the system not only detects the target but also actively maneuvers the platform to keep it in view. While much of the literature focuses on ground robots, solutions for aerial platforms remain scarce. To address these issues, in this extended abstract we present our ongoing work on visual active tracking for micro aerial vehicles, focusing on the comparative evaluation of two complementary design strategies. The first adopts a modular approach, in which perception and control components are developed separately and then integrated into a pipeline; the second follows an End-to-End approach, where a single deep policy directly maps visual observations to control commands through reinforcement learning. By contrasting these paradigms, our goal is to assess their relative strengths, limitations, and applicability to real-world deployment. I. INTRODUCTION Micro Aerial Vehicles (MAVs) have attracted increasing interest due to their agility, light weight, and low cost, which make them well-suited for a wide range of robotic applications. These operational scenarios demand fast, reliable perception and control loops to guarantee task success. In this work we focus on the Visual Active Tracking (VAT) problem, where a tracker vehicle must keep a moving target visible and centered within the camera field-of-view (FoV). Active tracking tightly couples perception and control so that the platform actively maneuvers to maintain the target within the FoV. This makes VAT substantially more challenging, since it requires a direct mapping from high-dimensional visual observations to real-time control actions. Classical vision-based control has been mainly addressed through position-based visual servoing (PBVS) and imagebased visual servoing (IBVS) [1]. In PBVS, image features are processed to extract the 3D pose of the target, which is then used to generate control inputs. IBVS instead computes the control error directly in image space, avoiding explicit pose reconstruction but still requiring depth cues. More recently, the rise of Deep Reinforcement Learning (DRL) has inspired the development of End-to-End controllers capable of learning robust VAT policies from data. These methods directly couple perception and control showing impressive results in ground robots and constrained scenarios [2], [3]. However, their application to aerial platforms remains limited: existing works often rely on simplifying assumptions, such as discrete action spaces [4] or restricted target types, which compromise robustness and generalization. The authors are with the Department of Engineering, University of Perugia, 06125 Perugia, Italy {alessandro.denarda}@collaboratori.unipg.it, {alberto.dionigi, gabriele.costante}@unipg.it. Fig. 1. Methodology Overview. To overcome these limitations, in this work we investigate two complementary strategies for VAT of MAVs Fig. 1. The first follows a modular design, where perception and control components are trained or tuned separately. The second adopts an End-to-End approach, in which a single policy maps raw visual observations directly into continuous control actions through DRL. By contrasting these two paradigms, we aim to assess their respective strengths and limitations in terms of robustness, generalization, and deployability on lightweight aerial platforms. II. PROJECT MOTIVATION AND METHODOLOGY Our research is driven by practical constraints and the goal of bridging the gap between simulation and field-ready VAT solutions for lightweight aerial platforms. MAVs face tight payload, compute, and energy limits, which pushes us toward compact, latency-aware perception and control pipelines. By prioritizing lightweight and integrated solutions, we aim to enable VAT in realistic outdoor missions such as infrastructure monitoring, environmental exploration, or search-andrescue, where external localization may be unavailable. Purely model-based controllers such as PID or LQR remain attractive for their predictability and certified stability margins [1], but they struggle in scenarios with strong nonlinearities, occlusions, or unmodeled dynamics. Conversely, DRL-based controllers can extract richer visual features and adapt to disturbances but often lack theoretical guarantees [2], [3]. Instead of committing to a single design, our work systematically benchmarks two complementary strategies. On one hand, modular pipelines couple specialized perception networks with classical controllers, offering interpretability and established stability properties. On the other hand, endto-end learning approaches integrate perception and control into a single trainable policy, with the potential to extract richer features and adapt to complex disturbances. In our setting, both the tracker and the target are freeflying MAVs operating in full 3D. The tracker uses only the incoming stream of rectified stereo frames to drive the 2025 I-RIM Conference October 17-19, Rome, Italy ISBN: 9788894580570 10.5281/zenodo.17629830 195 TABLE I EXPERIMENTAL RESULTS IN THE BOX ENVIRONMENTS AND IN THE PHOTOREALISTIC SCENARIOS (URBAN, PARK AND OFFICE) Method Experimental Scenarios and Metrics Box Environments Urban Park Office PθPφPρPcPθPφPρPcPθPφPρPcPθPφPρPc Modular LQR 0.27 0.20 0.20 0.22 0.10 0.08 0.07 0.09 0.37 0.27 0.27 0.30 0.11 0.08 0.09 0.09 Modular PID 0.65 0.65 0.55 0.62 0.57 0.56 0.46 0.53 0.94 0.93 0.76 0.88 0.78 0.75 0.67 0.73 End-to-End DRL 0.94 0.94 0.86 0.91 0.94 0.94 0.91 0.93 0.95 0.95 0.92 0.94 0.95 0.95 0.86 0.92 control loop and generate continuous commands that satisfy the tracking objective. In the modular setting, perception and control are treated as distinct blocks. At the perception layer, state-of-the-art vision networks are employed: a YOLOv12 detector [5] provides robust target acquisition under challenging illumination or scale variations, while a SiamRPN++ tracker [6] maintains a stable target lock at high frame rates. These components are complemented by stereo depth cues, which provide reliable metric distance estimates and allow the recovery of relative 3D target pose with respect to the tracker. The resulting perception outputs feed into different control policies. Three families of controllers are considered, i.e., PID, LQR, and DRL controller. This modular pipeline is designed to emphasize clarity of function separation, making it possible to benchmark how well different control strategies exploit the same vision-based input. The second methodological direction is a fully End-toEnd VAT policy, in which perception and control are learned jointly within a single deep network. Here, the agent directly receives rectified stereo image sequences and produces continuous thrust and angular-rate commands. We adopt an asymmetric actor–critic scheme [3]. End-to-End learning is expected to exploit richer visual cues and capture nonlinear interactions that modular designs might miss. At the same time, it introduces challenges in training stability, sim-to-real transfer, and interpretability. III. EXPERIMENTS Our approach is evaluated on two environment types: Box Environments, which resemble the training scenes but with different room shapes, object layouts, and textures; and more complex, photo-realistic environments (Urban, Park, and Office), designed to test D-VAT’s generalization capabilities. To evaluate VAT performance, we adapted standard tracking metrics to a 3D setting, expressing the target’s position relative to the tracker in spherical coordinates (ρ, θ, ϕ) aligned with the tracker’s body frame. The metrics include: Distance Score (ability to maintain the desired distance ρ∗), Elevation Score (vertical alignment of the target with the center of the FoV, θ∗), and Azimuth Score (horizontal alignment with the center of the FoV, ϕ∗).ATotal Score is computed as the average of these three metrics, reaching 1 for perfect tracking: ˜ Pc(k) = ˜ Pρ(k) + ˜ Pθ(k) + ˜ Pϕ(k) 3. Scores are then averaged over the episode time and across 20 runs per scenario to provide a robust performance measure: Pm=1 20Ne 20 X i=1 Nc−1 X k=0 ˜ P(i) m(k), m ∈ {ρ, θ, ϕ, c}, where Ncis the number of samples per episode and ˜ P(i) m(k) indicates the performance of the i-th run. Preliminary results are summarized in Table I. All investigated approaches are capable of accomplishing the VAT task. End-to-End DRL pipelines generally achieve higher performance in simulation. However, their effectiveness drops in real-world conditions due to the difficulty of achieving zero-shot sim-to-real transfer. In contrast, modular policies exhibit better transfer from simulation to reality, despite lower performance in simulation. These findings suggest two complementary directions for future work: improving the robustness of End-to-End approaches to reduce the sim-toreal gap, and enhancing the performance of modular pipelines to better leverage their reliable real-world transfer. ACKNOWLEDGMENTS G. Costante acknowledges funding by PNRR-M4C2 - I1.4MUR Call for propoisal n. 3138 del 16-12-2021, modificato con DD n. 3175 del 18-122021, DD di concessione del finanziamento n.1033 del 17-06-2022 - Project title: “Visual Active Tracking for Micro Aerial Vehicles - VAT4MAV” finanziato nell’ambito del Bando a Cascata pubblicato dallo SPK6 – “Connected and Autonomous Vehicle (CAV)” - Università degli studi di Modena e Reggio Emilia - Programma “Sustainable Mobility Center (Centro Nazionale per la Mobilità Sostenibile – CNMS)” “Potenziamento strutture di ricerca e creazione di "campioni nazionali di R&S" su alcune Key Enabling Technologies”, Funded by European Union – NextGenerationEU - CN00000023 - CUP E93C22001070001. REFERENCES [1] B. Siciliano, O. Khatib, and T. Kröger, Springer handbook of robotics. Springer, 2008, vol. 200. [2] A. Devo, A. Dionigi, and G. Costante, “Enhancing continuous control of mobile robots for end-to-end visual active tracking,” Robotics and Autonomous Systems, vol. 142, p. 103799, 2021. [3] A. Dionigi, A. Devo, L. Guiducci, and G. Costante, “E-vat: An asymmetric end-to-end approach to visual active exploration and tracking,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 4259–4266, 2022. [4] W. Zhao, Z. Meng, K. Wang, J. Zhang, and S. Lu, “Hierarchical active tracking control for UAVs via deep reinforcement learning,” Applied Sciences, vol. 11, no. 22, 2021. [5] Y. Tian, Q. Ye, and D. Doermann, “Yolov12: Attention-centric real-time object detectors,” arXiv preprint arXiv:2502.12524, 2025. [6] B. Li, W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan, “Siamrpn++: Evolution of siamese visual tracking with very deep networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 4282–4291. 196