TOWARDS ANNOTATION-FREE FISH MONITORING IN AQUATIC ENVIRONMENT BASED ON VISION TRANSFORMERS

TOWARDS ANNOTATION-FREE FISH MONITORING IN AQUATIC ENVIRONMENT BASED ON VISION TRANSFORMERS P.L. Cebri´ an*,G. Rosa,G. Obelleiro,J.A. Vi˜ nuelas,M. Chavarr´ ıas, and C. S´ anz All authors are with CITSEM. Universidad Polit´ ecnica de Madrid, 28040, Madrid, Spain. Email: [email protected] Introduction Monitoring fish behavior in aquaculture environments is essential for welfare assessment, early detection of anomalies, and production efficiency. However, existing computer vision methods for fish monitoring tasks often rely on large annotated datasets, which are scarce in underwater context due to the time-intensive effort of manual labeling. This work presents a fully unsupervised pipeline for instance-level fish segmentation and skeletal pose estimation. These pose representations enrich the object-level information beyond the spatial information provided by the bounding boxes in object detection, and also support robust individual tracking and individual behavioral analysis. Unlike fish segmentation methods based on static background, this approach works under moving camera conditions. Thus, this work contributes to scalable, annotation-free monitoring systems in precision aquaculture. Materials and methods In this work, it is proposed a methodology based on a modular pipeline, illustrated in Figure 1, which comprises three stages designed to transform a video frame into a skeletal pose estimation of individual fish based on keypoints head, body, and tail. Figure 1: Block-diagram representing method stages pipeline. The first stage uses Grounding DINO (Liu et al, 2024), an extension of DINOv2, which learns visual features without labeled data. It integrates language models to associate textual prompts (e.g., ”fish”) with image regions, enabling multi-instance object segmentation without task-specific training. Once fishes are detected, the second stage employs the pre-trained sam vit h variant of the Segment Anything Model (SAM) (Kirillov et al., 2023) for instance segmentation, generating accurate binary masks for each fish detected by DINO using the bounding boxes and the centroid points as promtps, without requiring specific domain annotations. Finally, the third stage involves skeletonizing the binary mask to create a one-pixel-thick representation of the fish, which is then treated as a graph. The longest path in the graph defines the main axis, with keypoints for the head and tail at its endpoints. Optical flow is subsequently applied to refine head-tail assignment based on movement between frames. The body keypoint is located near the centroid of the fish, and two midpoints along the head-body and tail-body segments are computed to more accurately represent its shape. These five keypoints form a reliable basis for tracking and analyzing the shape over time. This skeletal pose estimation goes beyond traditional object detection and tracking methods, which typically rely on bounding boxes, by enabling the analysis of fish orientation, social interactions, and behavioral responses to environmental stimuli through keypointbased identification. 1 Results The proposed unsupervised pipeline was evaluated on three publicly available underwater video datasets representing diverse aquatic scenarios. The main evaluation was conducted on the OLAQUA SINTEF dataset, which contains real-world deployment footage from large-scale aquaculture offshore facilities, characterized by camera movements, dynamic backgrounds, high-density fish population, and low image quality conditions. Additional qualitative assessment was performed on two additional datasets: one with nonconsecutive frames of Etroplus maculatus, and Seagrass (Ditria et al.), both recorded in the wild with a fixed camera. Although the latter dataset includes ground truth annotations, only qualitative results are reported for consistency (see Figure 2). Visual inspection of all these datasets confirms the generalization of the pipeline across environments, species, and recording conditions without requiring annotated training data for computer vision deep learning models. Figure 2: Skeletal pose estimation results. Top left: SINTEF. Bottom left: Seagrass. Right: Etroplus maculatus In conclusion, this work presents a novel, fully unsupervised pipeline for fish instance segmentation and skeletal pose estimation in non-fixed underwater camera videos, using recent vision transformer models (DINOv2 + SAM), followed by a skeltonization module. This approach eliminates the need for manual annotations while producing rich spatiotemporal descriptors of fish posture (head-body-tail) that enable fine-grained behavioral insights. By reducing the annotation burden and enabling pose-aware fish monitoring, this work contributes to scalable, annotation-free monitoring in precision aquaculture. Kirillov, A., et al. (2023). Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 4015-4026). Liu, S.et al. (2024, September). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. https://doi.org/10.1007/978-3-031-72970-63 Hansen, O. M. et al. (2025). OLAQUA: A large aquaculture robotics dataset. arXiv preprint arXiv:2504.01790. https://doi.org/10.48550/arXiv.2504.01790 Medeiros, G., Hansen, O. M., Øvsthus, J., Sture, E., et al. (2024). Pose annotated video dataset of ornamental fish in aquaculture tanks [Dataset]. Data in Brief. https://doi.org/10.1016/j.dib.2024.111007 Ditria, Ellen M. et al. Sebastian (2021): Annotated video footage for automated identification and counting of fish in unconstrained marine environments [dataset]. PANGAEA, https://doi.org/10.1594/PANGAEA.926930 Acknowledgement This work has been funded by Ministerio de Ciencia e Innovaci´ on, ANEMONA project (TED2021-130225A-100). 2