ARTEMIS: animal recognition through enhanced multimodal integration system
Full text
Vol.:(0123456789) International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 https://doi.org/10.1007/s13042-025-02602-3 ORIGINAL ARTICLE ARTEMIS: animal recognition throughenhanced multimodal integration system EdoardoFazzari1,2,3· DonatoRomano1,2· FabrizioFalchi1,3· CesareStefanini1,2 Received: 28 October 2024 / Accepted: 3 March 2025 / Published online: 18 March 2025 © The Author(s) 2025 Abstract This paper introduces Animal Recognition Through Enhanced Multimodal Integration System (ARTEMIS), a transformerbased framework designed for multilabel animal action recognition by fusing video, image, and textual modalities. ARTEMIS utilizes state-of-the-art captioning and language models, such as BLIP2 and Llama 3, to generate textual descriptions from video frames, which are input to the model, significantly enhancing its performance unlikely previous results that do not consider this modality. Through comprehensive ablation studies, we explore the contribution of various model components and propose optimization strategies, including genetic algorithms and reinforcement learning, to dynamically adjust ensemble weights. Our feature alignment techniques-using contrastive and cosine similarity losses-further improve multimodal integration. Evaluations on the Animal Kingdom dataset, which includes 30,100 clips across 140 action classes, demonstrate that ARTEMIS achieves a new state-of-the-art mAP of 79.82, outperforming existing methods. The combination of multimodal fusion and ensemble strategies makes ARTEMIS a robust solution for complex animal action recognition tasks. The code of our fusion method is available at https:// github. com/ edofa zza/ ARTEM IS. Keywords Animal action recognition· Multi-modal deep learning· Ensemble· Genetic algorithm· Reinforcement learning· Contrastive learning 1 Introduction Animals exhibit a wide range of behaviors, from affectionate and defensive actions to feeding, movement, and aggression. These behaviors are of great interest in ethological studies[1], as they can be indicative of an animal’s health and well-being[2]. By associating specific actions with health conditions, researchers can potentially identify pathological behaviors or detect anomalies in wildlife and livestock[3, 4]. This is particularly valuable for environmental monitoring and improving animal welfare[5]. In pursuit of understanding these behaviors, researchers have published datasets[6] and developed methodologies for extracting meaningful information from sensor data, images, and videos[7, 8]. However, much of this research has been limited to individual species and a narrow set of actions[9, 10]. The release of the Animal Kingdom[11] dataset marked a significant shift in this field. Animal Kingdom introduced a dataset encompassing a wide variety of animal species and reorganized actions in a way that allows for cross-species application. The hope is that models trained on this benchmark dataset will generalize across species, paving the way for the development of a foundational model for animal action recognition. Such a model could have widespread applications in agriculture[12], (neuro-)ethology[6], and other research fields, significantly reducing the need for retraining or fine-tuning for specific species. * Edoardo Fazzari edoardo.fazzar[email protected] Donato Romano [email protected] Fabrizio Falchi [email protected] Cesare Stefanini cesare.stef[email protected] 1 The BioRobotics Institute, Sant’Anna School ofAdvanced Studies, Viale Rinaldo Piaggio, 56025Pontedera, Italy 2 Department ofExcellence inRobotics andAI, Sant’Anna School ofAdvanced Studies, Piazza Martiri della Libertà, 56127Pisa, Italy 3 Institute ofInformation Science andTechnologies, National Research Council ofItaly, via G. Moruzzi, 56124Pisa, Italy
5878 International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 Deep learning has emerged as a key tool for solving the problem of animal action recognition. Researchers initially employed Convolutional Neural Networks[13], but more recent efforts have shifted towards multimodal deep learning approaches[14]. However, current models applied to the Animal Kingdom dataset have struggled to achieve satisfactory performance, indicating that further research is required. In response to these challenges, we propose a novel multimodal approach for recognizing animal actions in videos, images, labels, and video descriptions, which we call ARTEMIS (Animal Recognition Through Enhanced Multimodal Integration Systems), as depicted in Fig.1. ARTEMIS consists of two branches, where the information is fused in two stages. In the first stage, each branch undergoes an initial fusion between two modalities via simple concatenation: one branch fuses video data with the corresponding video descriptions, while the other combines averaged CLIP embeddings from input frames with label embeddings. The second stage of fusion occurs by integrating the two branches through a Transformer layer, followed by a fully connected layer for final action recognition. We conducted an extensive ablation study to determine which components-such as the sequence processing layers for video descriptions and residual connections-were most effective in improving performance on the Animal Kingdom dataset. To further enhance the results, we employed both average and weighted ensembles. One challenge when using weighted ensembles is determining the appropriate weights. To address this, we utilized genetic algorithms and reinforcement learning. Genetic algorithms are well-suited for optimizing a fitness function and have proven effective through evolutionary mechanisms[15], though they do not guarantee the discovery of the optimal solution[16]. On the other hand, reinforcement learning has a strong track record in improving deep learning results[17], with applications spanning diverse fields such as physics[18] and video game playing[19]. Another critical aspect of our research was generating video descriptions as input to our models. The Animal Kingdom dataset does not provide textual descriptions of the actions occurring in each clip, only the video clips and frames. Manually obtaining this information from experts is impractical, as it is not automated or scalable. Therefore, we used deep learning models, leveraging captioning techniques and large language models (LLMs), to automatically generate these descriptions. The main contributions of this paper are as follows: 1. We introduce ARTEMIS, a novel multimodal architecture for animal action recognition that achieves state-of-the-art performance. Unlike previous approaches, ARTEMIS integrates textual descriptions of videos into its processing pipeline. 2. We utilize large language models (LLMs) to generate textual descriptions of videos and augment the Animal Kingdom dataset with these descriptions. 3. We introduce a set of optimization strategies, including ensemble methods, genetic algorithms, and reinforcement learning, to further improve model performance. 4. All generated descriptions and source code are made publicly available to support reproducibility and further research. The remainder of this paper is organized as follows: Sect.2 reviews related work on animal action recognition (AAR). In Sect. 3, we detail the network architecture, fusion modules, additional loss functions for feature alignment, and optimization strategies. Section4 provides experimental evaluations and discusses the results, and conclusions are drawn in Sect.5. Fig. 1 Overview of ARTEMIS for multimodal multilabel animal action recognition. It has three components: a video-text fusion branch, an image-text fusion branch and a multimodal Transformer decoder
5879International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 2 Related work Animal Action Recognition (AAR) initially emerged as an extension of Human Action Recognition (HAR)[20], with deep learning strategies from HAR being directly adapted to this new domain. Two primary approaches dominated early AAR research[21]: pose estimation of animal body parts, followed by the association of these movements to specific actions through classical machine learning or deep learning methods; and the use of sensor data for action recognition. The first approach significantly advanced animal pose estimation techniques, although these developments were often driven by progress in human pose estimation. Tools such as SLEAP[22], DeepLabCut[7], DeepEthogram[23], and MARS [8] have been instrumental in enabling researchers to study animal behavior for purposes ranging from improving farming welfare-such as detecting feeding and fighting behaviors in pigs[24]-to neuroethological investigations, like studying the effects of diseases[25]. On top of pose estimation, machine learning models such as Support Vector Machines (SVMs), Long Short-Term Memory networks (LSTMs), or convolutional neural networks are commonly applied to identify specific behaviors[26, 27]. While pose estimation-based AAR begins with visual input from videos or images, sensor-based AAR relies on temporal data captured by gyroscopes, GPS, and accelerometers[28]. This sensor data is fused and processed using machine learning algorithms, primarily for recognizing actions in livestock such as cows and sheep[29]. However, the use of sensors comes with significant limitations, particularly for animals in the wild. Sensor readings can be affected by external factors such as temperature, humidity, or vibrations, leading to unreliable data[30]. Moreover, improper placement of sensors can result in poor data quality or misinterpretation of actions. Additionally, wearable sensors may be intrusive or uncomfortable, potentially interfering with the animals’ natural behaviors[31]. In multimodal learning, one possible fusion strategy involves combining visual and textual data. This is widely used in tasks such as image captioning[32], visual question answering[33], and generating images from textual descriptions[34]. In AAR, textual information has also been leveraged to enhance recognition accuracy. For instance, MSQNet[14] employs text embeddings of action labels generated by CLIP to improve the model’s ability to recognize actions in the Animal Kingdom dataset’s multilabel action recognition task[11]. MSQNet outperformed previous models that relied solely on video data. Further advancements were made with MambaMSQNet[35], a more efficient version of MSQNet that uses fewer parameters and computational resources while achieving superior mean Average Precision (mAP). Even if the Animal Kingdom is the biggest and more complete dataset for animal action recognition, spanning 140 distinct actions over 850 different species, making it being considerable as a benchmark dataset for AAR it is not the only one available. Quite recently, other research efforts in providing public dataset for this topic became available, BaboonLand and MammalNet. The BaboonLand dataset[36] is a collection of mini-scenes limited on a single species, baboons, performing 12 distinct actions and an additional actions for marking videos where no animal is visible. The MammalNet dataset[37] focused only on mammals and their most common behaviors, limiting the number of species to 173, way less than Animal Kingdom, and only 12 actions. While Animal Kingdom is the largest and most comprehensive dataset for animal action recognitioncovering 140 distinct actions across 850 species, establishing itself as a benchmark dataset for AAR, it is not the only one available. Recently, other research efforts have provided additional public datasets for this topic, notably BaboonLand and MammalNet. The BaboonLand dataset[36] is a focused collection of mini-scenes featuring a single species, baboons, performing 12 distinct actions, along with an additional category for frames where no animal is visible. On the other hand, the MammalNet dataset[37] concentrates exclusively on mammals and their most common behaviors, reducing its scope to 173 species and only 12 actions, significantly fewer than Animal Kingdom. 3 Proposed framework In this section, we design the ARTEMIS model. We begin by describing the architecture and specific components of our model, highlighting how the model operates to achieve effective action recognition. Following that, we explain the process used to generate video description for each video clip and what we used to further improve ARTEMIS. 3.1 ARTEMIS details We present ARTEMIS, the Animal Recognition Through Enhanced Multimodal Integration System, designed for multilabel multimodal action recognition in animals. As illustrated in Fig.1, the model is composed of three key components: (1) a video-text fusion branch, which processes and integrates video information from a video encoder with embeddings of the video description; (2) an image-text fusion branch, which combines embeddings from individual frames with action-specific class embeddings; and (3) a multimodal Transformer decoder, which performs
5880 International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 late-stage fusion of the two branches and generates the final predictions. 3.1.1 Video‑text fusion branch In the video-text fusion branch, video data is processed to extract frame-level features, which are subsequently fused with external textual information in the form of summary embeddings. This branch begins with video input consisting of t frames, where each frame I is represented as a tensor of size C×H×W , with C channels, and spatial dimensions H and W . These frames are passed through a Timesformer backbone[38], which is a transformer-based architecture for spatiotemporal learning from video data. The Timesformer extracts frame-level feature representations, X∈ℝT×D , where T×D represents the dimension of the latent feature space. To align the first dimension with the expected temporal resolution in subsequent operations, we apply adaptive average pooling. This operation down-samples T to t through: where 𝐗�∈ ℝ t×D represents a compressed feature map that reduces the temporal size while maintaining the final dimensions intact. Simultaneously, textual data corresponding to the video is processed. The textual video descriptions are embedded into a latent space through CLIP text encoder[39], producing embeddings S∈ ℝ E , where E is the dimension of the embedding space. Depending on the model configuration, this embedding may be transformed using a recurrent. If a BiLSTM is employed, the text embedding becomes: where the output maintains the same embedding dimension E . Other recurrent options like biGRU or 1D convolution could similarly be used for the summary transformation. This textual embedding is then expanded across the temporal dimension to match the length t of the video feature sequence: The expanded textual embeddings are concatenated with the pooled video features 𝐗′ , resulting in a concatenated tensor Z∈ℝt×(D+E) . This tensor is then passed through a linear transformation: where W1 ∈ℝ (D+E)×E , producing the output Hv ∈ℝ t×E , which represents the fused video-text features. (1) 𝐗� = AdaptiveAvgPool1D (X ⊤,t ) ⊤, (2) 𝐒�=BiLSTM(S), (3) 𝐒� ⊗1 t ∈ℝ t×E . (4) Hv =ZW1, A residual connection is used to preserve important information from the original Timesformer backbone and the sequence features. First, the original video features X are passed through a linear layer and averaged over time: where Wres1 ∈ℝ D×E transforms the backbone features into the appropriate dimensionality. Similarly, the fused videotext features Hv are linearly transformed into a residual form: where Wseq ∈ℝ E×E . These two residuals are summed together before applying the positional encoding: To preserve temporal context, a positional encoding is applied to Hv , which adds temporal positional information to each frame in the sequence. The positional encoding process is mathematically described as: where PE(t)∈ ℝ t×E is the positional encoding for the sequence length t . This addition enables the model to incorporate the temporal order of frames, critical for understanding the temporal dynamics of actions. This process produces a final fused representation for the video-text branch, Hfinal v =H pos v , which integrates both the spatiotemporal features from the video and the contextual information from the textual summary. The residual connections ensure that important features from both the backbone and sequence fusion are preserved throughout the process. 3.1.2 Image‑text fusion branch Parallel to the video-text branch, the image-text fusion branch operates on individual video frames. Each frame It is processed through the CLIP image encoder, producing a feature vector Ft ∈ℝ D , which shares the same embedding dimension E as the textual representation. To summarize the frame-level information, the features from all t frames are aggregated using a temporal average pooling operation: which results in a single vector F∈ℝD that captures the visual content across the entire video sequence. At the same time, a separate CLIP text encoder processes the set of action labels associated with the video. These action labels are embedded into the same space as (5) Rbackbone =AvgPool(XWres1), (6) Rsequence =HvWseq, (7) Rtotal =Rbackbone +Rsequence, (8) Hvpos =R total + PE ( t ) , (9) F = 1 t t ∑ i=1 Fi ,
5881International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 the image embeddings, yielding a matrix L∈ ℝ K×E , where K represents the number of action labels, and each row corresponds to the embedding of a specific label. To perform fusion, the image embedding F is repeated K times and concatenated with the corresponding action label embeddings: A linear transformation then projects this concatenated representation into the shared embedding space E : where W2 ∈ℝ (D+E)×E . The result, Hq ∈ℝ K×E , is a fused representation that encapsulates both visual and textual information about the actions. To enhance the expressiveness of the model, we optionally introduce a residual connection for the image features. The image embedding F , repeated across all K action labels, is linearly transformed: This residual is then added to the fused image-text representation, resulting in the final output of this branch: This image-text fusion branch thus integrates action-level textual information with visual representations of the video, with residual connections used to enhance the fusion process. 3.1.3 Late fusion andrecognition In the late fusion stage, the outputs from the two fusion branchesHfinal v ∈ℝt× E from the video-text fusion branch and Hfinal q ∈ℝK× E from the image-text fusion branch-are combined to form a comprehensive multimodal representation of the video. First, the concatenated outputs of the two branches are passed into a Transformer layer[40], with Hfinal v acting as one set of tokens and Hfinal q as another. This concatenation ensures that the model can learn complex dependencies between the temporal features of the video, the descriptive text, the global image features, and the action-specific textual information: The Transformer layer applies multi-head self-attention, where each head learns distinct attention patterns between the concatenated inputs. The self-attention mechanism computes an attention score for each pair of tokens, (10) Q = concat (L , F⊗1 K )∈ℝ K×(D+E). (11) Hq =QW2, (12) Fresidual =FW res2 ∈ℝ K×E. (13) H final q =Hq+Fresidual . (14) Hconcat =concat(H final v ,H final q )∈ℝ(t+K)×E . allowing the model to focus on the most relevant parts of the multimodal inputs: where Q , K , and V are the query, key, and value matrices derived from the concatenated inputs Hconcat . The output of the multi-head attention is then passed through a feed-forward network, which further refines the fused representation. The final output of the Transformer layer, Hfused ∈ℝ K×E , integrates in this way the information from both the video-text and image-text branches. At this stage, two residual connections are incorporated into the fused representation Hfused to enhance the flow of information and mitigate potential degradation of the learned features. These residuals correspond to the final outputs of the video-text fusion branch Hfinal v and the imagetext fusion branch Hfinal q . The addition of these residuals ensures that crucial multimodal features from both branches are preserved in the final fused representation. Formally, the updated fused representation 𝐇′ fused is given by: where Wj ∈ℝ t×K is a learnable weight matrix that aligns the dimensionality of Hfinal v with that of Hfused , ensuring compatibility during the fusion process. This representation 𝐇′ fused is then used to predict the occurrence of specific actions in the video. A final fully forward layer is applied to each of the K action classes, producing a score for each action: where Wi ∈ℝ E is the weight matrix for action i , and bi is the corresponding bias term. This enables the model to output a probability distribution over all potential actions, completing the recognition task. 3.2 Obtaining video descriptions One of the modalities used in ARTEMIS is a textual description that summarizes the input video. Since the Animal Kingdom dataset does not provide this information, we generated the descriptions ourselves. Given an input video I with frames I1,…,IK , each frame is passed through a captioning model, producing individual captions c1,…,cK , collectively denoted as C . To generate these captions, we used BLIP2[41] with a maximum caption length of 50 and a beam search of size 4. The set C provides frame-specific descriptions, but does not capture a holistic summary of the video. To create a (15) A =softmax � QK⊤ √E�, 𝐇� fused =Hfused +H final q +H final v Wj , (16) yi =𝐇 � fused W i +b i,
5882 International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 concise video description, C is reduced to a subset C ’, containing only unique captions. We then construct a query Q by appending the text: “Can you write a very short sentence summarizing the information below, focusing on the actions of the animals?” to C′ . This query is fed into LLaMA 3-8B[42], a pretrained Large Language Model, which generates the final summary used as the video description input for ARTEMIS. 3.3 Feature alignment A common approach in multimodal learning is to encourage the alignment or similarity of features across modalities within a shared embedding space. This helps to ensure that representations from different modalities are semantically aligned[43]. In our model, we aimed to align the features of the video-text fusion branch ( fb1 ) with those of the imagetext fusion branch ( fb2 ). To achieve this, we applied the techniques: a contrastive loss, a cosine similarity loss and Canonical Correlation Analysis (CCA). The contrastive loss[44] is designed to minimize the distance between semantically similar pairs and maximize the distance between dissimilar pairs in the embedding space. It is defined as: where y=1 if the pair shares the same labels, and y=0 otherwise. The distance metric ‘dist’ used is the Euclidean distance, with a margin set to 1. The cosine similarity loss[45] is employed to maximize the cosine similarity between features from different modalities. This method is particularly effective for aligning features in a shared embedding space and is defined as: Additionally, we employed Canonical Correlation Analysis (CCA)[46], a statistical technique used to find and maximize the correlation between two sets of variables by identifying linear transformations of the data that are maximally correlated. When integrated into a deep learning model, CCA can be formulated as a loss function. The goal is to maximize the correlation between the learned projections of the two modalities by minimizing the negative correlation. The CCA loss function is formulated as: (17) L contrastive =y ⋅ dist(f b1 ,f b2 ) +(1−y)⋅max(0, margin −dist(f b1 ,f b2 )) , (18) Lcosine =1−cos (fb1,fb2). (19) L CCA =− Tr(W T fb1 Σfb1fb2 Wfb2) √ Tr(WT f b1 Σfb1fb1 Wfb1)⋅ √ Tr(WT f b2 Σfb2fb2 Wfb2) , where Wfb1 and Wfb2 are the linear transformation matrices for fb1 and fb2 , respectively. Tr( ⋅ ) represents the trace of a matrix, while Σfb1fb2 , Σfb1fb1 , and Σfb2fb2 are the cross-covariance and auto-covariance matrices between and within the modalities. The minus sign is used to maximize the correlation between the two sets of features during the training of the neural networks. Only one of these alignment losses is applied at a time during training. Whichever is used, it is added to the binary cross-entropy loss, with the feature alignment loss weighted by a factor of 0.1 (0.0001 for CCA) before being combined with the primary loss function. 3.4 Ensemble strategies To improve results we decided to create ensemble models using the best configuration of ARTEMIS we obtained. Ensemble in deep learning is a popular approach to improve results and consists in using multiple trained architecture and combine to make a single prediction, producing better, more accurate and robust predictions than any single model on its own. In our experiments, three types of ensemble are defined: (1) an average ensemble giving equal weight to each model used; (2) a weighted ensemble where the weights assigned are obtained by using a genetic algorithm; (3) a weighted ensemble where the weights are dynamically decided based on the input given through the used on reinforcement learning. The first approach is defined as: where M is the number of models in the ensemble, fi(x) denotes the prediction of the i th model for a given input x . In case of weighted ensemble like in the other two cases, the formulation chages to: where wi is the weight assigned to the i th model, which is normalized such that ∑M i=1 w i = 1 . 3.4.1 Genetic algorithm weights To optimize the weights for the ensemble, we employed a genetic algorithm (GA)[47], which works by evolving a population of weight vectors over multiple generations. The process is described as follows: (20) F (x)= 1 M M ∑ i=1 fi(x) , (21) F (x)= ∑M i=1wifi(x) ∑ M i=1 w i ,
5883International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 • Initialization We randomly generated an initial population of 15 individuals. Each individual is represented as a vector of M entries, corresponding to the number of models, with random values between 0 and 1. The weights are normalized so that ∑M i=1 w i = 1 . • Evaluation The fitness of each individual is evaluated using the mean Average Precision (mAP) on a validation set, obtained through 5-fold cross-validation on the Animal Kingdom dataset. • Selection Tournament selection[48] is used, where 3 individuals are randomly chosen, and the one with the highest fitness is selected for reproduction. Additionally, elitism is employed to ensure that the top 2 individuals from the current generation are directly carried over to the next generation. • Crossover We used BLX𝛼 (Blend Crossover)[49] to generate new offspring from two parent individuals. For each gene i , the offspring (o1,o2) are created as: where p1,i indicates the parent p1 gene i and 𝛾 is defined as: where 𝛼 is a parameter controlling the blending range, set in our case to 0.5, and r is a random number sampled uniformly between 0 and 1. After generating the offspring, normalization is applied to ensure that the sum of the weights remains 1. • Mutation. Mutation is applied with a probability of 0.2, where each gene oi is perturbed by adding a Gaussiandistributed random value: where N(𝜇,𝜎) is a random sample from a normal distribution with mean 𝜇 and standard deviation 𝜎 . • Termination. The genetic algorithm terminates after 10 epochs. 3.4.2 Reinforcement learning weights Unlike the static weights obtained through the genetic algorithm, the reinforcement learning approach dynamically assigns weights based on the input. The goal is to create an agent that learns to predict a set of weights tailored to each input sample. To this end, we used several deep reinforcement learning algorithms, specifically Proximal Policy Optimization (PPO)[50]. (22) o1,i=𝛾p1,i+(1−𝛾)p2,i, (23) o2,i=𝛾p2,i+(1−𝛾)p1,i, (24) 𝛾=(1−2𝛼) ⋅ r+𝛼 (25) o� i =o i + N (𝜇,𝜎 ) The agent receives a single observation, which consists of the input video frames, merged across the frame and channel dimensions. The agent’s action is to output a set of weights, and the reward is calculated as the negative of the loss function for that input. Updates are processed after all the data in the validation set is viewed by the agent, and each update conducts a training of 3 epochs. Finally, the agent learns over a number of timesteps equal to the length of the validation set multiplied by 50. We used the default parameters of the reinforcement learning algorithms from the stable-baselines3 library[51], and a convolutional neural network for the policy. 4 Experimental results anddiscussion 4.1 Experiment settings Our models were trained using the Animal Kingdom dataset[11], which consists of 30,100 video clips depicting animals performing various actions, encompassing a total of 140 action classes. Training was divided into two phases: initially, the video encoder was kept frozen for 50 epochs, allowing only the other layers to train. Afterward, the video encoder was unfrozen and trained for an additional 150 epochs, or 200 epochs in the case of ARTEMIS and the top three ablation models, as well as for the contrastive loss analysis. All training was conducted using an NVIDIA A100 GPU. We used binary cross-entropy with logits as the loss function, with Adam[52] as the optimizer. Additionally, a cosine annealing scheduler with warm restarts was employed[53], resetting every 10 iterations. For data augmentation, we adopted the approach from Mamba-MSQNet[35], applying multi-scale cropping, horizontal flipping, random color jittering, and random grayscale effects. 4.2 Performance 4.2.1 Ablation study An ablation study was conducted to assess the impact of various residual connections in ARTEMIS and the choice of processing layers for video descriptions on performance. Table 1 outlines all tests performed, which included switching between no processing layer, biLSTM, GRU, or 1-D convolution for each possible combination of residual connections. Among all tested models, we identified three that outperformed the others: BEST1 ( R H final v , Rbackbone , Fresidual , conv), BEST2 ( R H final v , Rbackbone , conv), and BEST3
5884 International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 Table 1 Ablation study comparing different sequence processing layers for the text input to our model and various configurations of residual connections Text processing R H final v Rsequence Rbackbone R H final q Fresidual mAP – × × × × × 75.96 biLSTM × × × × × 76.23 GRU × × × × × 75.91 Conv × × × × × 76.17 – ✓ × × × × 76.04 biLSTM ✓ × × × × 75.87 GRU ✓ × × × × 75.71 Conv ✓ × × × × 76.52 – × ✓ × × × 74.96 biLSTM × ✓ × × × 76.19 GRU × ✓ × × × 75.96 Conv × ✓ × × × 75.21 – ✓ ✓ × × × 75.49 biLSTM ✓ ✓ × × × 76.51 GRU ✓ ✓ × × × 76.09 Conv ✓ ✓ × × × 75.78 – × × ✓ × × 76.29 biLSTM × × ✓ × × 76.46 GRU × × ✓ × × 76.07 Conv × × ✓ × × 76.30 – ✓ × ✓ × × 76.50 biLSTM ✓ × ✓ × × 76.15 GRU ✓ × ✓ × × 75.78 Conv ✓ × ✓ × × 76.57 (77.19) – × ✓ ✓ × × 75.13 biLSTM × ✓ ✓ × × 76.39 GRU × ✓ ✓ × × 75.78 Conv × ✓ ✓ × × 75.75 – ✓ ✓ ✓ × × 76.02 biLSTM ✓ ✓ ✓ × × 76.24 GRU ✓ ✓ ✓ × × 75.66 Conv ✓ ✓ ✓ × × 76.28 – × × × ✓ × 75.26 biLSTM × × × ✓ × 75.75 GRU × × × ✓ × 75.40 Conv × × × ✓ × 75.37 – ✓ × × ✓ × 75.78 biLSTM ✓ × × ✓ × 76.12 GRU ✓ × × ✓ × 75.93 Conv ✓ × × ✓ × 75.68 – × ✓ × ✓ × 73.54 biLSTM × ✓ × ✓ × 75.32 GRU × ✓ × ✓ × 74.83 Conv × ✓ × ✓ × 74.74 – ✓ ✓ × ✓ × 74.63 biLSTM ✓ ✓ × ✓ × 75.75 GRU ✓ ✓ × ✓ × 75.39 Conv ✓ ✓ × ✓ × 74.92 – × × ✓ ✓ × 75.85 biLSTM × × ✓ ✓ × 76.16 GRU × × ✓ ✓ × 75.63
5885International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 Table 1 (continued) Text processing R H final v Rsequence Rbackbone R H final q Fresidual mAP Conv × × ✓ ✓ × 75.92 – ✓ × ✓ ✓ × 76.50 biLSTM ✓ × ✓ ✓ × 76.15 GRU ✓ × ✓ ✓ × 75.78 Conv ✓ × ✓ ✓ × 76.57 (76.97) – × ✓ ✓ ✓ × 75.13 biLSTM × ✓ ✓ ✓ × 76.39 GRU × ✓ ✓ ✓ × 75.78 Conv × ✓ ✓ ✓ × 75.75 – ✓ ✓ ✓ ✓ × 76.02 biLSTM ✓ ✓ ✓ ✓ × 76.24 GRU ✓ ✓ ✓ ✓ × 75.66 Conv ✓ ✓ ✓ ✓ × 76.28 – × × × × ✓ 76.10 biLSTM × × × × ✓ 76.49 GRU × × × × ✓ 75.99 Conv × × × × ✓ 76.44 – ✓ × × × ✓ 76.18 biLSTM ✓ × × × ✓ 75.59 GRU ✓ × × × ✓ 75.68 Conv ✓ × × × ✓ 76.21 – × ✓ × × ✓ 74.73 biLSTM × ✓ × × ✓ 76.24 GRU × ✓ × × ✓ 75.39 Conv × ✓ × × ✓ 75.59 – ✓ ✓ × × ✓ 75.51 biLSTM ✓ ✓ × × ✓ 75.89 GRU ✓ ✓ × × ✓ 75.35 Conv ✓ ✓ × × ✓ 76.21 – × × ✓ × ✓ 76.19 biLSTM × × ✓ × ✓ 75.92 GRU × × ✓ × ✓ 76.01 Conv × × ✓ × ✓ 76.57 – ✓ × ✓ × ✓ 76.15 biLSTM ✓ × ✓ × ✓ 76.11 GRU ✓ × ✓ × ✓ 75.67 Conv ✓ × ✓ × ✓ 76.71 (77.33) – × ✓ ✓ × ✓ 74.89 biLSTM × ✓ ✓ × ✓ 76.03 GRU × ✓ ✓ × ✓ 75.73 Conv × ✓ ✓ × ✓ 75.90 – ✓ ✓ ✓ × ✓ 75.57 biLSTM ✓ ✓ ✓ × ✓ 76.14 GRU ✓ ✓ ✓ × ✓ 75.73 Conv ✓ ✓ ✓ × ✓ 76.16 – × × × ✓ ✓ 75.59 biLSTM × × × ✓ ✓ 76.04 GRU × × × ✓ ✓ 75.56 Conv × × × ✓ ✓ 76.03 – ✓ × × ✓ ✓ 75.54 biLSTM ✓ × × ✓ ✓ 76.23
5892 International Journal of Machine Learning and Cybernetics (2025) 16:5877–5892 32. Zhao D, Chang Z, Guo S (2019) A multimodal fusion approach for image captioning. Neurocomputing 329:476–485 33. Gong X, Mohan S, Dhingra N, Bazin J-C, Li Y, Wang Z, Ranjan R (2023) MMG-EGO4D: multimodal generalization in egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6481–6491 34. Xia W, Yang Y, Xue J-H, Wu B (2021) Tedigan: text-guided diverse face image generation and manipulation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 2256–2265 35. Fazzari E, Romano D, Falchi F, Stefanini C (2024) Selective state models are what you need for animal action recognition. Ecol Inf 25:102955 36. Duporge I, Kholiavchenko M, Harel R, Wolf S, Rubenstein D, Crofoot M, Berger-Wolf T, Lee S, Barreau J, Kline J etal (2024) BaboonLand dataset: tracking primates in the wild and automating behaviour recognition from drone videos. arXiv preprint arXiv: 2405. 17698 37. Chen J, Hu M, Coker DJ, Berumen ML, Costelloe B, Beery S, Rohrbach A, Elhoseiny M (2023) Mammalnet: a large-scale video benchmark for mammal recognition and behavior understanding. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 13052–13061 38. Bertasius G, Wang H, Torresani L (2021) Is space-time attention all you need for video understanding? In: ICML, vol 2, p 4 39. Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J (2021) Learning transferable visual models from natural language supervision. In: International conference on machine learning. PMLR, pp 8748–8763 40. Vaswani A (2017) Attention is all you need. Adv Neural Inf Process Syst 41. Li J, Li D, Savarese S, Hoi S (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: International conference on machine learning. PMLR, pp 19730–19742 42. Dubey A, Jauhri A, Pandey A, Kadian A, Al-Dahle A, Letman A, Mathur A, Schelten A, Yang A, Fan A etal (2024) The llama 3 herd of models. arXiv preprint arXiv: 2407. 21783 43. Akbari H, Yuan L, Qian R, Chuang W-H, Chang S-F, Cui Y, Gong B (2021) VATT: transformers for multimodal self-supervised learning from raw video, audio and text. Adv Neural Inf Process Syst 34:24206–24221 44. Kipf T, Pol E, Welling M (2019) Contrastive learning of structured world models. In: International conference on learning representations 45. Wang H, Wang Y, Zhou Z, Ji X, Gong D, Zhou J, Li Z, Liu W (2018) COSFACE: large margin cosine loss for deep face recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5265–5274 46. Sun Q-S, Zeng S-G, Liu Y, Heng P-A, Xia D-S (2005) A new method of feature fusion and its application in image recognition. Pattern Recognit 38(12):2437–2448 47. Zhiyuan L (2024) Investigating an ensemble classifier based on multi-objective genetic algorithm for machine learning applications. Int J Adv Comput Sci Appl 15(5):756 48. Rudnick EM, Patel JH, Greenstein GS, Niermann TM (1997) A genetic algorithm framework for test generation. IEEE Trans Comput Aided Des Integr Circuits Syst 16(9):1034–1044 49. Takahashi M, Kita H (2001) A crossover operator using independent component analysis for real-coded genetic algorithms. In: Proceedings of the 2001 congress on evolutionary computation (IEEE Cat. No. 01th8546). IEEE, vol 1, pp 643–649 50. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv: 1707. 06347 51. Raffin A, Hill A, Gleave A, Kanervisto A, Ernestus M, Dormann N (2021) Stable-baselines3: reliable reinforcement learning implementations. J Mach Learn Res 22(268):1–8 52. Kingma DP (2014) ADAM: a method for stochastic optimization. arXiv preprint arXiv: 1412. 6980 53. Loshchilov I, Hutter F (2017) SGDR: stochastic gradient descent with warm restarts. In: International conference on learning representations. https:// openr eview. net/ forum? id= Skq89 Scxx 54. Student (1908) The probable error of a mean. Biometrika 1–25 55. Wilcoxon F (1992) Individual comparisons by ranking methods. In: Breakthroughs in statistics: methodology and distribution. Springer, New York, pp 196–202 Publisher's Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.