A Dataset and Metric for Textual Video Content Description

A Dataset and Metric for Textual Video Content Description Stefan J. Arzberger [email protected] JOANNEUM RESEARCH – DIGITAL Graz, Austria Paul Raith∗ [email protected] JOANNEUM RESEARCH – DIGITAL Graz, Austria Werner Bailer werner[email protected] JOANNEUM RESEARCH – DIGITAL Graz, Austria Marion Jaks [email protected] Austrian Mediathek Vienna, Austria InternLM-XComposer-2.5 Tarsier VideoLLaMa2 0.0 0.2 0.4 0.6 0.8 1.0 value Direction ENT(ref pred) ENT(pred ref) CONT(ref pred) CONT(pred ref) Long-200 Medium-50 Short-25 Brief-10 0.0 0.2 0.4 0.6 0.8 1.0 value Direction ENT(ref pred) ENT(pred ref) CONT(ref pred) CONT(pred ref) Figure 1: Left: Distribution of entailment and contradiction when matching apporx. 200-word caption predictions from stateof-the-art MLLM-based models with reference captions; right: comparison of matching captions of different lengths predicted using InternLM against the references. Abstract Obtaining textual descriptions of the visual content of images and videos is often required in multimedia analysis and retrieval. Traditional video captioning approaches are usually evaluated on very short captions using rather simple metrics from NLP, while multimodal large language model (MLLM)-based approaches are mostly evaluated with question answering, which is query specific. We provide a dataset (FM-V2T) with 258 video clips from a media archive, annotated with detailed manually curated descriptions in English and German (long and short). We propose an LLM-based metric, which assesses the entailment and contradiction of facts extracted from a description with a reference, addressing shortcomings of existing metrics small changes with semantic impact and comparing descriptions with substantially different lengths. We provide experimental results on the reliability of the metric, and apply it to baseline results of three MLLM-based approaches on the FM-V2T dataset, comparing it with other metrics. ∗This work was done while Paul was working at JOANNEUM RESEARCH. Publication rights licensed to ACM. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only. Request permissions from owner/author(s). MM ’25, Dublin, Ireland ©2025 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 979-8-4007-2035-2/2025/10 https://doi.org/10.1145/3746027.3758224 CCS Concepts •Information systems → Multimedia information systems;• Computing methodologies → Information extraction;Language resources. Keywords Benchmarking, video to text, video captioning, evaluation, metrics ACM Reference Format: Stefan J. Arzberger, Paul Raith, Werner Bailer, and Marion Jaks. 2025. A Dataset and Metric for Textual Video Content Description. In Proceedings of the 33rd ACM International Conference on Multimedia (MM ’25), October 27–31, 2025, Dublin, Ireland. ACM, New York, NY, USA, 7 pages. https: //doi.org/10.1145/3746027.3758224 1 Introduction Enabling the exploration, retrieval, and understanding of multimedia data using text queries is a common task since the early days of research on multimedia databases. Generating textual descriptions of multimedia content allows not only to feed them into text processing and indexing pipelines, but also has the advantage of a long-term interoperable representation, which may not be the case for multimodal embeddings such as CLIP [ 19 ]. Video captioning or video to text (V2T) methods address this task, and like for many other multimedia analysis tasks, the approaches have recently moved from task specific approaches to methods relying on multimodal large language models (MLLMs). MM ’25, October 27–31, 2025, Dublin, Ireland Stefan J. Arzberger, Paul Raith, Werner Bailer, and Marion Jaks MLLMs are commonly evaluated with tasks such as visual question answering (VQA). This does not only hinder the comparability with specific captioning methods, but also excludes many practically relevant applications. For example, in multimedia retrieval, it is not feasible to run an MLLM on the fly against a large database to answer a specific query. It is required to index information generated from the multimedia content at ingest time. If a textual description is used for this purpose, it needs to be a query agnostic description, capturing the key elements of the video content. Traditional video captioning benchmarks, such as MSR-VTT [ 32 ], provide very short captions that do not cover all relevant content of the video. We observe thus a lack of datasets with longer reference captions, as well as datasets covering other genres than news. In addition most datasets are only available in English and some in Chinese, but other languages are not well covered. The fast-paced advances in MLLMs make general purpose models capable of reaching state of the art performance for the video to text task, when prompted appropriately. However, the literature on how these prompts are crafted is sparse, and the choices seem rather ad-hoc. This paper proposes a dataset for the evaluation of state of the art approaches to the video to text task, and a metric for comprehensive descriptions with different level of detail and length. In particular, the contributions of this paper are: • We provide FM-V2T, a bilingual dataset (English, German) for video to text evaluation, created from archival material with detailed text descriptions, and derived short annotations to allow compatibility with traditional captioning benchmarks. • We propose an LLM-based metric (LLMFactsF1) to compare descriptions based on entailed and contradicting facts. • We provide experimental results for assessing the reliability of the proposed metric. • We run three state of the art MLLMs on the FM-V2T dataset to generate baseline results, testing different prompts, and evaluate the results with established metrics and LLMFactsF1. We provide a brief overview of video to text methods as well as benchmarks and metrics for this task in Section 2, and introduce the FM-V2T dataset in Section 3. Section 4 proposes the LLMFactsF1 metric, which is used in the baseline experiments in Section 5. Section 6 concludes the paper. 2 Related work We briefly review related work on video to text methods, including specific video caption approaches (Section 2.1) and MLLM-based ones (Section 2.2). A full review of approaches is out of scope of this paper, but there are recent surveys (e.g. [ 24 ]). In addition, we provide an overview on evaluation approaches for this task. Video captions may vary in language, format (e.g., natural language, keywords), and focus (e.g., sentiment, dynamic activities, static descriptions of objects/subjects), introducing uncertainty in defining ground truth. The performance metrics used to evaluate textual correspondence can vary widely based on these priorities and formats, and thus must be aligned accordingly. Unlike image captioning, which focuses on visual information within single frames (spatial information), video captioning must also account for the semantics over time (temporal information), making it a more complex challenge [1]. 2.1 Video captioning models For several years, the video captioning task was addressed by specific models, for example, built on LSTMs [ 21 ]. Early approaches were based on image captioning work, and often provide single sentence descriptions. Around 10 years ago, approaches aiming at longer descriptions emerged. These strand of work is often referred to as “dense video captioning”, and approaches include segmenting the video into single actions [ 22 ] or events [ 13 ] to be turned into sentences. The emergence of multimodal embeddings have led to video captioning models making use of their capabilities, for example, VideoBERT [ 23 ]. Further advances in language and vision-languages models have been adopted in recent transformerbased methods, such as mPLUG-2 [ 31 ] and VAST [ 6 ]. Despite the good performance of these models, keeping up with the fast-paced advances of general purpose MLLMs has become challenging. 2.2 MLLM-based video to text models The introduction of multimodal large language models (MLLMs) has accelerated advances in video captioning, both in methodology and evaluation approaches [ 24 ]. In contrast to the specific approaches, generic MLLMs can be fine-tuned to the video captioning task, or simply prompted. Most of these models are the vision/video variant of a larger family of LLMs, such as VideoLLaMA [ 7 ], InternVideo [ 29 ], Deepseek-VL [ 30 ] or Pixtral [ 2 ]. We provide an overview of some recent methods in Table 1, focusing on open source methods with permissive licenses. We observe that it is hard to compare them with specific video captioning approaches because MLLM-based methods are evaluated on different task settings such as question answering. In addition, the choice of prompts seem ad-hoc, and there is hardly any literature on best practices for prompting MLLMs for video to text tasks. 2.3 Benchmarks and metrics Evaluating video captioning involves determining how closely the generated captions match a reference, provided as one or more possible captions, each generated or at least reviewed/revised by a human assessor. This introduces a subjective challenge: relevance may vary depending on the context and focus. For instance, ‘a thief is pursued’ and ‘a German shepherd is sprinting’ are both valid descriptions but emphasize different aspects of the scene. Similarly, ‘a running dog’ may be considered a correct but less detailed description. Due these challenges, human evaluation remains the gold standard for assessing semantic correspondence [26]. Quantitative evaluation can be categorized into text-based metrics and benchmarks. Text-based metrics include traditional NLP measures which struggle with synonymy and multi-sentence texts. These traditional NLP metrics such as BLEU [ 18 ], ROUGE [ 15 ] exhibit the limitation of word-wise matching ( 𝑛 -gram-based), whereas improvements such as CIDEr+[ 25 ] or METEOR [ 3 ] also consider synonyms or word stems. Embedding-based metrics, such as BERTScore [ 35 ], assess semantic similarity in embedding space and handle paraphrasing and longer texts better (but may e.g. fail handling negations properly, as they may be projected close to the positive statement). Recent LLM-based evaluations have the potential to approximate human understanding of text similarity [26]. A Dataset and Metric for Textual Video Content Description MM ’25, October 27–31, 2025, Dublin, Ireland Method Type Year License Comment mPLUG-2 [31] Specific 2023 Apache-2.0 Combines pre-trained vision and language transformers for video to text tasks. VAST [6] Specific 2023 MIT Fuses video, audio, and subtitle information via a transformer architecture. Tarsier [26] MLLM 2024 Apache-2.0 Temporal modeling through LLMs with the DREAM evaluation benchmark. VideoLLaMA2 [7] MLLM 2024 Apache-2.0 Enhances spatial-temporal understanding with a convolutional connector. InternLM-XC-2.5 [34] MLLM 2023 Apache-2.0 Supports long-context inputs/outputs and high-resolution inputs. VideoChat2 [14] MLLM 2024 Apache-2.0 MVBench benchmark for spatio-temporal understanding with VideoChat2 as a baseline. InternVideo2 [29] MLLM 2024 Apache-2.0 Leader on MVBench for fine-grained action description; long-context inputs. Deepseek-VL2 [30] MLLM 2024 MIT & DsML Efficient MLLM with competitive performance. Pixtral 12B [2] MLLM 2024 Apache-2.0 Efficient model from the mistral family with image support. Table 1: A selection of recent video to text methods with permissive licenses. Benchmarks can be classified into traditional and LLM-based types. Traditional datasets typically contain multiple single-sentence captions with simple textual structure and are benchmarked using traditional NLP metrics. For example, MSR-VTT [ 32 ] provides 20 single-sentence reference captions for a single video, each addressing subjective focus, and has an average length of 9 words per caption. Other traditional dataset benchmarks are often specialized for specific video domains like MSVD [ 5 ], ActivityNet [ 4 ] and YouCook [ 9 ]. LLM-driven benchmarks, however, are often tailored to specific priorities, with varied annotation structure and are frequently evaluated with the aid of LLMs. For example, the Tarsier method introduced the DREAM-1k benchmark [ 26 ] with a twostep LLM-based evaluation using single multi-sentence captions and key event extraction. MVBench [ 14 ], designed for emphasizing temporal understanding, also introduced the VideoChat2 model. Other examples include VATEX [ 27 ] and MMBench [ 16 ]. There are larger datasets with longer descriptions such as Vript [ 33 ], InternVid [ 28 ], MiraData[ 12 ] and HowTo100M [ 17 ] which are only fully automatically annotated. We observe a gap between the traditional captioning metrics making a comparison against a reference – addressing often brief captions and metrics with limited ability to capture complex semantics – and approaches from LLM benchmarks. 3 FM-V2T Dataset We provide the FAIRmedia Video to Text (FM-V2T) dataset, containing detailed textual descriptions of the visual content of video shots. The dataset uses content of Österreichische Mediathek, the Austrian audio and video archive, and was developed in the context of the project FAIRmedia 1 . The dataset uses a selection of content of the Wiener Video Rekorder collection 2 ) containing content created with consumer video cameras, and documenting everyday life. The content is diverse, covering events in public space, festivities, family events of people living in or around Vienna, but contains quite diverse content, including travel to nearby and faraway places. The content has been cleared for publication in Mediathek’s online catalog, addressing the potential copyright and privacy issues. 1Fair and trusted datasets for media computing https://www.joanneum.at/digital/en/ projects/fairmedia/ 2 https://www.mediathek.at/wiener-videorekorder/english-information (content available at https://www.mediathek.at/digitale-sammlung From the collection, 268 videos have been selected by archive experts, aiming at diversity of the content. These videos have been temporally segmented using the shot boundary detector described in [ 11 ], and key frame extraction based on visual activity in the content has been performed with that algorithm. As annotation of the entire collection is not feasible, but we aim at a heterogeneous and representative content set, we select one shot per video manually. As shot boundaries may delineate completely different content, describing multiple shots together does not generally provide added value. In order not to select very short or long clips, the shots are required to fall into a length interval of [ 5; 35 ] s. As we also want to exclude almost static or extremely dynamic content, we require the number of keyframes to be [ 2; 100 ] and the number of keyframes/s to be [ 0 . 5; 4 ] . After this process, 258 clips remain, with mean length 15.45 ±7.79s (min 5.12s, max 34.92s). For these clips, textual annotations in English and German are generated. The English description is generated using VideoLLaMa2 [ 7 ], using the prompt ‘Describe this video, exactly and only focus on what is visible, without imagining any details that are not visible! Answer what can be seen, where the video was shot, what persons, animals or buildings etc. can be seen. What is happening in the video? When was the video filmed at day or night for example . . . Is there something unique to this video? Limit the description to 200 words!’. The resulting text is then manually checked, corrected and amended as needed. As preliminary tests with German video to text models showed inferior quality in comparison to English models, we translate the revised annotation to German using NLLB [ 8 ]. Again, the resulting annotation is manually checked and revised. In order to establish interoperability with widely used benchmarks such as MSR-VTT [ 32 ], we derive 20 single sentence short captions (similar to the “gold captions” in MSR-VTT). This has been done using ChatGPT-4o mini 3 , using the prompt ‘I will give you a video caption and you have to extract the most important information into a short 10 word description and make different variations of the "pred_caption". These "short_caption" are variations of the "pred_caption" and have the same meaning and maybe focus on a few other details from the original video caption and are all formulated in other words but without inventing any other details that where not in the original video caption. So similar like in the MSR_VTT dataset. 3https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence MM ’25, October 27–31, 2025, Dublin, Ireland Stefan J. Arzberger, Paul Raith, Werner Bailer, and Marion Jaks Here this is a example please also stay in this json format: { ... }’. The resulting captions were again manually refined. This dataset, based on diverse archive video content, is primarily intended for assessing and comparing video to text methods, supporting two languages. The provision of long detailed annotations and a set of short captions enables the application of a wide range of metrics and benchmarking approaches. In addition, the dataset can serve related downstream tasks, such as fact extraction or visual question answering. A release of the dataset is available at https://github.com/FAIRmedia-AT/FM-V2T. In addition to the annotations (published under a CC-BY 4.0 license), it contains the references and metadata of the video, including code for downloading the clips for which annotations are available. In addition to established NLP metrics, we provide two variants of cosine similarity of text embeddings produced by SentenceBERT [ 20 ] 4 for comparison. These scores rely on dense representations produced by SentenceBERT. For the input text span 𝑡 , we obtain an embedding 𝑒(𝑡)=MeanPoolMiniLM(𝑡)∈R384, where the hidden states of all tokens are averaged (mean pooling). Given a predicted caption 𝑐𝑝and a reference caption 𝑐𝑟, we report: CosPar (paragraph_cosine_similarity). A single embedding is computed for the entire caption; the score is the cosine between the two paragraph vectors CosPar(𝑐𝑝,𝑐𝑟)=cos𝑒para(𝑐𝑝), 𝑒para (𝑐𝑟). CosSent (sentence_median_similarity). Both captions are first segmented into sentences. Sentence embeddings 𝑒sent(·) are paired one-to-one by greedily choosing the highest remaining cosine similarity, producing a matching set M(𝑐𝑟,𝑐𝑝) . The score is the median of these pairwise cosines, which is robust to outliers such as very dissimilar or unmatched sentences: CosSent(𝑐𝑝,𝑐𝑟)=median (𝑠𝑟,𝑠𝑐) ∈ M (𝑐𝑟,𝑐𝑝)cos𝑒sent(𝑠𝑟), 𝑒sent(𝑠𝑐). Both metrics theoretically range from -1 to 1, but in our experiments on natural language captions, scores fell in the practical range of 0 (minimal semantic similarity) to 1 (identical embeddings). Both use the same embedding-pooling configuration {mean,median_similarity} specified in the experimental settings. 4 Fact-based description metric Traditional metrics for assessing video captioning, such as BLEU, ROUGE or CIDEr, are not well suited for handling longer and more expressive descriptions. Using text embeddings such as SentenceBERT [ 20 ] in order to measure distances in the embedding space addresses some of the issues of these metrics. However, embeddings may still be similar when many of the concepts in the sentences align, ignoring that e.g. a part of a sentence has been negated. In addition, these metrics do not perform reliably when comparing texts of different lengths. This issue can partly be addressed by breaking each of the texts into sentences, raising the question how to aggregate the set of their pairwise comparison scores. We thus propose a metric based on the overlap of factual statements between two texts, e.g. generated and reference description of a video clip. We discuss in this section the design of the metric, describe the concrete implementation and present the experiments performed to validate the robustness of the metric. 4sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 4.1 Design The basic idea of the metric is to extract single sentence statements contained in a text 𝑇𝐴 (“facts”), and check them against another text 𝑇𝐵 . 𝑇𝐵 may support these facts (entailment), be in conflict with them (contradiction) or not contain information related to this statement (neutral). This two step pipeline is realized using two prompts for LLMs: the fact extraction model M𝑓 𝑒𝑥 (𝑇) → 𝐹 , where 𝐹 is the set of facts 𝐹={𝑓1, . . . , 𝑓𝑘} , and the checking model M𝑐ℎ𝑘 (𝑇, 𝐹) → (𝐸,𝐶, 𝑁 ) , where 𝐸 , 𝐶 and 𝑁 are binary vectors of size 𝑘 , encoding for each fact whether it is entailed, contradicting or neutral (|𝐸|+|𝐶|+|𝑁|=𝑘,|·|denotes the 𝐿1norm). We obtain (𝐸𝐴𝐵,𝐶𝐴𝐵, 𝑁𝐴𝐵)=M𝑐ℎ𝑘 (𝑇𝐵,M𝑓 𝑒𝑥 (𝑇𝐴)) , and calculate the rates 𝐸𝑁𝑇𝐴𝐵 = |𝐸𝐴𝐵 | 𝑘𝐴 ,𝐶𝑂𝑁𝑇𝐴𝐵 = |𝐶𝐴𝐵 | 𝑘𝐴 . By including entailments and contradictions normalised by the number of facts in the metric, the amount of neutral statements is implicitly included. In order to compensate for effects of different lengths of 𝑇𝐴 and 𝑇𝐵 , we perform the extraction and checking process in both directions, and calculate the harmonic mean 𝐸𝑁𝑇𝐹1= 2|𝐸𝐴𝐵 ||𝐸𝐵𝐴 | |𝐸𝐴𝐵 |+|𝐸𝐵𝐴|,𝐶𝑂𝑁𝑇 𝐹1= 2|𝐶𝐴𝐵 ||𝐶𝐵𝐴 | |𝐶𝐴𝐵 |+|𝐶𝐵𝐴 |. In order to express semantic overlap as a single number, we define 𝐿𝐿𝑀𝐹𝑎𝑐𝑡𝑠𝐹1= 2𝐸𝑁𝑇𝐹1(1−𝐶𝑂𝑁𝑇 𝐹1) 𝐸𝑁𝑇𝐹1+ (1−𝐶𝑂𝑁𝑇 𝐹1). 4.2 Implementation We implement the proposed LLMFactsF1 metric as a two-stage pipeline comprising fact extraction and relational classification using LLMs. The components are integrated into a locally hosted inference system using the HuggingFace Transformers interface. In the fact extraction stage, the LLM is prompted to identify atomic factual statements from paragraph-length input. Extracted facts must be self-contained, including subject, predicate, and object, and must retain linguistic modifiers such as modality, negation, and quantification. In the classification stage, each fact is evaluated for entailment, contradiction, or neutrality with respect to a comparison paragraph. This step is performed collectively: the entire fact set is assessed against the full reference or hypothesis text, rather than individually. If parsing fails due to syntactic inconsistencies, an auxiliary LLM call attempts structural correction. Manual inspection was occasionally needed to fix formatting, such as removing spurious quotation marks around named entities, but did not modify the statements themselves. The system uses the Llama 3.1 8B5[10] model with deterministic hyperparameters 6 and a high token limit to preserve context integrity. Smaller models (e.g., 3B variants) showed deficiencies in meaningful fact extraction. On a single NVIDIA A6000 GPU (48GB RAM), the fact extraction step takes 16 . 8 ± 4 . 9s and the fact alignment check takes 35.9±18.5s. 5https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct 6do_sample=False,temperature=None,top_p=None A Dataset and Metric for Textual Video Content Description MM ’25, October 27–31, 2025, Dublin, Ireland (a) Ref2Ref (b) InternLM2InternLM 0.0 0.2 0.4 0.6 0.8 1.0 value Direction ENT(ref ref) ENT(ref ref) CONT(ref ref) CONT(ref ref) Figure 2: Bidirectional validation of LLM Metric for ENT (green) & CONT (red) for (a) reference-against-reference and (b) InternLM-to-InternLM. Prompts were iteratively refined for robustness; the final versions are documented in the GitHub repository of the dataset. The pipeline provides counts of entailments, contradictions, and neutral relations in both directions, which are used to compute the final LLMFactsF1 score as detailed above. 4.3 Validation of the metric In order to assess the reliability of the metric, we test the metric on self-matching the references in the FM-V2T dataset. This should result in 𝐸𝑁𝑇𝐹1 close to 1, and 𝐶𝑂𝑁𝑇𝐹1 close to 0. We perform the experiment twice, matching forward and backward, in order to test the reproducibility of the models. In addition, we repeat the same experiment with self-matching the output of InternLMXComposer-2.5 [ 34 ] on the FM-V2T dataset. Figure 2 shows the results, indicating that the metric is very reliable in terms of contradictions, which are almost 0. The rate of entailments is very high, though with more outliers, i.e. captions resulting in lower entailment rates. This means that support for some extracted statements could not be verified, and they are thus considered neutral. We also match the InternLM video to text model output against itself, which turns out to be even more reliable, with a lower number of outliers. This is probably due to simpler and shorter nature of the outputs compared to the references. We also perform a test with randomly misaligned references, shown in Figure 3. As expected, the entailment is very low, contradictions are quite high. As the randomly aligned references may describe different content, and not necessarily contradict each other, the contradictions are almost uniformly distributed, also across different lengths. 4.4 Qualitative example In order to illustrate the advantages of the proposed metric, we provide a qualitative example. Table 3 provides an example of a video caption, including the reference, outputs of three models and two manually changed versions of the reference (one changing a name, one introducing a negation). Table 4 shows the results of the proposed LLMFactsF1 and a set of other metrics. It becomes evident, that the proposed metric scores the variants of the reference clearly higher than the predictions, and penalizes the negation more than the name change, which is not the case for most of the other metrics. The cosine similarity Ref2InternLM (Long-200) Ref2InternLM (Medium-50) Ref2InternLM (Short-25) Ref2InternLM (Brief-10) 0.0 0.2 0.4 0.6 0.8 1.0 value Direction ENT(ref pred) ENT(pred ref) CONT(ref pred) CONT(pred ref) Figure 3: Results with randomly misaligned references across different caption lengths. metrics of text embeddings come closest, but fail to make a clear distinction from the prediction with a wrong text on the banner and the one omitting the text entirely. 5 Experiments In order to obtain baseline results, we run three state-of-the-art models on the FM-V2T datasets: Tarsier [ 26 ], VideoLLaMA2 [ 7 ] and InternLM-XComposer-2.5 [34]. Figure 1 (left) shows the distribution of entailment and contradiction of the model outputs against the reference on the FM-V2T datasets. These outputs are obtained by prompting the models for detailed ( ≤ 200 words) captions. The results show that the InternLM model has the highest fraction of entailment and the lowest fraction of contradictions, while the results of the other two models are worse. The fact that InternLM outperforms VideoLLaMa shows that there is no bias in terms of description quality stemming from the fact that VideoLLaMa was used as starting point for the human annotation. However, the VideoLLaMa model provides a more similar level of detail of the description, as evident from the similar distribution of entailment – this might be a slight bias from the process. For the other two models, the entailment of facts extracted from the prediction is higher than from the reference, indicating that the reference is more comprehensive, and thus it is easier for the LLM to align facts with it. The differences in matching directions for some models confirm also the decision to use the harmonic mean of both directions as an integrated score. We also analyse the impact of the description length on the score. We use InternLM to predict descriptions of 200, 50, 25 and 10 words, and compare to the references. The results are shown in Figure 1 (right). As expected, the shorter descriptions result in a lower number of entailed and contradicting facts when matched against the longer reference. The extraction of facts from the shorter prediction results in higher entailment when matched against the longer one, and the related information can be found there, however, the absolute number of facts is lower than in the opposite direction. Table 2 provides an overview of results with prompting the baseline models for descriptions of different lengths, and matching against the long reference or the short captions. These results also show that the lengths asked in the prompts are followed to a different degree by the different models. The results indicate that state of the art models provide usable results on this task, but also that the MM ’25, October 27–31, 2025, Dublin, Ireland Stefan J. Arzberger, Paul Raith, Werner Bailer, and Marion Jaks Model Ref Pred Words LLMFactsF1 METEOR BLEU-1 BLEU-2 ROUGE-L CosPar CosSent VideoLLama Ref Long-200 148±31 0.496±0.146 0.212±0.035 0.390±0.091 0.245±0.073 0.296±0.052 0.751±0.113 0.670±0.087 InternLM Ref Long-200 136±22 0.529±0.184 0.198±0.047 0.380±0.085 0.223±0.083 0.283±0.072 0.711±0.112 0.648±0.099 Tarsier Ref Long-200 59±14 0.476±0.175 0.135±0.032 0.257±0.107 0.150±0.069 0.257±0.047 0.771±0.092 0.644±0.094 InternLM Ref Medium-50 61±22 0.508±0.186 0.129±0.044 0.248±0.125 0.142±0.086 0.246±0.065 0.729±0.108 0.640±0.101 InternLM Ref Short-25 49±30 0.473±0.207 0.106±0.051 0.173±0.153 0.097±0.092 0.205±0.072 0.682±0.125 0.653±0.116 InternLM Ref Brief-10 10±3 0.431±0.230 0.033±0.015 0.002±0.014 0.001±0.007 0.085±0.034 0.629±0.121 0.686±0.124 InternLM ShortRefs Brief-10 10±3 - 0.213±0.076 0.720±0.182 0.450±0.226 0.415±0.144 - - Tarsier ShortRefs Brief-10 20±35 - 0.204±0.066 0.590±0.160 0.353±0.183 0.373±0.120 - - Table 2: Metrics obtained for differently prompted model outputs using the three baseline methods on the FM-V2T dataset. ShortRef refers to the short version of the reference caption. Type Text: Short-25 Reference A small airplane is flying across the sky, with a banner reading "BUSSI SUSI-LEO" trailing behind it. Negation A small airplane is flying across the sky, with a no banner reading "BUSSI SUSI-LEO" trailing behind it. NameChange A small airplane is flying across the sky, with a banner reading " BUSSI SUSI-LEO BUSSI JOSEF-MARIA" trailing behind it. InternLM-XC A helicopter is flying across a clear blue sky, with a banner reading "BUSSI SUSHI -LEO" trailing behind it. Tarsier Aplane flies across the sky with a sign displaying ’BUSSI SUJU-LEON’. VideoLLaMa A small airplane flies over a runway, followed by a helicopter, both seen from a distance against a gray sky. Table 3: Generated descriptions for the video across models and edits. Text in red indicates contradicting and in green entailed statements. For the edited variants, red text marks deletions, and blue text insertions. Short-25 LLMFactsF1 METEOR BLEU-1 BLEU-2 ROUGE-L CosPar CosSent Negation 0.585 0.594 0.941 0.907 0.941 0.967 0.967 NameChange 0.750 0.518 0.941 0.907 0.941 0.981 0.981 InternLM 0.400 0.402 0.722 0.618 0.747 0.722 0.722 Tarsier 0.635 0.186 0.385 0.304 0.468 0.908 0.908 VideoLLaMa 0.273 0.127 0.250 0.162 0.219 0.728 0.728 Table 4: Metrics obtained for the example in Table 3. dataset is challenging enough to leave room for improvement. The results also show that shorter descriptions are able to capture the most relevant aspects of the content. The last two lines of the table provides results for the derived short captions, showing that the dataset provides compatibility with MSR-VTT style benchmarks. We also analyse the correlation of the proposed metric with baseline metrics for InternLM results on the FM-V2T dataset (Figure 4). Generally, the correlation is low, showing that the metrics assess different aspects of the descriptions. There is a weak correlation with the metrics using cosine similarity of the text embeddings, while there is almost no correlation with some of the traditional NLP metrics. Based on samples we assume that this is due to cases where syntactic and semantic impact of differences diverges. We 0.0 0.2 0.4 0.6 0.8 1.0 Baseline metric 0.0 0.2 0.4 0.6 0.8 1.0 LLMFactsF1 BLEU-1 BLEU-2 METEOR ROUGE-L CosPar CosSent Figure 4: Comparison of the proposed metric with baseline metrics for InternLM results on the FM-V2T dataset. have made experiments for selecting prompts for the MLLMs. Like often in prompt engineering, it is hard to predict which changes will have substantial effects. We have thus created 30 prompts and ran experiments with them. We provide the list of prompts and an interactive plot on the dataset’s Github repository. 6 Conclusion We have provided a bilingual dataset for the evaluation video to text methods, compatible with the evaluation methods used for traditional captioning as well as MLLM-based methods. We have also proposed a novel LLM-based metric, and validated the metric in a range of experiments on the dataset. This also includes comparisons with other metrics on the outputs of three state of the art MLLMs on the proposed dataset. The dataset provides the annotations in two languages, but the machine translation and revision workflow we have used to obtain the German annotations can be efficiently replicated for other languages. In a similar way, the visual content description could be amended by speech to text. Acknowledgments This work has been funded partially by the Austrian Research Promotion Agency (FFG) under the Digital Technologies project FAIRmedia (https://www.joanneum.at/digital/en/projects/fairmedia/), and by European Union’s Horizon Europe programme under grant agreement n ◦ 101070250 XRECO (https://xreco.eu/). The authors thank Georg Thallinger for his feedback on the paper. A Dataset and Metric for Textual Video Content Description MM ’25, October 27–31, 2025, Dublin, Ireland References [1] Moloud Abdar, Meenakshi Kollati, Swaraja Kuraparthi, Farhad Pourpanah, Daniel McDuff, Mohammad Ghavamzadeh, Shuicheng Yan, Abduallah Mohamed, Abbas Khosravi, Erik Cambria, et al . 2024. A review of deep learning for video captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024). [2] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al . 2024. Pixtral 12B. arXiv preprint arXiv:2410.07073 (2024). [3] Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72. [4] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. 2015. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition. 961–970. [5] David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies. 190–200. [6] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. 2023. Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems 36 (2023), 72842–72866. [7] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al . 2024. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476 (2024). [8] Marta R Costa-Jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al . 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672 (2022). [9] Pradipto Das, Chenliang Xu, Richard F Doell, and Jason J Corso. 2013. A thousand frames in just a few words: Lingual description of videos through latent topics and sparse object stitching. In Proceedings of the IEEE conference on computer vision and pattern recognition. 2634–2641. [10] Aaron Grattafiori et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [11] Hannes Fassold. 2024. Faster than real-time detection of shot boundaries, sampling structure and dynamic keyframes in video. In 2024 8th International Conference on Imaging, Signal Processing and Communications (ICISPC). IEEE, 33–36. [12] Xuan Ju, Yiming Gao, Zhaoyang Zhang, Ziyang Yuan, Xintao Wang, Ailing Zeng, Yu Xiong, Qiang Xu, and Ying Shan. 2024. Miradata: A large-scale video dataset with long durations and structured captions. Advances in Neural Information Processing Systems 37 (2024), 48955–48970. [13] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. 2017. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision. 706–715. [14] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al . 2024. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 22195–22206. [15] Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81. [16] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al . 2024. Mmbench: Is your multi-modal model an all-around player?. In European conference on computer vision. Springer, 216–233. [17] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. 2019. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision. 2630–2640. [18] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318. [19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al . 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763. [20] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 3982–3992. [21] Anna Rohrbach, Marcus Rohrbach, and Bernt Schiele. 2015. The long-short story of movie description. In Pattern Recognition: 37th German Conference, GCPR 2015, Aachen, Germany, October 7-10, 2015, Proceedings 37. Springer, 209–221. [22] Andrew Shin, Katsunori Ohnishi, and Tatsuya Harada. 2016. Beyond caption to narrative: Video captioning with multiple sentences. In 2016 IEEE International conference on image processing (ICIP). IEEE, 3364–3368. [23] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. 2019. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF international conference on computer vision. 7464– 7473. [24] Yunlong Tang, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, et al . 2025. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology (2025). [25] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4566–4575. [26] Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. [n. d.]. Tarsier: Recipes for training and evaluating large video description models, 2024. URL https://arxiv. org/abs/2407.00634 8 ([n. d.]). [27] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. 2019. Vatex: A large-scale, high-quality multilingual dataset for videoand-language research. In Proceedings of the IEEE/CVF international conference on computer vision. 4581–4591. [28] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al . 2023. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942 (2023). [29] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al . 2024. Internvideo2: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision. Springer, 396–416. [30] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al . 2024. Deepseekvl2: Mixture-of-experts vision-language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10302 (2024). [31] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al . 2023. mplug-2: A modularized multi-modal foundation model across text, image and video. In International Conference on Machine Learning. PMLR, 38728–38748. [32] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5288–5296. [33] Dongjie Yang, Suyuan Huang, Chengqiang Lu, Xiaodong Han, Haoxin Zhang, Yan Gao, Yao Hu, and Hai Zhao. 2024. Vript: A video is worth thousands of words. Advances in Neural Information Processing Systems 37 (2024), 57240–57261. [34] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al . 2024. Internlm-xcomposer2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320 (2024). [35] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.