Evaluating Instruction-Tuned Large Language Models for Automatic Annotation of Multilingual Climate Disaster-Related Social Media Messages

Evaluating Instruction-Tuned Large Language Models for Automatic Annotation of Multilingual Climate Disaster-Related Social Media Messages Jonathan Ayebakuro ORAMA a,1, Marc JUVILL ` A GARCIA aand Maite MELERO a aBarcelona Supercomputer Center (BSC) ORCiD ID: Jonathan Ayebakuro Orama https://orcid.org/0000-0002-2622-3224, Marc Juvill` a Garcia https://orcid.org/0009-0001-9489-4839, Maite Melero https://orcid.org/0000-0001-9933-3224 Abstract. Analysing social media messages shared during a climate disaster to extract relevant information is a difficult task due to the high volume of messages and the subjective nature of relevance. Nevertheless, this is a necessary step when annotating texts to train models for crisis management. Traditionally, samples are annotated manually, which is time consuming and challenging to maintain consistency, especially when dealing with multilingual corpora. Instruction-tuned generative language models are a potential solution to this problem due to their ability to follow instructions and their state of the art performance in several natural language processing tasks including text classification. This paper evaluates several opensource multilingual instructed models with the goal of seeing how well they perform as annotators or text filtering tools in a crisis. We conduct experiments using both zero-shot and few-shot prompting, exploring different ways of incorporating examples in few-shot prompting. Finally, we examine the impact of post-annotation label cleaning and further refine results through score thresholding. Keywords. Large Language Models, Prompt Tuning, Few-shot Learning, AutoAnnotation, Crisis Management 1. Introduction Social media messages published during a weather crisis are a valuable data source for studying the trends of an ongoing disaster and extracting useful information [1]. This information could be used in planning strategies and in decision-making which could potentially increase the efficiency of rescue operations. Furthermore, because of the abundance and semi-permanence of these messages, past events could be studied to develop new policies or improve existing policies for operational conduct during similar disaster scenarios. 1Corresponding Author: Jonathan Ayebakuro Orama, Barcelona Supercomputing Center, Plac¸a Eusebi G¨ uell, 1-3, 08034, Barcelona (Spain); E-mail: [email protected] Artificial Intelligence Research and Development K. Trejo et al. (Eds.) © 2025 The Authors. This article is published online with Open Access by IOS Press and distributed under the terms of the Creative Commons Attribution Non-Commercial License 4.0 (CC BY-NC 4.0). doi:10.3233/FAIA250578 79 To study and extract useful information from social media messages, it is important to identify those messages that are relevant to an ongoing disaster. This is a difficult task due to the volume of data, the unstructured nature of social media platforms, and the subjective nature of relevance. Traditionally, to study a crisis event, social media messages are collected and categorised manually by human annotators using defined criteria of what is relevant or related to the event and what is not. This is time consuming and can create some inconsistencies if the criteria are not well defined, or if multiple human annotators are involved in the process. This problem is not unique to crisis management alone, and has encouraged research into automatic annotation methods, including rulebased annotation [2], lexicon-based annotations [3], ontology-based annotations [4], etc. Instruction-tuned large language models are also a possible solution to this problem because of their ability to follow instructions. A well-crafted prompt that details instructions on what should be contained in relevant messages will help these models to make an acceptable automatic annotation [5]. In this paper, we evaluated ten open-weight instructed large language models with the goal of seeing how well they perform as automatic annotators of social media messages in a climate disaster. We first evaluated the models’ zero-shot annotation capability in crisis management by crafting a prompt with instructions to guide their annotations. We then evaluated the impact of few-shot prompting and label cleaning using label quality score thresholds in improving the zero-shot annotations. Finally, we evaluated the models in a multilingual setting. The rest of the paper is organised as follows. Section 2details some related works in automatic annotation for text classification and crisis management, Section 3presents our evaluation methodology in detail, Section 4explains the results of our evaluations, and Section 5concludes. 2. Related Works 2.1. Automatic Annotation in Crisis Management In crisis scenarios, information is highly valuable. It is required by affected people, crisis response officials, and by the populace. Social media messages have become the main source to gather useful insights. The information on social media is highly voluminous and unstructured, as such recent research has gone into developing models that detect crisis events from social media messages [6]. To train these models, labelled social media data is required which is expensive to manually label. Some works have looked into ways to automatically annotate large collection of crisis messages for training crisis event detectors. The authors in [7] employed distant supervision to label tweets for crisis response. Keywords extracted from existing labelled crisis data sets are expanded with similar lexical units from FrameNet (a linguistic knowledge base for English). The expanded keyword list is then used as a lexicon to auto-label tweets. [8] focuses on modelling a relevance labelling function that includes a severity score to filter crisis data from heterogeneous sources. Their method considers the time and location of retrieved data, and performs checks to evaluate their integrity and acceptability. Finally, they cluster the data using k-medoids algorithm by their relevance. These methods, though adequate in specific cases, are either dependent on a robust knowledge base or require other data modalities to function. LLMs, on the other hand, could be a viable substitute. J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation80 2.2. Generative Model-Based Automatic Annotation Generative pre-trained transformers (GPT) are large language models (LLMs) built on the transformer architecture that use attention mechanisms to learn the semantics and nuances of natural language from large sets of unlabelled data. They rose in popularity with OpenAIs release of ChatGPT2in 2022. These models are great at generating texts that resemble human-written texts and display state-of-the-art performance in various NLP tasks [9]. GPT models can be further fine-tuned on conversational data sets to give them the ability to follow instructions and converse with humans [10]. This is called instruction tuning. It is possible to use instruct GPT models for automatic annotation of textual data in scientific analysis.The authors in [5] performed a detailed analysis of GPT models for social scientific analysis. They considered prompt-based zero-shot learning, few-shot learning, and instruction tuning in their analysis of the performance of GPT models in labelling the stance of social media posts toward candidates in a past USA Presidential election. In their findings, GPT models performed better than traditional methods of training a model for text classification using labelled data. A more reliable way of using GPT models is to deploy them as pre-annotators. In [11] legal texts are pre-annotated by GPT models and law students go through the annotations, denying or accepting them. The authors then compare the pre-annotation, to human annotated data, and also the rejected and accepted labels. They found that GPT models are a reasonable starting point for annotation. This means they can be used as an initial filtering stage to ease the burden on human annotators. Another reliable way of using GPT models for auto-annotating is Retrieval Augmented Generation (RAG). In RAG, GPT models consult a database that contain factual documents relevant to the task, rooting its responses in facts from those documents. The authors in [12] apply RAG in phenotypic observations in biology. They use RAG to parse established ontologies of the phenotype of forests trees, ensuring that GPT responses adhere to them. This resulted in annotations that are highly similar to expert-curated annotations. It is also possible to use GPT models to assist text clustering in classification of textual documents. This is useful when specific labels for a large collection of documents is not known. The authors in [13] first provide a collection of documents to GPT-3.5 and prompt it to suggest suitable labels, they then adjust the labels merging similar labels, and finally prompt GPT to assign appropriate labels from the suggested labels to the documents in the data set. They found that this method performs comparable to, and often better than state-of-the-art clustering methods in automatic annotation. 3. Evaluating Instruct LLMs as Crisis Annotators In this paper we organised our evaluation of instruct large language models as crisis annotators around the following research questions. • RQ1: Are LLMs acceptable zero-shot annotators of crisis social media messages? In this RQ we want to assess if given an appropriate prompt, an LLM can properly annotate a crisis event. 2https://chatgpt.com/ J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation 81 • RQ2: Can label cleaning powered by machine learning or few-shot prompting improve zero-shot annotation performance? In this RQ we want to assess if the zero-shot annotation of LLMs can be further improved with label cleaning or fewshot prompting. • RQ3: Can current state-of-the-art LLMs adapt to multilingual crisis social media messages? In this RQ we want to assess if LLMs can also provide acceptable annotations when dealing with multilingual texts. In the following subsections, we explain the models we selected for evaluation, the data, and our methods for zero-shot prompting, few-shot prompting, and label cleaning to answer these research questions. 3.1. Models In recent years, there has been a race among tech companies and research institutes to develop the highest-performing large language models (LLMs). Even in the course of writing this paper, newer models have been released that were not included in our analysis. To select models for evaluation, we put the following into consideration: •Their accessibility (i.e., open-weight vs. proprietary). Open-weight models make their parameters publicly available, allowing researchers to use and fine-tune them for downstream tasks. •Their size. Specifically models within the range of 4 billion parameters and 14 billion parameters. This is mainly due to computational concerns. •Their source. We focused on models from reputable companies and research institutes. This is mainly to ensure the quality of annotations. In Table 1, we highlight the models chosen for evaluation, their size, context length, and the number of languages they support. All models are available on HuggingFace3. Table 1. Models selected for evaluation with their model name on huggingface HuggingFace Model name Parameter size Context size No. of languages supported EuroLLM-9B-Instruct 9B 4k 35 gemma-2-9b-it 9B 8k primarily english gemma-3-4b-it 4B 128k +140 Mistral-7B-Instruct-v0.3 7B 8k primarily english occiglot-7b-eu5-instruct 7B 8k 5 Phi-3-medium-4k-instruct 14B 4k primarily english Phi-4-mini-instruct 3.8B 128k primarily english Qwen2.5-14B-Instruct 14B 128k 29 Qwen2.5-7B-Instruct 7B 128k 29 salamandra-7b-instruct 7B 8k 35 3.2. Data To evaluate the performance of the selected models in Table 1, we used the CrisisLexT264data set [1], which contains tweets from 26 incidents that occurred between 3https://huggingface.co/ 4https://www.crisislex.org/data-collections.html#CrisisLexT26 J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation82 [Task] Analyze the given body of text and determine whether it contains information relevant to an ongoing fire disaster, flood disaster , or neither. The classification should be based on the presence of specific, actionable details or key indicators of relevance to such disasters. [Instructions] Examine the text for mentions of fire-related or flood-related events, such as locations , impacts , safety advisories , or emergency responses. Classify as "fire" if the text contains information about wildfires , house fires , smoke hazards , evacuation orders due to fire, or related topics. Classify as "flood" if the text discusses water inundation , flash floods , storm surges, flood warnings , or evacuation due to flooding. Classify as "none" if the text does not provide information relevant to either disaster or lacks actionable disaster -related content. Texts that express sympathy , offer prayers, or request donations should also be classified as "none". Only assign "flood" or "fire" if the relevance is clear; otherwise , default to "none". [OutputFormat] The response should be formatted as follows: {"classification": "[flood/fire/none]"} Example Input: "Evacuation orders have been issued for Riverside due to rapidly spreading wildfire near the canyon." Output: {"classification": "fire"}<EXAMPLES > Classify: Input: "<TEXT >" Figure 1. Zero-shot annotation prompt. 2012 and 2013. Each tweet in the data set has been manually labeled based on its informativeness, information type, and information source. We selected this data set because these labels are suitable for our evaluation. We wanted to focus on wildfire and flood incidents with primarily English texts so we selected the following incidents from the data set: (1) Colorado, USA wildfires 2012 (2) Alberta, Canada floods 2013 (3) Australia bushfire 2013 (4) Colorado, USA floods 2013 (5) Queensland, Australia floods 2013. This resulted in a total of 5,376 tweets. To evaluate the LLMs capabilities in event type and relevance auto-annotation, we created our own labeling scheme by merging existing annotations in the data set. Posts marked as ’Not related’ or ’Related but not informative’ in terms of informativeness were labeled as ’none’ meaning they are not relevant. We also applied the ’none’ label to posts categorized under the information types ’Donations and volunteering’ or ’Sympathy and support’. Conversely, posts were labeled as ’fire’ or ’flood’ with respect to the incident, if they were marked as ’Related and informative’ and did not fall under the aforementioned information types. This scheme realigns the labels to focus on the ongoing crisis, and the posts providing valuable or important information about it. Furthermore, to evaluate the performances of the selected models in multilingual auto-annotation, we translated the data set into Spanish, Catalan, and German. We used SalamandraTA-7B5,a state-ofthe-art translation model developed by the Barcelona Supercomputing Center, capable of translating between 35 languages. 5https://huggingface.co/BSC-LT/salamandraTA-7b-instruct J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation 83 {"role": "user", "content": "I need help with a math problem."}, {"role": "assistant", "content": "Sure, What’s the math problem?"} Figure 2. Sample chat template. 3.3. Zero-shot Prompting Zero-shot prompting is the process of crafting a prompt for a LLM that includes instructions on how to perform a certain task. For the best results, the prompt should include the task, specific instructions on how to perform the task, and the format of response. To craft our prompt for auto-annotation, we followed OpenAI’s guides for prompt generation6.The guide is designed to be provided as an LLM prompt. We supplied this prompt to GPT-4 and after several tweaking came to our final prompt shown in Figure 1. The prompt has three parts, the ’[Task]’ describes the task to the model, ’[Instructions]’ provides instructions to the model, and ’[Output format]’ defines the format of the output, in our case we opted for the JSON format. These labels ’[Task]’, ’[Instructions]’, ’[OutputFormat]’ are not included in the main prompt, they are included for clarity in this paper. Also, ’<TEXT>’ and ’<EXAMPLES>’ are placeholders used to insert the text to be annotated, and the few-shot examples respectively. Our final prompt is a bit wordy but includes instructions we believe to be necessary when annotating crisis texts. 3.4. Few-shot Prompting Few-shot prompting is a method for improving the performance of LLMs in a specific task. Examples and solutions are provided to the LLM allowing it to learn from the prompt when performing the task. For example, in a translation task, example texts in the source language and their translations in the target language are supplied within the prompt. We explored two different ways to supply the examples for few-shot prompting: In prompt: This is the conventional way of supplying few-shot examples. In this case, we iteratively insert the examples into the prompt in place of ’<EXAMPLES>’ placeholder. Examples are in the form: Input: ’example text’ and Output: {”classification”: ”label”}. In chat: In this method we utilize the fact that LLMs retain the context of previous messages sent during a conversation. LLMs use chat templates to simulate a conversation. An example chat template is show in Figure 2. The role ’user’ is reserved for sending prompts to the LLM, while role ’assistant’ is the response from the LLM. We use this template to iteratively provide our whole prompt including the text to be annotated, and the example response of the LLM to simulate a conversation where by the LLM has been prompted with several texts to annotate. This method is more computationally expensive and the model’s learning is dependent on its context length. Table 2shows the texts we used as few-shot prompting examples. We included examples for ’fire’ and ’flood’ which includes some tricky sentences that might be incorrectly annotated. We also included ’none’ examples that includes texts that are not relevant, expressing sympathy, and asking for donations. 6https://platform.openai.com/docs/guides/prompt-generation J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation84 Table 2. Few-shot examples employed in our evaluation. text label description Firefighters are battling a 100 acre fire off of Lambs Canyon Road and Dump Road, near Beaumont (Riverside County). #HillsFire fire Relevant to fire Fires are turning the sky red. Last night #Humboldt #California #Fire #NationalPhotographyDay fire Relevant to fire Widespread flood damage in the Montecito area has significantly compromised water service to our.. flood flood Relevant to flood Santa Barbara had to deal with the #ThomasFire and is now dealing with mudslides from heavy rain hope everyone back home stays safe! flood Relevant to flood My heart goes out to all those affected by the Santa Barbara fire none Expressing sympathy good to know who to boycott.Millionaires. none Not relevant There will be a fund raiser for those affected by the floods in Montecito, visit this website if interested.... none Asking for donations 3.5. Label Cleaning The performance of machine learning models is directly tied their training data. This is why proper annotation of data is important in any text classification task. We experimented with cleaning the labels suggested by the LLMs in zero-shot prompting to improve the quality of the annotations. For this purpose, we use cleanlab7an open-source framework from CleanlabAI. Cleanlab was developed on the idea of confidence learning [14] which is a data-centric approach based on identifying error labels using the predicted probabilities of a machine learning model. We performed label cleaning using the suggested process in the cleanlab documentation8.A logistic regression model is used to generate predicted probabilities of the data set after zero-shot annotation, then data with error labels are separated, and a label quality score within the range [0,1] is provided for each sample. The remaining clean data is then used to train a logistic regression classifier which is then used to classify the data with error labels thereby cleaning them. We further experimented with score thresholding using the label quality score provided by cleanlab. We evaluated two thresholds 0.3, and 0.5. For each threshold, all data samples in the clean data (i.e. the data without error labels by cleanlab) with label quality value below the threshold are considered as having error labels and are also cleaned using the logistic regression classifier. 3.6. Evaluation metrics We used the standard metrics for evaluating performance in text classification; precision which is the proportion of instances correctly classified, Recall which is the proportion of correctly classified instances out of the actual instances, and F1-score which is the harmonic mean of precision and recall used to balance both metrics. We calculated the individual precision and recall for each label instance (i.e. fire, flood, none) and then calculated the F1-score for each label. The mean F1-score of all labels is the main focus in our analysis. 7https://github.com/cleanlab/cleanlab 8https://docs.cleanlab.ai/stable/index.html J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation 85 4. Results and discussion Table 3. Auto-annotation results on English data sets. model name mean f1-score zero-shot zero-shot with label cleaning few-shot mean no score-thresholding 0.3 0.5 in-prompt in-chat eurollm_9b 0.649 0.668 0.648 0.651 0.739 0.571 0.654 gemma2_9b 0.702 0.700 0.688 0.682 0.658 0.718 0.691 gemma3_4b 0.696 0.695 0.681 0.677 0.642 0.630 0.670 mistral_7b 0.757 0.768 0.767 0.766 0.722 0.743 0.754 occiglot_7b_eu5 0.689 0.743 0.740 0.738 0.666 0.742 0.720 phi3medium 0.732 0.744 0.742 0.743 0.673 0.767 0.734 phi4mini 0.725 0.737 0.731 0.730 0.650 0.677 0.648 qwen2.5_14b 0.771 0.779 0.776 0.774 0.681 0.721 0.750 qwen2.5_7b 0.737 0.748 0.747 0.753 0.707 0.741 0.739 salamandra_7b 0.737 0.753 0.752 0.756 0.704 0.589 0.715 mean 0.720 0.734 0.688 0.706 0.684 0.690 4.1. RQ1: Are LLMs acceptable zero-shot annotators of crisis social media messages? Table 3presents the results of experiments performed for zero-shot prompting, label cleaning with score thresholding, and few-shot prompting. All models demonstrate strong performance in most settings, with high average F1 across all experiments. For zero-shot prompting, all models perform well enough in English to be considered effective as auto-annotators. Qwen2.5 14B performed the best in zero-shot which may be due to its parameter size, as larger parameter models follow instructions better. But, the model size does not reflect exponential increase in performance as the 7B and 4B models are not far behind. Phi4mini is worth noting, being a 4B parameter model, it performs better than some 7B and 9B models, and almost as well as the older generation 14B model from Microsoft—phi3medium. 4.2. RQ2: Can label cleaning powered by machine learning or few-shot prompting improve zero-shot annotation performance? With respect to RQ2, from Table 3we can compare the results of zero-shot against the results from the label cleaning and few-shot experiments. For label cleaning in zero-shot annotation, most models show improved performance after post-annotation label correction, while score thresholding typically results in a performance decline. This means that the significant label errors are caught by cleanlab and it is not necessary to filter further using the label quality score. In the few-shot experiments overall performance tends to be lower than zero-shot, and whether in-prompt or in-chat performs better largely depends on the specific model used. This underperformance is likely due to the few-shot examples supplied to the models. The models overfit to these specific samples, resulting in a loss of task generalization. While supplying a larger number of examples could address this issue, it may lead to significantly higher computational costs. Thus, we conclude that zero-shot with label cleaning is the preferred method for getting the best results. J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation86 Table 4. Zero-shot auto-annotation results with label cleaning on multilingual data sets. model name mean f1-score mean across languages en ca es de eurollm_9b 0.668 0.643 0.645 0.605 0.640 gemma2_9b 0.700 0.662 0.672 0.673 0.677 gemma3_4b 0.695 0.670 0.680 0.673 0.680 mistral_7b 0.768 0.734 0.749 0.749 0.750 occiglot_7b_eu5 0.743 0.296 0.448 0.394 0.470 phi3medium 0.744 0.668 0.704 0.692 0.702 phi4mini 0.737 0.701 0.712 0.723 0.718 qwen2.5_14b 0.779 0.744 0.746 0.744 0.751 qwen2.5_7b 0.748 0.712 0.717 0.718 0.724 salamandra_7b 0.753 0.739 0.730 0.732 0.738 mean per language 0.734 0.656 0.680 0.670 4.3. RQ3: Can current state-of-the-art LLMs adapt to multilingual crisis social media messages? Given the performance of the few-shot and score thresholding experiments, we only considered zero-shot annotation with label cleaning and no score thresholding in our multilingual experiments. Table 4shows the results using zero-shot annotation and label cleaning in the multilingual setting. As expected, for most models there’s a drop in performance when comparing English with the other languages because they are pretrained on mostly English data. The top-performing models overall are Mistral 7B, the two Qwen2.5 variants, and Salamandra7B—each achieving the highest score in at least one of the evaluated languages. Mistral 7B is the highest performing model in Spanish and German, despite being trained with primarily English data, while Salamandra 7B leads in Catalan, benefiting from access to high-quality Catalan training data. Meanwhile, Qwen2.5 14B stands out as the top performer in English likely due to its parameter size. Although Gemma3 4B is significantly smaller than its predecessor Gemma 2 9B, they perform similarly across languages. This might be due to better training data and a larger context window. Lastly, the Occiglot model performs reasonably well in English but shows a significant performance drop in other languages. In summary, the results show that the models adapt well to multilingual texts, especially in the case of Catalan which is a low resource language. 5. Conclusion In this work, we have evaluated the performance of current generation LLMs as annotators of climate-related social media messages published during a crisis. We first evaluated their zero-shot capability and found them to perform reasonably well. In addition, we evaluated if their zero-shot performance could be improved with machine learningenabled label cleaning or few-shot prompting. The results indicated that label cleaning is the more effective approach, while few-shot prompting tends to result in weaker performance. Finally, we evaluated their adaptability to multilingual texts, focusing on zeroshot with label cleaning. The results showed decent performance across most models in J.A. Orama et al. / Evaluating Instruction-Tuned Large Language Models for Automatic Annotation 87