Label Set Optimization via Activation Distribution Kurtosis for Zero-Shot Classification with Generative Models

Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31713–31730 November 4-9, 2025 ©2025 Association for Computational Linguistics Label Set Optimization via Activation Distribution Kurtosis for Zero-Shot Classification with Generative Models Yue Li, Zhixue Zhao and Carolina Scarton Department of Computer Science, University of Sheffield, UK {yli381,zhixue.zhao,c.scarton}@sheffield.ac.uk Abstract In-context learning (ICL) performance is highly sensitive to prompt design, yet the impact of class label options (e.g. lexicon or order) in zero-shot classification remains underexplored. This study proposes LOADS (Label set Optimization via Activation Distribution kurtosiS), a post-hoc method for selecting optimal label sets in zero-shot ICL with large language models (LLMs). LOADS is built upon the observations in our empirical analysis, the first to systematically examine how label option design (i.e., lexical choice, order, and elaboration) impacts classification performance. This analysis shows that the lexical choice of the labels in the prompt (such as agree vs. support in stance classification) plays an important role in both model performance and model’s sensitivity to the label order. A further investigation demonstrates that optimal label words tend to activate fewer outlier neurons in LLMs’ feed-forward networks. LOADS then leverages kurtosis to measure the neuron activation distribution for label selection, requiring only a single forward pass without gradient propagation or labelled data. The LOADS-selected label words consistently demonstrate effectiveness for zero-shot ICL across classification tasks, datasets, models and languages, achieving maximum performance gain from 0.54 to 0.76 compared to the conventional approach of using original dataset label words. 1 Introduction Generative large language models (LLMs) are increasingly used for classification tasks via zeroshot in-context learning (ICL), where models are prompted to select an option from a pre-defined set of labels (Wang et al.,2022;Antypas et al.,2023; Gonen et al.,2023;Mu et al.,2024). While some classification tasks employ a relatively fixed set of lexicons to represent class labels, such as sentiment analysis (positive and negative) and textual entailment (entailment and contradiction), other tasks may present more ambiguous choices in lexical selection. Stance classification, for instance, uses diverse pairs of antonyms to represent positive and negative stances across different datasets, e.g. agree-disagree vs. favor-against. As a result, when crafting prompts for these classification tasks, practitioners face decisions regarding label options in the prompt, such as lexical selection and ordering. Despite studies suggesting the sensitivity of ICL to prompt design (Lu et al.,2022;Yoo et al.,2022; Wei et al.,2024;Mao et al.,2024;Liu et al.,2024a; Zhang et al.,2022;Liu et al.,2022b;Peng et al., 2024;Gonen et al.,2023;Mu et al.,2024), this subtle yet critical consideration of label options in prompt for zero-shot ICL has received limited attention. To fill in this research gap, we explore the impact of three types of label variants (i.e., lexical choice, label order and elaborations) in zero-shot ICL with both encoder-decoder and decoder-only LLMs. We mainly ground our research on stance classification, a task where label adaptation is a known problem due to various label inventories in different studies (Hardalov et al.,2021). We demonstrate that the lexical choice of the label options significantly impacts model performance. The model’s sensitivity to the label order also depends on the lexical choice, while elaborating on task-related information (e.g. agree with the claim elaborating agree) has minimum effect. Inspired by recent studies on neuron analysis (Kuzmin et al.,2023;Voita et al.,2024;Stolfo et al.,2024;Kurz et al.,2024), we further investigate the neurons in the feed-forward network (FFN) in the decoder of the LLMs. We empirically show that prompts with optimal label sets activate fewer outlier neurons. Consequently, we propose a new method, Label set Optimization via Activation Distribution kurtosiS(LOADS), to select optimal label sets for a given classification dataset in zeroshot ICL. LOADS could stably and effectively work with only 100 unlabelled samples of the val31713 Figure 1: Illustration of the three aspects (i.e., lexical choice, label order and label elaboration) for designing the label option in the prompt in zero-shot ICL for classification, and our LOADS to post-hoc select the optimal label set (top half figure). idation dataset, also demonstrating transferability across datasets and languages. Our contributions are summarized as follows: • The first benchmark on how variants of label options in prompts affect zero-shot ICL models’ performance for classification tasks. We provide useful recommendations on label designing to practitioners working on zero-shot classification with LLMs 1. • The empirical demonstration that zero-shot ICL performance negatively correlates with the number of outlier neurons in FFN when varying the lexical choices for label options. The correlation holds true across diverse English stance classification datasets and topic classification datasets with different models. • Anovel and efficient post-hoc method (LOADS) for label selection in zero-shot ICL. Compared with common strategies in practice, our approach demonstrates statistically significant performance improvements across model types, model sizes and languages with only 100 unlabeled data samples. Our analysis also suggests that the LOADS-selected label set is potentially transferable across similar datasets for a specific LLM, further alleviating the cost to collect samples for a target new dataset. We present our experimental setups, results and discussions on the impact of label options in zeroshot ICL in Section 3. Then, we describe our neuron analysis of the lexical choice in label options in Section 4and our proposed method LOADS for selecting optimal label sets in Section 5. 1 Code and resources: https://github.com/YLi999/ Stance_LOADS. 2 Related Work ICL Performance Few-shot ICL mainly focuses on cases where LLMs are directly prompted with N demonstration examples. The studies highlight the substantial impact of example ordering (Lu et al.,2022), formatting (Yoo et al.,2022;Wei et al., 2024;Mao et al.,2024;Liu et al.,2024a), and examples selection (Zhang et al.,2022;Liu et al.,2022b; Peng et al.,2024). The parallel lines of work focus on improving few-shot ICL via the optimal selection or arrangement of examples (Liu et al., 2022b;Rubin et al.,2022;Lu et al.,2022;Zhang et al.,2022;Liu et al.,2024b;Xu et al.,2024), reweighting examples (Yang et al.,2023), automatic reformat or generation of demonstration representations (Kim et al.,2022;Liu et al.,2023), and introduction of intermediate reasoning steps (Wei et al.,2022;Zhang et al.,2023b). However, the impact of lexical choices for label names in classification received little attention, with the only closely related work suggesting that LLMs are likely to confuse classes which share similar key vectors in the attention modules (Wang et al.,2023). For zero-shot ICL, Mu et al., (2024) demonstrate the effect of using synonyms for class options, but they neither consider the order of the label options nor propose an effective strategy to choose the label names. Gonen et al., (2023) empirically show that perplexity could be an effective indicator for prompt selection, but they do not account for class options. Notably, behavioral differences between few-shot and zero-shot ICL have been frequently observed, suggesting that findings from few-shot ICL do not necessarily hold in the zero-shot context (Lin and Lee,2024). 31714 Prompt-Tuning and Verbalizer Prompt-tuning aims to automatically find or generate an optimal discrete prompt (e.g., through gradient-based search (Shin et al.,2020;Shi et al.,2023) or fine-tuning (Gao et al.,2021;Le Scao and Rush, 2021;Deng et al.,2022)) or by training continuous prompt (Lester et al.,2021;Liu et al.,2022c). Verbalizer can be taken as a mapping function that links discrete class labels to corresponding tokens or phrases in a model’s vocabulary. A range of methods developed to build the verbalizer, including manually created verbalizer (Schick and Schütze,2021), search-based verbalizer that identifies label words automatically from the dataset (Gao et al.,2021;Shin et al.,2020), and soft verbalizers that uses continuous embeddings obtained through fine-tuning (Hambardzumyan et al.,2021; Cui et al.,2022). Prompt-tuning often does not focus on label set selection for zero-shot ICL, and the verbalizer introduces additional components to the decoding of the generative models, distinguishing it fundamentally from our work. 3 Prompting with Varied Label Options for Zero-shot Classification In zero-shot ICL for classification, a common approach is to provide a set of class label options in the prompt to instruct the LLMs to choose one of the options as the classification prediction. Although the label option is a subtle component in the prompt, we are interested in whether it has a significant impact on model performance. Specifically, we explore three types of variants around label options in the prompt: (1) lexical choice; (2) label order; and (3) label elaboration. To accurately measure the impact of these factors, we only manipulate the label options within the same prompt template (Section 3.2). We show examples of the three variants in Figure 1. 3.1 Methodology Lexical Choice We use single-word synonyms to represent class labels (e.g. support and agree), forming various label sets. For each dataset, we compare the zero-shot ICL performance when LLMs are prompted to select from different label sets, as illustrated in Figure 1. For this purpose, we design a pipeline to create a pool of label sets: 1. Collect a seed set of label names. We obtain this set by collecting the label names in the datasets we experiment with. 2. Expand label sets with WordNet and LLMs. We use WordNet (Fellbaum,1998) as a reliable source and Claude 2 as a supplementary source to obtain synonyms for label names in the seed set. For pairs of label names with semantically opposite meanings (such as “agree" and “disagree"), we also consider antonyms to avoid potential ambiguity and present clear contrast for the predicted models. 3. Manual selection. We manually filter out semantically unrelated or inappropriate label sets generated by Claude to mitigate the impact of noisy label names. The label names are arranged in the sequence presented in their original study (see Table 1). We refer to this arrangement as the default order. Label Order We consider every possible order of the single-word labels in the prompt and compare the model performance against that obtained with the default order. For binary datasets, there is only one alternative arrangement besides the default order, while N -way multi-class classification would yield N!−1alternative orders. Label Elaboration We investigate whether transforming single-word labels (e.g., “agree") into more detailed phrases (e.g., “agree with the claim") has an impact on model performance. On the one hand, elaborating on task details with the label may provide the model with a stronger alignment signal between the label and the task, emphasizing what the label is referring to. On the other hand, it also increases the label length and may introduce noise in the prompt (Liu et al.,2024a). Therefore, we design three levels of elaborations (shorted for E1,E2 and E3) by progressively adding more taskrelated (and potentially redundant) information to the single-word label names, as shown in Figure 1. 3.2 Experimental Setups Datasets We focus on the stance classification task due to its rich label inventories across various readily available datasets. Stance classification aims to identify the type of an expressed opinion (e.g., “agree" or “disagree") in a given piece of text towards a particular topic, claim, or entity. We consider four binary (scd (Hasan and Ng,2013), perspectrum (Chen et al.,2019), snopes (Hanselowski et al.,2019) and ibmcs (Bar-Haim et al.,2017)) and 2https://claude.ai/new 31715 Dataset Name Original Label Words Optimal Label Words scd for, against pro, con perspectrum support, undermine validate, refute snopes agree, refute affirm, refute ibmcs pro, con endorse, deny vast pro, con, neutral confirm, dispute, neither emergent for, against, observing endorse, reject, neutral semeval favour, against, neither accept, reject, neutral rumoureval support, deny, query, comment confirm, reject, question, neutral arc agree, disagree, discuss, unrelated affirm, refute, discuss, unrelated Table 1: Lists of the English stance classification datasets, their labels in original dataset, and the optimal labels with the highest zero-shot ICL performance on Flan-T5-xl as an example to justify our motivation on LOADS. five multi-class datasets (vast (Allaway and McKeown,2020), emergent (Ferreira and Vlachos,2016), semeval (Mohammad et al.,2016), rumoureval (Gorrell et al.,2019) and arc (Hanselowski et al., 2018)) from existing English stance classification benchmarks (Schiller et al.,2021;Hardalov et al., 2021;Chen et al.,2023), as shown in Table 1. The nine datasets cover different domains, such as social media posts, news articles and online debates forums. We also experiment with topic classification in Section 4and 5to demonstrate the generalizability of our findings to other NLP tasks. Models We cover both encoder-decoder and decoder-only LLMs, and experiment with the prevalent open-sourced Flan-T5 (Chung et al., 2024), Llama 3 and Llama 3.1 (Dubey et al.,2024) model families as representatives for these two types of LLMs. We choose their moderate-sized instruction-tuned versions, Flan-T5-xl (3b), Llama -3-Instruct (8b) and Llama-3.1-Instruct (8b), as our primary models for investigation due to our hardware resources constraints and their decent zeroshot ICL performance (Aiyappa et al.,2024;Chung et al.,2024;Dubey et al.,2024). We conduct experiments with Gemma-2-it (9b) 3 and Flan-T5xxl (13b) to further show the generalizability of LOADS in Section 5. Prompt Template We refer to the prompt template used in the supervised fine-tuning of FlanT5 and Llama (Wang et al.,2022;Chung et al., 2024;Dubey et al.,2024). We present the results with the following prompt template in this paper: Given a [text1_name] and a [text2_name], detect the stance that the [text2_name] has towards the [text1_name]. There are {N} options: “{label 0 }", “{label 1 }, ... , and {label N−1 }". Now complete the following example. [text1_name]: 3https://huggingface.co/google/gemma-2-9b-it {text1}. [text2_name]: {text2}. We also test other templates and find the results are consistent (e.g., prompting with label explanations in Appendix E). Evaluation We adopt macroF1 for model performance evaluation to align with prior studies (Schiller et al.,2021;Hardalov et al.,2021;Chen et al.,2023). We use wF 24 to account for data imbalance in rumoureval (Scarton et al.,2020). Implementation Details To ensure reproducibility, we use greedy search for decoding 5 . In more than 95% cases, LLMs exactly follow the instruction and output the stance name within the required stance options. Therefore, we directly use the model generation as the predicted label without post-processing or mapping. We run experiments on the validation set of each dataset. See Appendix Cfor details on the label sets experimented with. We exclude the topic/entities-based (such as stance towards Obama) stance classification datasets (i.e. scd, semeval and vast) in label elaboration experiments to avoid unnecessary ambiguity or bias during elaboration (e.g., the text to be classified may refer to anything from policy to personal behavior about Obama, and elaborating agree to agree with the opinion of Obama could lead to biased prediction). 3.3 Results and Analysis We first discuss the impact of lexical choice, label order, and label elaboration on zero-shot ICL for classification. Then we provide suggestions for practitioners in zero-shot ICL for classification. Lexical Choice As shown in Table 2, performance varies across datasets and models solely due to changes in the label names within the prompt. The variations are more pronounced than those reported in previous studies (Mu et al.,2024). The gap between the highest and lowest performance exceeds 0.1 for all datasets and models, and surpasses 0.2 on more than half of the datasets. We observe that certain stance labels could potentially trigger biased predictions, leading to extremely low performance. For example, Flan-T5 tends to always output support when using support,deny, neither for the semeval dataset, and Llama 3 overly predicts con when prompted by the label set pro, 4wF 2 gives different weights for each stance: deny = support = 0.40,query = 0.15 and comment = 0.05. 5 Potential impact of the decoding strategy can be found in the Appendix D. 31716 Fscore perspectrum ibmcs snopes scd emergent semeval rumoureval arc vast Llama 3 (8b) max 0.869 0.834 0.770 0.759 0.740 0.688 0.659 0.453 0.382 min 0.738 0.592 0.540 0.639 0.387 0.544 0.286 0.249 0.173 Original 0.799 0.679 0.656 0.709 0.467 0.643 0.521 0.364 0.219 avg±var 0.824±0.001 0.756±0.003 0.666±0.003 0.713±0.001 0.547±0.006 0.624±0.001 0.514±0.008 0.329±0.002 0.279±0.002 Llama 3.1 (8b) max 0.909 0.898 0.748 0.769 0.660 0.735 0.584 0.466 0.409 min 0.376 0.300 0.435 0.629 0.166 0.456 0.320 0.252 0.175 Original 0.809 0.759 0.660 0.749 0.494 0.676 0.480 0.426 0.237 avg±var 0.832±0.009 0.805±0.011 0.668±0.003 0.737±0.001 0.496±0.013 0.652±0.003 0.488±0.003 0.387±0.002 0.282±0.002 Flan-T5xl (3b) max 0.940 0.960 0.809 0.776 0.743 0.706 0.761 0.685 0.496 min 0.836 0.807 0.576 0.449 0.487 0.166 0.281 0.358 0.155 Original 0.939 0.939 0.746 0.631 0.649 0.467 0.381 0.493 0.311 avg±var 0.899±0.001 0.901±0.002 0.695±0.004 0.661±0.009 0.626±0.003 0.539±0.012 0.520±0.010 0.507±0.008 0.328±0.008 Table 2: The maximum (max), minimum (min), average (avg), variance (var) of the model performance across different label names in the prompt for each validation set. The performance of the original label set (Original) is also included, showing that they fail to reach the maximum performance LLMs could get. The extent of the gap between the maximum and minimum performances is represented using colors: max−min>0.3 , 0.2<max−min<0.3 . The greater the variance, the greater the impact of label lexical choice, and the greater the potential utility of optimizing the label set. con,neutral for the emergent dataset. Label Order On average, we observe limited influence of label order (see the Appendix Ffor details). However, we are also interested in extremes of performance change caused by shifting label orders, particularly the maximum performance gain and drop, and whether the extreme gain or drop correlates with the lexicons used for the label names. Therefore, for each dataset, we first select the topk 6 optimal and poor label sets based on their performance in the same order (i.e. the default order). We then examine the maximum increase or decrease of the performance after re-arranging the label orders for optimal and poor label sets, respectively. We find that altering the order for the optimal label sets has the risk of high performance drops (e.g., even more than 0.2 on the binary classification snopes dataset, Figure 2a in Appendix). While performance improvements are possible, the gains are relatively limited (lower than 0.1 on all datasets). Conversely, re-arranging the order for the poorly performing label sets offers potential for substantial improvement (Figure 2b). This high improvements for certain label sets after re-ordering suggests that the poor performance may partly stem from the initial sub-optimal ordering. Label Elaborations Similarly, we select the topk 7 optimal and sub-optimal single-word label sets based on their performance, and examine the per6 Due to the exponential increasing of label order options and our limited computational resources, we set k=15 for binary classification; k=10 for three-way classification and k=2 for four-way classification 7 k=15 for binary classification, k=30 for three/four-way classification. formance change after elaborations. We observe that the models are robust to the elaborations for either optimal or poor single-name label sets (full results in Table 9), indicating that adding task related details or increasing the label token lengths brings limited impact on average. However, we also observe relatively large performance change on certain datasets. Specifically, for the rumoureval dataset with Llama 3, we notice a performance drop larger than 0.2 when elaborating an optimal label set. Performance on the ibmcs dataset with Llama 3.1 could increase 0.2 when elaborating a poorlyperforming label set. 3.4 Suggestions to Practitioners Based on our results and analysis, we provide the following suggestions to practitioners in zero-shot ICL for classification: 1. Lexical designing for the label names should be considered as an important step in prompt engineering. 2. Single-word class label without elaboration on task information is able to achieve high performance in most cases, i.e. adding extra information does not yield better results. 3. If the practitioner has selected a set of optimal lexicons for label options based on a specific order, exploring alternative label orders can be redundant due to the limited performance gain brought from high computational costs. However, if a label set is chosen randomly, experimenting with different label orders may yield meaningful improvements (see Figure 2). 31717 Model Stance Classificaion Topic Classification perspectrum ibmcs snopes scd emergent semeval rumoureval arc vast TweetTopic AG News Llama 3 (8b) -0.4921* -0.3787* -0.4359* -0.4217* -0.5781 -0.2618* -0.3764* -0.1662 -0.3639* -0.4994* -0.2447* Llama 3.1 (8b) -0.4103* -0.3642* -0.1874 -0.0686 -0.4310* -0.2232* -0.3944* -0.1708* -0.1208 -0.2196* 0.0200 Flan-T5-xl (3b) -0.4476* -0.3638* -0.6014* 0.2353 -0.1089 -0.2881* -0.1714* -0.5742 0.1003 0.1587 0.0398 Table 3: Spearman correlation co-efficiency between model performance on validation set and kurtosis of neurons in the last layer. Mark with * when p value is lower than 0.05. 4 Neuron Analysis for Label Selection Although our findings indicate the importance of label word selection for text classification in zeroshot ICL, current studies lack consideration of this factor. Therefore, we conduct empirical analysis to gain insights into the underlying mechanism of lexical choice for single-word label names. We preliminarily explored related approaches discussed in Section 2, including prompt perplexity (Gonen et al.,2023) and model internal representation of label words (Wang et al.,2023), but they did not yield any significant correlation with zero-shot ICL model performance (see the Appendix Jand Kfor details). Meanwhile, various studies have indicated the correlation between neuron activation pattern in FFN and model performance (Kuzmin et al.,2023;Tang et al.,2024; Stolfo et al.,2024;Wu et al.,2024). Inspired by the finding that the presence of outliers in neural networks is predictive of quantization and pruning performance for the layers of LLMs (Kuzmin et al., 2023), we establish a new hypothesis: the model performance influenced by label names is correlated with the number of outliers in the neurons within FFN in the decoder of the LLMs. We empirically validate our hypothesis on the nine stance classification datasets, as well as two topic classification datasets (AG News (Zhang et al.,2015) and TweetTopic (Antypas et al.,2022)) to show the generalizability to other NLP tasks. Methodology For each FFN module in layer i in the decoder, it can be denoted as follows: hi= (act_fn(˜ hiWi 1)⊗˜ hiWi 3)·Wi 2.(1) where ˜ hi is the output hidden states from multihead self-attention module. The activation function ( act_fn ) for Flan-T5 and Llama 3/3.1 is Gaussian Error Linear Unit (GELU) (Hendrycks and Gimpel, 2016) and Sigmoid Linear Unit (SiLU) (Hendrycks and Gimpel,2016;Elfwing et al.,2018), respectively. Aneuron is defined as the linear transformation of each column in Wi 1 followed by the activation function. Here, we study the last layer I ’s output of act_fn(˜ hIWI 1) (denoted as NI ) for the predicted first token of the label name in the model generation. Following Kuzmin et al., (2023), we measure the number of outliers over the neuron output distribution (NI) through kurtosis, given by: Kurtosis[NI] = E[(NI−µ)4] (E[(NI−µ)2])2(2) where µ is the mean of NI . For each dataset, we average the kurtosis scores over the validation set for each candidate label set. We then calculate the Spearman correlation between model performance and the averaged kurtosis score. Results Table 3shows that, for most datasets, there is statistically significant negative correlation between model performances and kurtosis scores across models and activation functions, indicating that fewer outliers in the neurons of the final layers are associated with enhanced zero-shot ICL performance. This observation implies that the kurtosis score of neuron activation distribution in FFN of the last decoder layer of LLMs could potentially serve as an effective signal for selecting optimal label names in zero-shot classification. 5 LOADS: Label set Optimization via Activation Distribution kurtosiS Motivated by the above observation that the fluctuated zero-shot ICL performance caused by different label names could be attributed to the number of outliers in neurons in the last layer of LLMs, we propose LOADS to obtain an optimal label set for a given classification task in a post-hoc setting. 5.1 Method We design a three-step pipeline based on LOADS for automatic label selection in zero-shot ICL: 1. Create a list of candidate label sets for class options in the prompt (see Section 3). The label names in each set should follow the same order. 2. Rank the list of label options based on the kurtosis score of the neuron activation in FFN of 31718 Model Method Stance Classification Topic Classification perspectrum ibmcs snopes scd emergent semeval rumoureval arc vast TweetTopic AG News Llama 3 (8b) LOADS 0.8431 0.7684 0.5516 0.6698 0.5870 0.6445 0.4097 0.3662 0.3190 0.7752 0.7660 Original Label 0.8187 0.5485 0.4387 0.6504 0.3416 0.5565 0.3487 0.3130 0.2395 0.7742 0.7594 Original + Verbalizer 0.8185 0.5485 0.4306 0.6578 0.3400 0.5576 0.3476 0.3131 0.2395 0.7742 0.7594 Self-generated 0.6912 0.5802 0.3314 0.6607 0.3692 0.6032 0.2745 0.2837 0.3070 0.5851 0.6235 Llama 3.1 (8b) LOADS 0.8789 0.8856 0.6212 0.7274 0.6403 0.6581 0.3642 0.3637 0.2458 0.7988 0.7306 Original Label 0.8064 0.6783 0.4426 0.6983 0.4337 0.6492 0.3528 0.3784 0.2126 0.7909 0.7273 Original + Verbalizer 0.8064 0.6783 0.4426 0.6983 0.4262 0.6448 0.3534 0.3838 0.2128 0.7909 0.7273 Self-generated 0.6244 0.4918 0.5146 0.6798 0.2984 0.6421 0.3409 0.3373 0.2578 0.5956 0.6217 Gemma 2 (9b) LOADS 0.9110 0.9247 0.6233 0.7595 0.5989 0.6784 0.4896 0.4858 0.3439 0.8032 0.8451 Original Label 0.8966 0.8728 0.6454 0.7707 0.5704 0.6874 0.4778 0.4858 0.3383 0.8241 0.8316 Original + Verbalizer 0.8966 0.8728 0.6424 0.7707 0.5523 0.6873 0.4689 0.4748 0.3383 0.8246 0.8322 Self-generated 0.9017 0.9113 0.6656 0.7480 0.5833 0.6621 0.3549 0.4657 0.3178 0.6303 0.8215 Flan-T5-xl (3b) LOADS 0.9334 0.9380 0.6881 0.5997 0.5813 0.4951 0.4759 0.4688 0.3857 0.8530 0.8556 Original Label 0.9305 0.8971 0.7267 0.6341 0.5580 0.5697 0.3837 0.4628 0.3473 0.8071 0.9209 Original + Verbalizer 0.9305 0.8953 0.7026 0.6507 0.5586 0.5712 0.3852 0.4613 0.3482 0.8091 0.9209 Self-generated 0.7914 0.7384 0.3677 0.6974 0.3883 0.5954 0.2986 0.3343 0.3042 0.8501 0.7368 Flan-T5-xxl (13b) LOADS 0.9428 0.9644 0.6905 0.6158 0.5938 0.5257 0.3852 0.6151 0.4218 0.7727 0.7742 Original Label 0.9407 0.9630 0.7814 0.7598 0.5614 0.5697 0.2016 0.6065 0.3278 0.6643 0.9177 Original + Verbalizer 0.9407 0.9621 0.7716 0.7598 0.5631 0.5705 0.2011 0.6062 0.3278 0.6643 0.9177 Self-generated 0.8622 0.8384 0.4226 0.7315 0.4309 0.6182 0.3110 0.6115 0.2881 0.7242 0.9177 Table 4: Comparison of zero-shot ICL performance on test sets between prompting with LOADS-selected label names versus the other three baseline approaches. We underline the highest model performance (statistically significant with paired chi-squared test). the last decoder layer (averaged across the validation set). 3. Choose the label set with the lowest averaged kurtosis score. The above LOADS-selected label set can then be used in the standard zero-shot ICL on test sets. 5.2 Evaluation Setups We randomly sample 100 data points from the validation set for label selection and test the selected label sets on the official test sets. • Baselines: We compare the model performance of using label words selected by LOADS to the following three approaches: (1) Original label words: we use the original label words from the dataset (i.e., the labels in Table 1), as it is the conventional and widely adopted practice; (2) Original label words with a verbalizer: after prompting the LLMs with the original label words, we employ our pool of candidate label words (See Section 3.1) as a verbalizer and incorporate their probabilities into the final probability of the same class; (3) Self-generated label words: we prompt the LLMs without providing any class options and select the candidate label words with the average highest probability at the first generated label token. • LLMs: In addition to Llama3 (8b), Llama 3.1 (8b) and Flan-T5-xl (3b), we also examine whether LOADS could generalize to other model families and model sizes by including instructiontuned Gemma-2-it (9b) and Flan-T5-xxl (13b). Results Table 4presents the zero-shot ICL model performances on stance classification and topic classification datasets with different label word selection strategies. The results demonstrate that employing LOADS to select label sets for zeroshot ICL prompts yields superior performance compared to other baseline approaches on most of the datasets. The improvement is consistent across NLP tasks and datasets, model architectures and sizes, as well as prompt templates8. Also, we observe limited benefits from adopting the verbalizer in post-processing, since LLMs tend to give the predicted label word tokens high probability in most of cases. Prompting with the self-generated label words rarely results in the best performance, while with the risk of leading to extremely low performance on certain datasets (e.g., perspectrum and snopes datasets with Llama 3). Furthermore, potential data leakage could have significant impact on LOADS, indicated by the high performance achieved by the original label words on the AG News dataset with Flan-T5 models which was instruction-tuned with this dataset. It also aligns with the results in Table 3where no statistically negative correlation was observed on the AG news dataset with Flan-T5-xl. 5.3 Analysis Cross-lingual Transferability Previous research (Zhang et al.,2023a) has demonstrated that for non-English datasets, prompting with English task 8 The analysis of prompt sensitivity can be found in the Appendix H. 31719 instructions (including label words) while keeping the input in the original language often yields superior performance than non-English instructions. Therefore, we investigate whether the optimal English label sets selected by LOADS on English dataset can also enhance performance for nonEnglish datasets in this scenario. We manually translate the English rumoureval Twitter test set into French and Portuguese. We select the optimal label set based on LOADS with 100 randomly sampled data from the English rumoureval validation set. We then prompt the LLMs with English task instructions and French/Portuguese inputs. Table 5presents the results, indicating that the optimal English label set identified by LOADS also effectively improves performance on non-English datasets when the instruction is provided in English. Model Method French Portuguese Llama 3 (8b) LOADS 0.5020 0.4528 Original Label 0.4544 0.4284 Llama 3.1 (8b) LOADS 0.4728 0.4278 Original Label 0.3731 0.3679 Flan-T5-xl (3b) LOADS 0.5137 0.3912 Original Label 0.4189 0.3317 Table 5: Performance when LLMs are prompted with English instructions (including label options) and French/Portuguese inputs. Label options are selected by LOADS with English validation data. Data Efficiency To show the merit of data efficiency of LOADS, we randomly sample 50, 100, 300, 500 or 1000 data points from the validation set and compare the rankings of the label sets based on LOADS. Due to the resource restriction, we conduct experiments on snopes (binary classification) and emergent (three-way classification) datasets with Flan-T5-xl and Llama 3. The results show that the rankings of the top 5 label sets remain consistent across different sample sizes. It suggests that LOADS can achieve comparable performance even with a smaller number of unlabeled data samples than 100, further highlighting the data efficiency of our proposed method. We provide computational cost estimation of LOADS using 100 unlabelled samples in Appendix I. Label Transferability We explore whether LOADS-selected label sets can be generalised across datasets or even models for NLP tasks such as stance classification where different label lexicons are used to represent the same classes across datasets. Specifically, for each dataset Di on each LLM Mj , we select the optimal label set LDiMj through LOADS. To analyse whether the LOADSselected label set on one dataset could be adapted to another related dataset with the same LLM M , we calculate the overlap of optimal labels ( LDi_M ) between each dataset. Similarly, we examine the overlap of optimal labels ( LD_Mi ) between each model to explore whether the LOADS-selected optimal labels for dataset D could be adapted across LLMs. We only focus on the positive and negative stances to enable comparison across binary, threeway and four-way stance classification datasets. Our results indicate that LOADS-selected label sets is transferable across datasets on the same LLM, highlighting the potential of leveraging LOADS to identify optimal label words with established related datasets, avoiding the need to collect samples for the target new dataset. For example, the positive-negative stance label pairs identified for Llama 3 is endorse and deny across all the stance classification datasets. However, we find that the label words selected for a specific dataset on one LLM often differ from those identified for another LLM, suggesting LOADS’ dependency on the underlying model architectures and parameters. In summary, the LOADS-selected label sets tend to be model-dependent rather than datasetdependent. This observation aligns with the mechanism of LOADS, as the neurons and their distributions are inherently tied to the specific model. We hypothesize that this may suggest a correlation between the LOADS-selected label words and the LLMs’ internal representation or understanding of the target NLP task or concept (e.g., what is stance), highlighting potential directions for future studies. 6 Conclusion We study the impact of label options in the prompt for classification in zero-shot ICL, including lexical choice, label order, and label elaborations. We observe a significant effect of the lexicons used to represent label words in the prompt, also linking to the models’ sensitivity to the label order. Through neuron activation analysis, we find that optimal label sets produce fewer outlier neurons in LLMs’ feed-forward networks. We then propose LOADS, a novel method for selecting optimal label sets using activation distribution kurtosis. Prompting with LOADS-selected label sets consistently outperforms the use of original dataset labels across different models. Our approach is post-hoc, 31720 data-efficient and requires no gradient propagation or model fine-tuning. It also demonstrates crosslingual transferability when using English instructions for non-English datasets. By showing that carefully selecting label sets based on neuron activation patterns can significantly enhance model performance without requiring additional training or labeled data, this paper has important implications for leveraging LLMs in zero-shot classification. Limitations Our experiments focused primarily on stance classification tasks. We chose this task because the label ambiguity is an identified challenge (i.e., label names could be replaced by a sufficient number of synonyms without altering their meanings and scopes in the original study), and it has sufficient datasets for empirical study. Although we have tested the generalisabity of our findings on topic classification, with more datasets released and new tasks proposed in future studies, studies could be conducted to explore whether our findings generalize to a broader range of classification tasks and domains. Also, although we examined multiple models from the Flan-T5 and Llama families, our study did not include other popular language models (such as Phi 9 or Mistral 10 ) due to computational resource limitation. Expanding the range of models would provide a more comprehensive understanding of label option’s impact across different architectures. Another limitation of our study is Englishlanguage bias. Although we have explored the cross-lingual transferability to French and Portuguese on one dataset, more extensive multilingual testing is needed to ensure the approach’s effectiveness across diverse languages and cultures. Our method, while more efficient than gradientbased approaches, still requires running inference on a subset of data to compute activation statistics. This may be challenging for resource-constrained environments or very large models. The efficiency of our method may also be challenged when the classification task contains a very large number of class categories. Furthermore, we ensure the inclusion of samples for each class. The effectiveness of our method might vary when the validation set is highly imbalanced or even lack of data for the 9https://huggingface.co/microsoft/phi-2 10https://huggingface.co/mistralai/ Mistral-7B-Instruct-v0.2 minority class. The effectiveness of different distribution metrics is out of scope but we acknowledge that it may have significant improvement for our method. Lastly, while we focused on technical performance, future work should consider potential biases introduced by label set choices and their implications for fairness and inclusivity in classification tasks. Addressing these limitations in future research will help to further validate and refine our approach to optimal label set selection for zero-shot ICL. Acknowledgments This work was partially funded by EMIF managed by the Calouste Gulbenkian Foundation 11 under the "Supporting Research into Media, Disinformation and Information Literacy Across Europe" call (ExU – project number: 291191) 12 and the UK’s innovation agency (InnovateUK) grant number 10039039 (approved under the Horizon Europe Programme as VIGILANT 13 , EU grant agreement number 101073921). Yue Li was supported by a Sheffield–China Scholarships Council PhD Scholarship. We acknowledge IT Services at The University of Sheffield for the provision of services for High Performance Computing. References Rachith Aiyappa, Shruthi Senthilmani, Jisun An, Haewoon Kwak, and Yong-Yeol Ahn. 2024. Benchmarking zero-shot stance detection with flant5-xxl: Insights from training data, prompting, and decoding strategies into its near-sota performance. arXiv preprint arXiv:2403.00236. Emily Allaway and Kathleen McKeown. 2020. ZeroShot Stance Detection: A Dataset and Model using Generalized Topic Representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8913– 8931, Online. Association for Computational Linguistics. Dimosthenis Antypas, Asahi Ushio, Francesco Barbieri, Leonardo Neves, Kiamehr Rezaee, Luis EspinosaAnke, Jiaxin Pei, and Jose Camacho-Collados. 2023. SuperTweetEval: A challenging, unified and heterogeneous benchmark for social media NLP research. 11 The sole responsibility for any content supported by the European Media and Information Fund lies with the author(s) and it may not necessarily reflect the positions of the EMIF and the Fund Partners, the Calouste Gulbenkian Foundation and the European University Institute. 12exuproject.sites.sheffield.ac.uk 13https://www.vigilantproject.eu 31721 (a) Llama3: top-k optimal label sets (b) Llama3: top-k sub-optimal label sets (c) Llama 3.1: top-k optimal label sets (d) Llama 3.1: top-k sub-optimal label sets (e) Flan-T5-xl: top-k optimal label sets (f) Flan-T5-xl: top-k sub-optimal label sets Figure 2: The maximum performance gain (positive value) and drop (negative value) on each dataset after reordering the label names for the top-k optimal and sub-optimal label sets with Llama3, Llama 3.1 and Flan-T5-xl. 31728 Dataset LOADS Original Label Prompt 1 snopes 0.5926 0.4984 ibmcs 0.8737 0.7303 perspectrum 0.8925 0.8658 scd 0.6895 0.6860 Prompt 2 snopes 0.6191 0.5336 ibmcs 0.8619 0.7523 perspectrum 0.8921 0.8619 scd 0.6836 0.6931 Table 10: Performance comparison on Llama 3 when using LOADS-selected label sets (lowest kurtosis) and using original label sets (original label) with prompt 1 or prompt 2. The higher performance is underlined. Therefore, the total FLOPs taken by LOADS would be 2N∗L∗X∗100 = 200NLX. J Perplexity Analysis As discussed in Section 2 in main paper, Gonen et al.,(2023) empirically show that zero-shot ICL performance is statistically negative correlated with the perplexity of the prompt with input. However, they did not take into account the label options in the prompt when calculating the perplexity. Therefore, we further investigate whether the perplexity is also correlated with the variance zero-shot ICL performance caused by different label names. Specifically, we use the prompt template in Section 3.2 in main paper, and calculate the perplexity of prompts with inputs and different label sets. Following Gonen et al.,(2023), for each label set, we average the perplexity over the dataset. And then we adopt spearman correlation test between the averaged perplexity scores and model performances. Due to the computational restriction, we experiment with all the binary datasets on Flan-T5-xl and Llama3-8b. Since Flan-T5 is an encoder-decoder model where perplexity has a loose definition, we treat the encoder input as an empty string when calculating perplexity. The results in Table 11 indicate that there is no statistically significant correlation between prompt perplexity and model performance if considering different label sets in the prompt. perspectrum ibmcs snopes scd Llama 3 coefficient 0.0068 0.1641 0.0394 0.1698 p value 0.9707 0.3774 0.8303 0.3608 Flan-T5 coefficient 0.0738 0.1733 -0.0500 -0.2273 p value 0.6929 0.3511 0.7892 0.2187 Table 11: Spearman correlation between model performance and prompt perplexity. P-values are all larger than 0.05, indicating no statistical significance. K Label Attention Key Similarity Analysis In this section, we explore whether the closely related observation on few-shot ICL could be directly adopted to zero-shot ICL. Specifically, we focus on the study discussed in Section 2 in main paper, where Wang et al.,(2023) suggest that the when the LLM is prompted by demonstration with examples in a few-shot ICL setting, the model is likely to confuse the label categories if their key vectors in the attention modules are similar to each other. Since this finding is easier to be tested on binary datasets, we experiment with the binary datasets on Llama 3 and Flan-T5-xl. We extract the key vectors in the attention module in each layer for each label name in the prompt. Then we calculate the cosine similarity between the vectors of two label names. Finally, we use spearman correlation test between similarity scores and model performances. As shown in Table 12, we do not observe statistically significant correlation between model performance and label key vector similarities. perspectrum ibmcs snopes scd Llama 3 coefficient -0.0181 -0.0051 -0.2791 -0.1696 p value 0.9244 0.9784 0.1283 0.3702 Flan-T5-xl coefficient -0.0595 0.0223 -0.0209 -0.0992 p value 0.7503 0.9047 0.9108 0.5953 Table 12: Spearman correlation between model performance and label’s key vector similarity. L Layer-wise Output Projections Analysis We hypothesize that LLM may jump to the output prediction at last layers when a sub-optimal label set is used in the prompt. Therefore, we extract the hidden states from each decoder layer and project them on the model vocabulary, so that we obtain the ranked position of the final predicted label’s token in each layer (Elhage et al.,2021;Geva et al., 2022). we observe that the hypothesis indeed holds in certain cases. We show an example on rumoureval dataset where we compare the averaged rank of the correctly predicted label comment/neutral in each decoder layer when Flan-T5-xl is prompt to choose from support, deny, query, comment or support, deny, query, neutral. Label set support, deny, query, comment performs worse than the set endorse, deny, query, neutral on this dataset. As shown in Figure 3, when using the relatively optimal label set endorse, deny, query, neutral, the rank of the final predicted 31729 label tends to move closer to the top at an earlier stage. Figure 3: The rank of the final correctly predicted label (comment orneutral) when Flan-t5-xl is prompted with two different label sets for rumoureval dataset. M Human Translation Details To translate the English Twitter rumoureval test set into French and Portuguese, we recruit volunteer students from translation studies in Brazilian and French universities. The students are given gift vouchers (0.6 pounds per tweet). Consent has been obtained from the students and our study has received approval from the Ethics Committee of our university. We instruct the students to translate the tweets accurately, and preserve the original meaning, context, and tone of the tweet. They are also encouraged to leave notes for their translations. The translations are finished on Google Sheets. 31730