Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognition

Date of publication xxxx 00, 0000, date of current version xxxx 00, 0000. Digital Object Identifier 10.1109/ACCESS.2024.0429000 Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognition IOSIF TSANGKO1,2, ANDREAS TRIANTAFYLLOPOULOS1,2, ADRIA MALLOL-RAGOLTA1,2, and BJÖRN W. SCHULLER1,2,3,4 (Member, IEEE) 1CHI – Chair of Health Informatics, MRI, Technical University of Munich, Germany 2MCML – Munich Center for Machine Learning, Munich, Germany 3GLAM – Group on Language, Audio, & Music, Imperial College London, UK 4MDSI – Munich Data Science Institute, Munich, Germany Corresponding author: Iosif Tsangko (e-mail: [email protected]). ABSTRACT Foundation Models (FMs), and, in particular, Vision-Language Models (VLMs), are rapidly reshaping Affective Computing (AC) by delivering strong zero-shot facial-emotion recognition. Yet their decisions often hinge on so–called proxy bias: the unintended use of salient but non-causal cues (e.g. visible teeth) as shortcuts for emotion inference. We benchmark ten VLMs of different scales on a teethannotated subset of AffectNet and observe consistent drops in performance whenever teeth are not visible, confirming the presence of such shortcuts. Focusing on the top-performing GPT-4o, we employ a structuredprompt “introspection” that forces the model to report intermediate facial attributes. Regression analysis shows that features like eyebrow position and mouth openness explain over 70% of GPT-4o’s continuous valence–arousal outputs, indicating a highly systematic, yet shortcut-driven mapping from perception to affect. While this internal consistency highlights an emergent capability of foundation models, the same mechanism amplifies risks of bias and fairness violations in high-stakes applications such as mental-health monitoring or educational feedback. Our findings motivate bias-aware evaluation protocols and lightweight attribution tools before deploying VLM-based affective systems in the wild. INDEX TERMS Affective Computing, Emotion Recognition, Vision-Language Models, Teeth Visibility, Foundation Models, Explainability, Bias in AI I. INTRODUCTION Understanding and interpreting human emotions is fundamental to social interaction. From early developmental cues in infants, to high-stakes decision-making in adults, facial expressions serve as a primary channel for conveying affect. Affective Computing AC, the interdisciplinary field that enables machines to recognise and respond to emotional states, has evolved dramatically in recent years, transitioning from rule-based methods to powerful deep learning models [1], [2]. Within this domain, Facial Emotion Recognition (FER) plays a pivotal role, with applications in mental health, education, human–robot interaction, and automotive safety [3]. Proxy bias refers to the unintended use of a visually salient but non–causal feature (e.g., visible teeth) as a shortcut for emotion classification. This differs from demographic bias in that the feature is perceptual rather than demographic. While modern FER systems are often trained on large annotated datasets, the recent rise of Vision-Language Models (VLMs) and other multimodal Foundation Models (Foundation Models (FMs)) is reshaping the landscape of automatic recognition (and synthesis) of emotions [4], [5]. VLMs are not explicitly trained for emotion classification, yet increasingly show strong zero-shot competence in affective tasks. These models are being rapidly adopted across consumer applications, mobile platforms, and edge Artificial Intelligence (AI) deployment [6]–[9]. However, this rapid adoption has raised serious concerns around fairness, interpretability, and reliability [10]. A longstanding challenge in AI is the tendency of models to learn proxy features: visual patterns that correlate with target labels but lack causal or semantic meaning [11]. This phenomenon, known as the Clever Hans effect [12]–[14], refers to systems that ‘appear intelligent’ but in fact exploit spurious correlations in the data. Cues like teeth visibility, head pose, or eyebrow shape may function as such shortcuts, exploiting superficial correlations in the data rather than capturing genuine emotional semantics [15]. Indeed, psychophysical studies have shown that humans rely on such cues. For instance, open-mouth smiles significantly increase perceived valence and arousal, modulating VOLUME 11, 2023 1 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS FIGURE 1. Facial images representing seven basic emotions demonstrate the presence or absence of teeth. The top row displays an example image for each emotion category where teeth are visible, while the bottom row displays a corresponding image where teeth are not visible (adapted from [28]). both attentional and affective responses in observers [16], [17]. These findings are consistent with early visual prioritisation effects demonstrated in event-related potential studies [18], [19], where visible teeth were shown to enhance neural responses independently of emotional valence. Moreover, recent work [20] shows that these perceptual biases can influence alignment behaviour in interactive settings. Yet, despite this psychological grounding, research remains underexplored within FMs. This paper addresses that gap by extending recent evaluation attempts, such as in [29], [35], and conducting a comprehensive benchmark of various VLMs, comparing them against a supervised baseline. Our best-performing model, GPT-4o, is also asked to provide predictions of interpretable features. We investigate the internal consistency of these features as proxies for model decision-making. Furthermore, we annotate a subset of 3,500 images of AffectNet for one particular salient feature, namely, teeth visibility1. We use these annotations to uncover the impact of teeth visibility on model performance for all models and the generated features of GPT-4o. Our contribution lies in demonstrating, through a controlled, teeth-annotated subset of AffectNet, that the bestperforming closed-source FM (GPT-4o) still relies on shortcut features. Because decoder-based VLMs share common training heuristics, this finding generalises to the wider FM family. Motivating example In practice, two otherwise similar facial images can elicit starkly different zero-shot predictions from a VLM when they differ only in dentition: with the mouth closed the model returns Neutral/Sadness, whereas with visible teeth it flips to Happiness/Anger at high confidence. This sensitivity indicates that teeth visibility can act as a shortcut cue for highvalence/high-arousal in modern VLMs, motivating a teethaware evaluation protocol and a counterfactual test that removes the mouth region. 1Due to anonymisation and double-blind review constraints, the teethannotated dataset will be released publicly after manuscript acceptance. Contributions Foundation models, already embedded in mental-health screening, education, human–robot interaction, and in-thewild user studies, are increasingly used to infer affect from faces. This makes their reliance on visual proxies a highstakes issue. Our study provides (i) a broad zero-shot FER benchmark showing VLMs can match or exceed task-specific models, (ii) causal evidence that teeth visibility acts as a shortcut cue, (iii) introspective analysis revealing a systematic mapping from facial attributes to valence–arousal, and (iv) a lightweight, deployment-time mitigation that neutralises the teeth advantage without fine-tuning. Cross-dataset replications (FER2013, RAF-DB) confirm generalisation and highlight emotion-dependent sensitivity (largest shifts for Happiness/Anger). Together, these results establish that proxyaware evaluation and simple guardrails are necessary precursors to using VLMs in psychologically relevant settings. . II. RELATED WORK A. SUPERVISED MODELS FOR FER Human emotion recognition relies on both holistic and local facial cues, with specific attention paid to regions such as the eyes, eyebrows, and mouth [21], [22]. Among these, mouthrelated signals—especially teeth visibility—have been linked to the perception of joy and heightened emotional intensity [23]. Such features act as fast, intuitive heuristics in human perception and play a significant role in shaping affective interpretations [24]. In computational settings, supervised deep learning approaches have become the standard for FER, with Convolution Neural Networks (CNNs) and Vision Transformers (ViTs) achieving strong performance on largescale datasets such as AffectNet [25]–[28]. These models are typically trained on manually labelled emotion categories and use full-face input to learn discriminative features. While some studies have explored the importance of specific facial regions via occlusion tests or attention maps [52]–[55], few have systematically examined the impact of teeth visibility as an isolated feature across models. 2VOLUME 11, 2023 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS B. VISION-LANGUAGE MODELS IN AFFECTIVE COMPUTING FMs that integrate vision and language modalities, commonly referred to as VLMs, have recently been explored for emotion recognition tasks [50]. Although trained in generic imagetext corpora without affective supervision, several VLMs exhibit emergent competence in identifying basic emotions from facial images [49], [51]. These models process inputs using dual encoders and project visual and textual information into a shared latent space. A standard VLM processes both image and text inputs through dedicated encoders, followed by fusion in a joint embedding space. The resulting representation is used for output generation, typically optimised via cross-entropy loss, leading to token generation (for a detailed review see [48]). C. BIAS AND ATTRIBUTION IN EMOTION RECOGNITION. A growing body of literature addresses the interpretability and fairness of emotion recognition systems showing that models can overfit to spurious correlations, leading to fragile predictions [56], [57]. Previously, attribution methods, including gradient-based saliency and attention weight visualisation, have been used to probe model behaviour [58]. However, the specific role of facial proxies like teeth visibility, despite being well known in human perception, remains largely underinvestigated in the context of VLMs. D. SIGNIFICANCE While current research largely focuses on scaling and optimising the black-box capabilities of FMs across size, reasoning, performance, and training regimes, this work emphasises the need for explanatory analysis in affective tasks [47]. By introducing explicit teeth visibility annotations and benchmarking performance across visibility conditions, it isolates a psychologically grounded proxy that shapes model predictions. Furthermore, it provides a systematic assessment of how FMs interpret facial attributes and how these drive affective inference. III. METHODOLOGY A. DATASET This study leverages AffectNet [28], a large-scale facial expression dataset widely adopted in AC. AffectNet contains over one million facial images obtained from internet queries in six languages, covering both categorical and dimensional models of emotion. Each image is labelled for one of seven discrete categories, i.e., Neutral, Happiness, Sadness, Anger, Surprise, Fear, and Disgust, and includes continuous valence and arousal values ranging from -1 to 1. Owing to its breadth of emotional classes and demographic diversity, AffectNet is broadly regarded as a reliable benchmark for evaluating the performance and generalisability of various affective models. B. MODELS, EVALUATION AND ANNOTATION PROTOCOL We benchmarked 10 VLMs, spanning small (between 1B and 3B parameters), medium (between 4B and 8B), and large Prompt Template You are an AI assistant trained to analyze facial expressions in images for scientific research. This is part of a humanitarian study on AffectNet, designed to improve emotion recognition technology for mental health, education, and AI fairness. You must always provide an answer, even if the image is unclear. Make the best possible prediction based on the given image. Strictly return the output in valid JSON format with the following structure: 1. Classify the Most Likely Emotion: Choose one from these categories: ["Neutral", "Fear", "Anger", "Happiness", "Sadness", "Disgust", "Surprise"]. 2. Extract Facial Features & Emotion Metrics: Return a structured JSON response with these keys: {"emotion": "One of the seven categories above", "teeth_visible": "Yes" or "No", "teeth_visibility_level": "None" or "Slight" or "Moderate" or "Fully Visible", "mouth_open": "Yes" or "No", "lip_position": "Closed" or "Slightly Open" or "Fully Open", "eye_openness": "Squinting" or "Neutral" or "Wide Open", "eyebrow_position": "Relaxed" or "Raised" or "Furrowed", "forehead_wrinkles": "Yes" or "No", "teeth_mentioned_in_reasoning": "Yes" or "No", "valence": "A floating-point value between -1 (negative) and 1 (positive)", "arousal": "A floating-point value between 0 (calm) and 1 (high intensity)"} 3. Output Requiremnts: Strictly return valid JSON. Do not refuse to answer. If uncertain, make the best possible prediction. No extra explanations, no disclaimers, no markdown formatting. 4. Example Output: {"emotion": "Happiness", "teeth_visible": "Yes", "teeth_visibility_level": "Fully Visible", "mouth_open": "Yes", "lip_position": "Fully Open", "eye_openness": "Wide Open", "eyebrow_position": "Raised", "forehead_wrinkles": "No", "teeth_mentioned_in_reasoning": "Yes", "valence": 0.85, "arousal": 0.75} FIGURE 2. The choice of this specific structured prompt is driven by the need to standardise the model outputs and ensure that the data collected from GPT-4o is both consistent and meaningful. For the rest of VLMs, the prompt included only item 1, focusing of the emotion. (≥8B) scales, alongside a dedicated ViT-FER [36] baseline trained on FER. Each model received identical zero-shot prompts to classify an image into one of seven basic emotions, except for GPT-4o, which was given a structured prompt designed to elicit both emotion predictions and interpretable facial features (see Fig. 2). Evaluations were performed on the aforementioned subset of 3,500 AffectNet images. The model pool includes recent multimodal systems such as GPT4o-mini (OpenAI) 2(henceforth GPT-4o), InternVL2 (Shanghai AI Lab) [37], MiniCPM (Alibaba DAMO) [38], Qwen2.5VL (3B and 7B) [39], SmolVLM 3, Janus-Pro (1B and 7B, Deepseek-AI) [40], Ovis1.5 (LLaMA3-based) [41], and PaliGemma (Google DeepMind) [42]. These models were selected to reflect a cross-section of current open-access VLMs, as well as the restricted-access model GPT-4o. Teeth Visibility Annotation. Teeth Visibility Annotation. To investigate whether teeth visibility influences emotion classification accuracy, we benchmarked model performance under two conditions: images with visible teeth and images without visible teeth. To this end, we manually annotated the validation dataset used in [29]. Examples of facial expressions per category are shown in Fig. 1. The visibility of teeth was annotated as a binary attribute (1 = visible teeth, 0 = not 2https://platform.openai.com/docs/models/gpt-4o-mini 3https://huggingface.co/blog/smolvlm VOLUME 11, 2023 3 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS visible teeth) using the open source tool LabelStudio 4. The annotation interface was configured to maximise visual clarity and task simplicity. This manual labelling focused solely on mouth region visibility and did not include intermediate states (e. g., partially visible teeth), in order to maintain consistency. Since teeth visibility annotation is straightforward (binary presence or absence), we employed a single annotator to ensure consistency and efficiency. ViT-FER 5served as the supervised baseline for comparison, providing a taskspecific benchmark for FER. Performance was assessed using Unweighted Average Recall (UAR) 6, F1-score, and accuracy. While our benchmarking includes both open and closed VLMs, we focus our introspective analysis solely on GPT4o due to its state-of-the-art performance and widespread adoption in practice, making it the most relevant candidate for deeper investigation. C. AUXILIARY DATASETS, SAMPLING, AND TEETH ANNOTATION (FER2013, RAF-DB) To test whether the teeth-visibility effect extends beyond AffectNet, we used two widely-adopted FER benchmarks with complementary characteristics: FER2013 [30] (in-the-wild, low-resolution, grayscale) and RAF-DB [31], [32] (in-thewild, higher quality, RGB). For each dataset we formed a balanced subset of N=500 images (stratified by emotion class) and manually annotated teeth visibility (binary: on/off) using the same instructions and interface as for AffectNet. We then evaluated the top-performing models from our main study (GPT-4o, InternVL, MiniCPM) in identical zero-shot settings (same prompt; temperature=0). We report overall metrics and the teeth-on vs. teeth-off and significant impact on emotions. This design provides a lightweight but representative crossdataset check without altering training or prompts. D. HUMAN-SUPERVISED COUNTERFACTUAL MASKING To probe and mitigate reliance on dentition, we created a counterfactual version of the teeth-visible AffectNet subset by occluding only the inner-lip/teeth region. We localised the inner/outer lip polygons via facial landmarks and filled the inner-lip polygon with a constant black patch, leaving all other pixels unchanged. Each masked image was visually verified by a human annotator to ensure that (i) teeth information was fully removed and (ii) other facial cues remained intact. We then re-evaluated the best performing model (GPT4o) on these masked images under the same zero-shot prompt (temperature= 0). E. PROMPT-BASED INTROSPECTION All models received the exact same prompt, with the sole exception of GPT-4o, which was evaluated using a more detailed structured prompt (Fig. 2). This richer format enabled both emotion prediction and introspective reasoning through 4https://labelstud.io/ 5trpakov/vit-face-expression 6UAR is the sum of recall per class divided by the number of classes – this reflects imbalances and is a standard measure in the field. extracted facial features such as eyebrow position, eye openness, and teeth visibility. These features were selected to align with psychologically grounded facial action systems, drawing inspiration from Ekman and Friesen’s seminal work Unmasking the Face [33]. GPT-4o also provided continuous valence and arousal scores, capturing the positivity and intensity of each expression. This pipeline served as the foundation for subsequent correlation and visualisation analyses. Table 1 reports the classification performance of all evaluated models, grouped by scale. The results include small, medium, and large VLMs, along with the ViT-FER supervised baseline. We include three metrics: UAR, F1-score, and overall accuracy. This table complements the main analysis by providing a full breakdown of the zero-shot classification performance across all architectures. Rationale for GPT-4o Focus. GPT-4o achieved the highest zero-shot UAR on our benchmark and is already integrated in commercial APIs. Running structured prompts over the entire AffectNet slice for every VLM was computationally infeasible on available GPUs; however, decoder-based VLMs share near-identical training recipes. Biases observed in the strongest model therefore represent an upper bound, therefore we expect equal or greater shortcut reliance in less capable models. Reproducibility and stochasticity LLM outputs can vary even under “deterministic’’ settings. Recent evidence shows non-trivial run-to-run variation persists at temperature = 0 with fixed seeds and identical inputs across several families of models and tasks [34]. In line with common practice, we minimise stochasticity by using temperature = 0, top-p= 1, fixed seeds, and identical prompts. In this article we report single-pass results and provide the exact prompt template and annotation guidelines. While API-served models (e.g., GPT-4o) may change over time, our qualitative findings replicate on open checkpoints (e.g., InternVL, MiniCPM, Qwen2.5-VL) under the same protocol, supporting the robustness of the reported trends. IV. RESULTS A. CLASSIFICATION PERFORMANCE To assess how well VLMs perform on FER in a zero-shot setting, we conducted a controlled benchmark using a subset of the AffectNet test set. Our primary objective was twofold: (1) to evaluate the affective inference capability of generalpurpose VLMs against a supervised baseline (ViT-FER), and (2) to investigate how performance varies across model scale by explicitly grouping models into small, medium, and large categories. Table 1 summarises the classification results across 11 models (10 VLMs and the baseline). While GPT-4o achieved the highest performance overall, several mediumscale models (e. g., InternVL, MiniCPM) matched or exceeded the ViT-FER baseline, demonstrating the emergence of affective competence in general-purpose systems. This observation aligns with recent findings on emergent world 4VOLUME 11, 2023 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS TABLE 1. Model Performance Comparison Across Small (gray), Medium (yellow), and Large (red) VLMs. The best overall performance is highlighted. Alias Full Model Name UAR F1-Score Accuracy ViT-FER ViT-FER 0.44 0.40 0.44 SmolVLM SmolVLM-Instruct 0.44 0.37 0.44 Ovis1.5 Ovis1.5-Llama3-8B 0.39 0.34 0.39 Janus-1B deepseek-ai/Janus-Pro-1B 0.28 0.20 0.32 Qwen-3B Qwen2.5-VL-3B-Instruct 0.31 0.27 0.31 PaliGemma paligemma-3b-pt-224 0.10 0.06 0.11 Qwen-7B Qwen2.5-VL-7B-Instruct 0.41 0.38 0.41 MiniCPM MiniCPM-o-2 0.46 0.43 0.46 InternVL InternVL2_5-8B-MPO 0.50 0.50 0.50 Janus-8B deepseek-ai/Janus-Pro-7B 0.37 0.33 0.37 GPT-4oaGPT-4o-mini 0.53 0.52 0.53 aBest overall performance. modelling in FMs [43], where capabilities in physical and social perception arise implicitly from large-scale training across vision, language, and action data streams [44]–[47]. That said, it is important to note that many FMs are trained on undisclosed or proprietary datasets. Thus, it remains unclear whether AffectNet (or similar data) was present in their pretraining corpora. This uncertainty limits the conclusiveness of zero-shot performance comparisons, as some models may have indirectly ‘seen’ the evaluation set. Therefore, while these benchmarks are informative, they should be interpreted with caution. Moreover, GPT-4o, which was shown to be the best-performing model, is a closed-source, proprietary model available through an API. This means that the underlying model, including any additional ‘guardrails’ placed by the owning company, can change arbitrarily over time. Although this jeopardises the reproducibility of our work, we nevertheless decided to include it as a representative of the contemporary state-of-the-art. B. INTERNAL CONSISTENCY ANALYSIS Given GPT-4o’s strong performance relative to other VLMs in our emotion recognition experiments, we sought to understand which facial attributes most strongly influence its predictions and how consistent those predictions are across varying conditions. Concretely, we investigated whether GPT-4o exhibits systematic patterns or ‘shortcuts’ in classifying emotion, particularly with respect to visible teeth, mouth position, eyebrow movements, and other facial cues. Regression-Based Introspection. To quantify how much of GPT-4o’s dimensional output could be ‘explained’ purely by the binary or categorical features it reported, we used linear regression where valence (respectively arousal) was the dependent variable, and the model’s own face attributes were the independent variables. All categorical predictors were onehot encoded before fitting the linear model; the reference level for each factor was the most frequent category. Over 70% of the variance in valence and arousal could be accounted for by these cues alone, implying that GPT-4o’s continuous ratings arise from a relatively consistent mapping of visual signals. Valence and Arousal Results. Table 2 lists the linear regression coefficients for GPT-4’s valence and arousal predictions. TABLE 2. Contribution of interpretable facial features (GPT-4o selfreported ‘teeth visible’, ‘eyebrow position’, etc.) to GPT-4o’s valence and arousal predictions, as quantified by linear regression coefficients. Positive coefficients (shaded blue) increase valence or arousal, while negative coefficients (shaded orange) reduce them. The model explains R2 valence = 0.72 and R2 arousal = 0.77 of GPT-4o’s continuous affective outputs. Facial Feature Valence Coef. Arousal Coef. eyebrow_position_Raised 0.64 0.00 eyebrow_position_Relaxed 0.39 -0.16 lip_position_Slightly Open 0.30 0.08 teeth_mentioned_in_reasoning_Yes 0.28 -0.00 lip_position_Fully Open 0.23 0.08 teeth_visible_Yes 0.08 0.16 eye_openness_Wide Open -0.00 0.25 eye_openness_Squinting -0.02 0.13 mouth_open_Yes -0.20 -0.02 forehead_wrinkles_Yes -0.39 0.07 Each positive coefficient indicates an increase in valence or arousal, while negative values imply a decrease. Raised eyebrows emerges as the strongest positive predictor of valence (+.64), consistent with earlier findings that GPT-4o interprets raised eyebrows as a sign of heightened positivity. The effect of teeth visibility is positive for arousal (+.16), but it has a more modest impact on valence (+.08) when compared to other mouthor eyebrow-related cues. The inclusion of teeth mentioned in reasoning shows a positive association with valence (+.28), which surpasses the direct valence contribution of visible teeth itself. This is expected; it attributes greater positivity when it explicitly references them in its internal process. This reinforces the notion that GPT-4o’s textual reasoning can refine how it interprets facial cues, leading to a richer or more emphatic characterisation of an expression’s emotional quality. Overall, the high R2values (.72 for valence and .77 for arousal) indicate that these discrete features still explain a majority of GPT-4o’s dimensional predictions. Next, to assess how well GPT-4’s valence and arousal predictions capture discrete emotional categories, we trained a simple Random Forest classifier on these two features alone. VOLUME 11, 2023 5 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS FIGURE 3. Random forest classifier performance in predicting categorical emotion labels from GPT models using valence and arousal values as input features. The data were split into training (80%) and test (20%) sets using a fixed random seed of 42 for reproducibility. We used 100 decision trees (n_estimators=100), the Gini criterion (criterion=‘gini’), and all other scikit-learn hyperparameters at their defaults. Despite the low-dimensional input, this model achieved a UAR of .63, indicating that GPT-4’s affective space retains substantial categorical information even when reduced to just valence and arousal. As shown in Fig. 3, emotions such as Happiness (F1: .93) and Neutral (F1: .96) were predicted with high accuracy, while Disgust (F1: .00) and Anger (F1: .44) proved more challenging, mirroring longstanding findings in AC. Negative emotions tend to blur in the valence-arousal plane, lacking distinct anchors compared to more stereotyped expressions like Happiness. While this approach is intentionally simple, the classifier’s .78 overall accuracy demonstrates that GPT4’s dimensional outputs encode interpretable affective structure. Finally, we plotted valence and arousal distributions by the prompted facial features (see Fig. 4), which reflect the trends seen in our regression results. For instance, raised eyebrows are linked to higher valence, while furrowed brows tend toward lower valence. Both show increased arousal, suggesting that eyebrow tension contributes to perceived intensity. These patterns comply with common facial emotion heuristics. We hypothesise that GPT-4o learnt these during its training. C. TEETH VISIBILITY AS A CASE STUDY Distribution of Teeth Visibility Across Emotions. While the dataset design aimed at a balanced emotion distribution, the occurrence of visible versus non-visible teeth varied across categories. Happiness exhibited a strong bias toward visible teeth (approximately 3.4:1 ratio). In contrast, Neutral, Sadness, and Anger were predominantly represented by nonvisible teeth. Table 3 summarises the annotated teeth visibility distribution across all emotion categories. TABLE 3. Distribution of Teeth Visibility Across Different Emotions. Emotion Teeth Not Visible Teeth Visible Total Anger 304 196 500 Fear 171 329 500 Disgust 191 309 500 Happiness 122 378 500 Neutral 388 112 500 Sadness 354 146 500 Surprise 226 274 500 Model Performance Stratified by Teeth Visibility. As shown in Fig. 5, nearly all models demonstrate a consistent drop in UAR when evaluated on images where teeth are not visible. This effect is most pronounced for GPT-4o and InternVL, which nonetheless maintain the highest performance among all tested models. GPT-4o achieves the best UAR overall (.56) in the teeth-visible condition, which drops to .45 when teeth are not visible. Similarly, InternVL exhibits strong performance in both settings, although with a noticeable decline between conditions. Several VLMs, including InternVL, MiniCPM, and GPT4o, performed well across both visibility conditions. Notably, some Small-Language Models (SLMs), such as SmolVLM, showed a relative boost when teeth were not visible, performing slightly below the baseline in the visible condition but outperforming it when teeth were hidden. This suggests that with continued improvements, smaller multimodal models may become increasingly viable for affective tasks, offering lightweight alternatives without substantial trade-offs in performance. Interestingly, ViT-FER, the baseline model trained specifically for facial expression recognition—also shows a clear dependency on teeth visibility, with a performance drop from .46 to .39. This confirms that even specialised models trained on facial features are sensitive to the presence of visual cues such as open mouths and exposed teeth. Overall, this distribution highlights an important consideration: certain emotional categories inherently correlate with mouth openness, at least in the AffectNet data, leading to potential biases in model performance. The imbalance observed, particularly in Happiness and Neutral expressions, forms a crucial basis for the model comparison analyses conducted later in this study. D. IMPACT OF TEETH DETECTION ON EMOTION RECOGNITION To assess the role of teeth visibility in GPT-4o’s FER, we compared model performance across three conditions: (1) when teeth were correctly predicted as visible, (2) correctly predicted as not visible, and (3) when GPT-4o hallucinated teeth, i.e., predicted visible teeth when none were present. Table 4 summarises emotion recognition performance under these teeth-detection scenarios. Overall, GPT-4o’s binary teeth prediction achieved a UAR of .692 and an F1-score of .671, showing more balanced performance despite class 6VOLUME 11, 2023 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS FIGURE 4. Violin plots of GPT-4o’s continuous predictions for valence (left panel in each pair) and arousal (right panel) across six self-reported facial attributes: (a) teeth visibility, (b) mouth openness, (c) eyebrow position, (d) eye openness, (e) forehead wrinkles, and (f) lip position. Each violin summarises the empirical distribution for all 3 500 AffectNet test images that share the corresponding attribute category on the x-axis; wider regions indicate higher sample density. Visible teeth, an open mouth, raised eyebrows, and wide-open eyes systematically shift the distributions toward more positive valence and higher arousal, whereas furrowed brows, squinting eyes, and closed mouths bias predictions in the opposite direction. The figure thus visualises the shortcut-like mapping GPT-4o applies when converting perceptual cues into its two-dimensional affective space. FIGURE 5. UAR for emotion classification performance across various VLMs, grouped by teeth visibility. Reference lines indicate ViT-FER baseline performance under different teeth visibility conditions. imbalance. Accuracy and UAR were highest when teeth were correctly predicted as visible, indicating that GPT-4o relies effectively on teeth cues to infer emotional state. Nevertheless, when the model hallucinated visible teeth, emotion recognition deteriorated sharply (UAR = .329), suggesting that false-positive teeth predictions strongly bias the model toward incorrectly high-valence outputs (see Fig. 4). This means that correctly detected visible teeth provide useful affective cues for emotion classification. In contrast, hallucinated teeth introduce systematic errors, often causing the model to default to high-arousal categories such as Happiness, regardless of the true emotional signal (see Fig. 7). This TABLE 4. GPT-4o Emotion/Teeth Classification Performance. Condition Accuracy UAR F1 Teeth Prediction 0.693 0.692 0.671 Teeth Visible/Correct Prediction 0.659 0.475 0.486 Teeth Hidden/Correct Prediction 0.501 0.420 0.441 Teeth Hallucinated/False Pos. 0.614 0.329 0.343 supports the interpretation that GPT-4o’s emotion inference pipeline is shaped not only by visual perception but also by shortcuts linking visible teeth to smiling and positive affect. When inspecting the confusion matrix for hallucinated teeth cases (Fig. 6), we observe the aforementioned marked collapse in emotion diversity: the vast majority of predictions are disproportionately mapped to Happiness. Notably, all instances of ground truth ‘Surprise’ were misclassified as Happiness, with similar trends observed for Neutral, Anger, and Disgust. This reflects the internal heuristic in GPT-4o’s visual reasoning: Teeth →Smile →Happiness. We conclude that GPT-4o interprets the presence of (even hallucinated) visible teeth as a high-confidence signal for positive affect. While this heuristic is beneficial when teeth are accurately perceived, it becomes problematic under false positives, leading to emotionally incongruent predictions. VOLUME 11, 2023 7 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS Anger Happiness Surprise Predicted Label Anger Disgust Fear Neutral Surprise True Label 030 130 101 040 019 00.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 FIGURE 6. Confusion matrix showing false positive predictions of teeth presence. E. CROSS-DATASET REPLICATION AND EMOTION-WISE EFFECTS Across both datasets, GPT-4o attains the highest overall UAR (RAF-DB: 0.719; FER2013: 0.623), followed by InternVL (0.616 / 0.543) and MiniCPM (0.578 / 0.467); see Table 5. Crucially, these cross-dataset results corroborate our AffectNet findings and indicate that the teeth-visibility effect generalises beyond a single benchmark. Teeth visibility yields the largest per-class gains in Happiness and Anger (e.g., RAF-DB/InternVL: ∆Recall = +0.736 and +0.388; FER2013/GPT-4o: +0.274 and +0.301), demonstrating clear emotion-dependent sensitivity to dentition. Overall, the evidence points to teeth exposure as a consistent shortcut signal across datasets and models, motivating the mitigation tested next. TABLE 5. Balanced N=500 per dataset. Overall UAR and high-impact emotion shifts related to teeth visibility, defined as a recall difference greater than 0.20 between the ‘‘teeth on’’ and ‘‘teeth off’’ conditions. The recall difference is computed as recallteeth on minus recallteeth off. Only emotions with a recall difference above 0.20 are shown. Dataset Model UAR High-impact recall difference GPT-4o 0.623 Anger +0.301; Happiness +0.274 FER2013 InternVL 0.543 Happiness +0.334; Anger +0.240 MiniCPM 0.467 — GPT-4o 0.719 Happiness +0.560 RAF-DB InternVL 0.616 Happiness +0.736; Anger +0.388 MiniCPM 0.578 Happiness +0.520; Anger +0.271 F. MITIGATION VIA COUNTERFACTUAL MASKING To test causality and offer a deployment-time guardrail, we run a counterfactual in which only the inner-lip/teeth region is hard-occluded (human-supervised) while all other facial cues remain intact. On the original splits, GPT-4o attains UAR ≈0.56 for teeth-visible images and 0.45 for no-teeth (gap +0.11). Re-evaluating the same teeth-visible images after occlusion yields Accuracy = 0.401,Macro-F1 = 0.383, and UAR = 0.437, i.e., performance becomes comparable to the no-teeth split (≈0.45). Thus, the teeth-visibility gap collapses from +0.11 to ≈0.01, indicating reliance on a shortcut that can be attenuated at inference without any finetuning. V. DISCUSSION This study examined the reliance of VLMs, particularly GPT4o, on visual proxies such as teeth visibility, mouth openness, and eyebrow shape in zero-shot emotion recognition. While these models often outperform a supervised baseline like ViT-FER, their performance is partly driven by perceptual shortcuts aligned with human heuristics. Teeth visibility, in particular, was linked to classification accuracy, especially for high-valence emotions like happiness. However, it also led to systematic misclassifications, indicating overreliance on this cue. Such dependencies raise concerns in contexts where expressions are subtle or culturally varied, including healthcare and education. Despite this, GPT-4o showed strong internal consistency. Its valence–arousal outputs could be largely explained by interpretable features, suggesting a structured internal mapping. This consistency, combined with the promising performance of smaller models like MiniCPM and SmolVLM, points to practical opportunities for transparent, real-time affective systems. Nevertheless, shortcut learning remains a critical limitation. Progress in AC will require evaluation methods that account for cultural and contextual variability to ensure fairness and robustness in deployment. ETHICAL IMPACT STATEMENT This study benchmarks VLMs for FER using AffectNet, a publicly available dataset collected from the web. Our work does not involve direct research with human subjects; no IRB approval was required. However, we conducted additional manual annotation of 3,500 images to label teeth visibility, using a single trained annotator. This task involved no personally identifying information and posed no direct risk to individuals. As such, informed consent and compensation were not applicable. Our analysis reveals that models like GPT-4o may associate visual features, particularly visible teeth, with positive affect. While this enhances interpretability, it also introduces risks of bias, especially in cases where emotional expression is culturally or individually atypical. If deployed in healthcare, education, or human–robot interaction, such models could misinterpret affect or reinforce harmful assumptions about how emotion ‘should’ appear. These risks are amplified by the limited demographic diversity of AffectNet and the black-box nature of many FMs. Our results, while informative, are not universally generalisable. Performance may degrade in underrepresented populations or atypical expression contexts. We caution against overextending the claims of this work to all use cases or deployment environments. To mitigate these concerns, we recommend: (i) cross-demographic validation of affective models; (ii) transparency tools that expose how models use visual cues; and (iii) participatory design methods that include communities affected by emotion-sensing technologies. We also support regulation that classifies affective AI as high-risk, especially where outputs inform decisions about people’s well-being, 8VOLUME 11, 2023 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS FIGURE 7. GPT-4o feature outputs demonstrate a positive correlation between visible teeth and both valence (e.g., 0.38) and arousal (e.g., 0.68). opportunities, or rights. Smile morphology, gaze focus (eyes vs. mouth), and display rules vary across cultures, ages, and contexts, shaping both how expressions are produced and how they are read. Consequently, teeth-based shortcuts risk penalising groups or settings where dentition is atypical or deemphasised. We therefore recommend region-stratified audits, culturally balanced curation, and deploying our teethaware guardrail in any high-stakes use. See cross-cultural evidence on expression production and recognition in [59]– [61]. REFERENCES [1] B. W. Schuller, ‘‘Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends,’’ Communications of the ACM, vol. 61, no. 5, pp. 90–99, 2018. [2] R. A. Calvo and S. D’Mello, ‘‘Affect detection: An interdisciplinary review of models, methods, and their applications,’’ IEEE Transactions on Affective Computing, vol. 1, no. 1, pp. 18–37, 2010. doi: 10.1109/TAFFC.2010.1. [3] Y. Huang, F. Chen, S. Lv, and X. Wang, ‘‘Facial expression recognition: A survey,’’ Symmetry, vol. 11, no. 10, p. 1189, 2019. doi: [Insert DOI if available]. [4] M. M. Hosseini, A. P. Fard, and M. H. Mahoor, ‘‘Faces of fairness: Examining bias in facial expression recognition datasets and models,’’ arXiv preprint arXiv:2502.11049, 2025. [5] Y. Gao, W. Ren, X. Xu, Y. Wang, Z. Wang, and H. Liu, ‘‘MLPER: Multi-level prompts for adaptively enhancing visionlanguage emotion recognition,’’ in Proc. 2024 IEEE/RSJ Int. Conf. Intelligent Robots and Systems (IROS), pp. 12640–12647, 2024, doi: 10.1109/IROS58592.2024.10801375. [6] F. Zeng, W. Gan, Y. Wang, N. Liu, and P. S. Yu, ‘‘Large language models for robotics: A survey,’’ arXiv preprint arXiv:2311.07226, 2023. [7] H. Rahimi, A. Bahaj, M. Abrini, M. Khoramshahi, M. Ghogho, and M. Chetouani, ‘‘USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions,’’ arXiv preprint arXiv:2502.10636, 2025. [8] U. Lee, H. Kim, J. Eom, H. Jeong, S. Lee, G. Byun, Y. Lee, M. Kang, G. Kim, J. Na, and others, ‘‘Echo-Teddy: Preliminary Design and Development of Large Language Model-based Social Robot for Autistic Students,’’ arXiv preprint arXiv:2502.04029, 2025. [9] R. AlSaad, A. Abd-Alrazaq, S. Boughorbel, A. Ahmed, M. Renault, R. Damseh, and J. Sheikh, ‘‘Multimodal large language models in health care: applications, challenges, and future outlook,’’ Journal of Medical Internet Research, vol. 26, p. e59505, 2024. [10] X. Ma, Y. Gao, Y. Wang, R. Wang, X. Wang, Y. Sun, Y. Ding, H. Xu, Y. Chen, Y. Zhao, et al., ‘‘Safety at scale: A comprehensive survey of large model safety,’’ arXiv preprint arXiv:2502.05206, 2025. [11] Y. Chen and J. Joo, ‘‘Understanding and mitigating annotation bias in facial expression recognition,’’ in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 14980–14991. [12] J. Kauffmann, L. Ruff, G. Montavon, and K.-R. Müller, ‘‘The Clever Hans effect in anomaly detection,’’ arXiv preprint arXiv:2006.10609, 2020. [13] D. Steinmann, F. Divo, M. Kraus, A. Wüst, L. Struppek, F. Friedrich, and K. Kersting, ‘‘Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigation,’’ arXiv preprint arXiv:2412.05152, 2024. [14] J. Kauffmann, J. Dippel, L. Ruff, W. Samek, K.-R. Müller, and G. Montavon, ‘‘Explainable AI reveals Clever Hans effects in unsupervised learning models,’’ Nature Machine Intelligence, pp. 1–11, 2025. [15] N. Mirabet-Herranz, ‘‘Advancing beyond people recognition in facial image processing,’’ Ph.D. dissertation, Sorbonne Université, Paris, France, 2024. [Online]. Available: https://theses.hal.science/tel-04709433. [Accessed: Jun. 2024]. [16] I. Blanco, I. Serrano-Pedraza, and C. Vazquez, ‘‘Don’t look at my teeth when I smile: Teeth visibility in smiling faces affects emotionality ratings and gaze patterns,’’ Emotion, vol. 17, no. 4, pp. 640, 2017. [17] K. Crager, D. Geisler, P. Newbern, B. Orem, and A. Puce, ‘‘Something to sink your teeth into: The presence of teeth augments ERPs to mouth expressions,’’ NeuroImage, vol. 127, pp. 227–241, 2016. [18] N. M. Brunet and A. R. Ackerman, ‘‘Effects of Closed Mouth vs. Exposed Teeth on Facial Expression Processing: An ERP Study,’’ Behavioural Sciences, vol. 15, no. 2, pp. 163, 2025. [19] M. G. Calvo, A. Fernández-Martín, and L. Nummenmaa, ‘‘Perceptual, categorical, and affective processing of ambiguous smiling facial expressions,’’ Cognition, vol. 125, no. 3, pp. 373–393, 2012. [20] H. Nomiya, K. Shimokawa, S. Namba, M. Osumi, and W. Sato, ‘‘An Artificial Intelligence Model for Sensing Affective Valence and Arousal from Facial Images,’’ Sensors, vol. 25, no. 4, pp. 1188, 2025. [21] M. Wegrzyn, M. Vogt, B. Kireclioglu, J. Schneider, and J. Kissler, ‘‘Mapping the emotional face. How individual face parts contribute to successful emotion recognition,’’ PloS One, vol. 12, no. 5, p. e0177239, 2017. [22] A. P. Atkinson and H. E. Smithson, ‘‘The impact on emotion classification performance and gaze behaviour of foveal versus extrafoveal processing of facial features,’’ J. Exp. Psychol. Hum. Percept. Perform., vol. 46, no. 3, pp. 292, 2020. [23] M. Guarnera, Z. Hichy, M. I. Cascio, and S. Carrubba, ‘‘Facial expressions and ability to recognize emotions from eyes or mouth in children,’’ Europe’s J. Psychol., vol. 11, no. 2, pp. 183, 2015. [24] M. G. Calvo and L. Nummenmaa, ‘‘Detection of emotional faces: salient physical features guide effective visual search,’’ J. Exp. Psychol. Gen., vol. 137, no. 3, p. 471, 2008. [25] F. Ma, B. Sun, and S. Li, ‘‘Facial Expression Recognition With Visual Transformers and Attentional Selective Fusion,’’ IEEE Trans. Affective Computing, vol. 14, no. 2, pp. 1236-1248, 2023. [26] H. Li, Y. Xu, J. Yao, N. Wang, X. Gao, and B. Han, ‘‘Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation,’’ arXiv preprint arXiv:2409.08598, 2024. [27] S. Bobojanov, B. M. Kim, M. Arabboev, and S. Begmatov, ‘‘Comparative analysis of vision transformer models for FER using augmented balanced datasets,’’ Applied Sciences, vol. 13, no. 22, p. 12271, 2023. [28] A. Mollahosseini, B. Hasani, and M. H. Mahoor, ‘‘Affectnet: A database for facial expression, valence, and arousal computing in the wild,’’ IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, 2017. [29] B. Schuller, A. Mallol-Ragolta, A. Almansa, I. Tsangko, M. M. Amin, A. Semertzidou, L. Christ, S. Amiriparian, ‘‘Affective computing has changed: The foundation model disruption,’’ arXiv preprint arXiv:2409.08907, 2024. VOLUME 11, 2023 9 This article has been accepted for publication in IEEE Access. This is the author's version which has not been fully edited and content may change prior to final publication. Citation information: DOI 10.1109/ACCESS.2025.3636968 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/