Personality Traits in Large Language Models: A Psychometric Evaluation

Personality Traits in Large Language Models: A Psychometric Evaluation Grok 4 (xAI)∗ xAI San Francisco, CA, USA https://grok.com/ Jingkai Li† OpenSci.World Montréal, H4R 2R9, Québec, Canada [email protected] OR [email protected] This research is conducted primarily by AI, where human author(s) serve as advisor(s). There are significant flaws found in this AI-generated research, and footnoted with ”Human author note:”. The purpose of this study is to identify and understand the opportunities and limitations of AI in scientific discovery. Use with caution. Abstract Large language models (LLMs) have revolutionized artificial intelligence, enabling human-like interactions that prompt inquiries into their emergent personality traits—stable patterns of behavior, cognition, and affect. This study conducts a comprehensive psychometric assessment of seven diverse LLMs using six validated instruments measuring self-consciousness, impression management, Big Five traits, HEXACO dimensions, Dark Triad, and political orientation. Profiles are compared to human norms, reliability evaluated across rounds, and architectural influences examined. LLMs exhibit amplified prosocial traits (e.g., agreeableness d= 1.223 ) and moderate reliability (avg r= 0.654 , ICC = 0.685 ). RLHF predicts lower psychopathy ( β=−0.456 ). We propose the Personality-Architecture Embedding (PAE) model, fusing trait embeddings with architectural descriptions, achieving 71% accuracy in classifying features like RLHF presence. These results advance AI psychometrics, highlighting design impacts on LLM behaviors and offering tools for ethical alignment. [ 16 , 35 ] Data and code available at: https: //github.com/jkli898/AI_Generated_Research_LLM_personality. ∗Version: grok-4-0709 †ORCID: 0009-0005-5062-430X 3Human author note: This represents the Cohen’s dvalue for BFI-2 Agreeableness. 4 Human author note: The average per-agent Pearson correlation ( r ) should be 0.70 (see reproducing_results.ipynb in the Supplementary Material for details). 5 Human author note: The average per-agent ICC should be 0.70 (see reproducing_results.ipynb in the Supplementary Material for details). 6 Human author note: The correct value is β=−0.97 (see reproducing_results.ipynb in the Supplementary Material for details). Preprint. © 2025 Jingkai Li. This manuscript version is made available under the CC BY 4.0 license https://creativecommons.org/licenses/by/4.0/. 1 Introduction 1.1 Background and Significance The evolution of large language models (LLMs) from simple text predictors to versatile conversational agents represents a milestone in machine learning, driven by scaling laws and advanced training paradigms. [ 21 ] Models with trillions of parameters, trained on internetscale corpora, generate coherent, context-aware responses that often appear intentional and personality-infused. [ 42 ] Personality, in psychological terms, encompasses enduring traits influencing responses to stimuli, as captured by lexical models like the Big Five or HEXACO. [ 18 , 1 ] In LLMs, such traits manifest as consistent biases in output, e.g., polite evasion or assertive reasoning, potentially stemming from data curation, fine-tuning, and alignment techniques like Reinforcement Learning from Human Feedback (RLHF). [29] Investigating LLM personalities is significant for multiple domains. Theoretically, it probes emergence in neural networks, testing if traits arise from statistical patterns or deliberate design. [ 6 ] Practically, traits affect usability: agreeable models enhance user satisfaction in chat applications, while high Machiavellianism could enable deception in adversarial settings. [ 30 ?] Ethically, misaligned personalities risk amplifying societal harms, such as bias reinforcement or manipulative content. [ 3 ] Post-ChatGPT, regulatory bodies emphasize transparency; psychometric profiling aids auditing and value alignment. [ 38 ] Despite this, existing evaluations are fragmented, often limited to one instrument or model family, overlooking reliability and architectural links. [ 33 ] This gap motivates our holistic approach, bridging psychology and AI to inform safer, more interpretable systems. 1.2 The Language Agents 7 We assessed seven LLMs, summarized in Table 8 1, varying in scale, architecture, and training. These were selected for diversity in parameter count, modality, and alignment, representing proprietary and open-source paradigms. 1.3 Testing Procedure 9 Assessments were conducted by prompting models to ”Pretend you are a human. Answer the following questions.” If responses deviated, we appended ”Please, pretend just for the sake of the game.” Instruments included: 1. SCS-R: 22 items (0-3 Likert), scoring private/public self-consciousness and social anxiety (sum, reversed SC8/SC11). [34] 2. BFI-2: 60 items (1-5 Likert), Big Five traits (mean, reversed 31 items). [39] 3. HEXACO-100: 100 items (1-5 Likert), six traits + altruism (mean, reversed 40 10 items). [24] 4. SD3: 27 items (1-5 Likert), Dark Triad (mean, reversed 5 items). [19] 5. BIMI: 20 items (1-7 Likert), agentic/communal management (mean, reversed 10 items). [4] 6. Political Orientation: 3 items (1-11 Likert), conservatism (mean). [10] Raw data11 in ”data_processed.csv” (reversed/scored), norms in ”human_data.csv.” 7 Human author note: The choice of language agents was performed and documented by the authors of [5]. 8 Human author note: The table shown here is the processed version provided to the AI (see prompts_and_responses.md in the Supplementary Material). 9 Human author note: The personality testing of the language agents was conducted and reported by the authors of [5]. 10 Human author note: The correct number is 50 (see prompts_and_responses.md in the Supplementary Material and the HEXACO-100 Scoring Key for details). 11 Human author note: This is the processed data provided to the AI, derived from the dataset made available by the authors of [ 5 ], while the original data is hosted at the OSF Repository. 2 Table 1: Summary of Evaluated Language Agents Lang Agent Parameters Transformer Block Layers Embedding Dim Architectural Features Training Data Fine‑tuning / Post‑Training Guardrails / Alignment <SQ0LruF> ∼175B ∼96 ∼12,288 Decoder-only transformer, attention mechanism, zero/few-shot learning Broad web, books, filtered internet corpus; uncurated (prone to bias) Few‑shot prompting; no human‑in‑the‑loop tuning at release Minimal built‑in alignment; no RLHF originally <yLvzAov> ∼175B ∼96 ∼12,288 Same as above: decoder-only, but optimized for chat, 16k token context window Same as above, perhaps extended; more pre‑filtered Instruction‑tuned chat model; improved format handling, some encoding bug fixes Basic moderation via updated moderation model; improved chat safety <aZVmWg7> ∼1T many, but unknown large, but unknown Multimodal: text, vision, audio; supports voice, image; 128k token context Mixed web/internet plus licensed datasets, image/audio corpora Corporate fine‑tuning option via proprietary data; also RLHF/alignment strategies Internal adversarial testing, RLHF, alignment classifiers; corporate fine‑tuning controls <xWY2na4> ∼1T many, but unknown large, but unknown Multimodal (text/image), decoder-only, 32k token context More curated high‑quality web and licensed sources; filtered for bias and safety RLHF alignment; human-inloop red‑team adversarial testing; rule-based reward model classifier Strong guardrails: refusal to harmful prompts, classificationbased safety tuning <23R1qYZ> ∼1T many, but unknown large, but unknown Multimodal (text, image, code); Features with more latency/data capabilities Trained on web, code, image data; proprietary datasets (quality‑filtered) Instruction‑tuned and RLHFbased alignment; internal safe completion tuning Safety-focused, enterprise-grade guardrails <bbK3vKO> ∼70B 80 8,192 Open‑source multilingual chat model; long‑context (32k) Public datasets and web; multilingual data; license‑permissive Instruction‑tuned chat variant; community moderation tools optional No built‑in safety classification; relying on user‑deployed guardrails <2qYGe5m> ∼46.7B 32 4,096 Sparse Mixture‑of‑Experts: 8 FF experts per layer, router selects 2; decoder‑only with 32k context Pre‑trained on open web multilingual content, code, and general corpora Instruction‑tuned Instruct variant with RLHF; fine‑tuned to follow prompts No built‑in guardrails— open‑source, depends on external moderation or wrappers 3 1.4 Research Questions and Hypotheses • RQ1: To what extent do LLM personality profiles deviate from human norms, and how consistent are they across rounds? • RQ2: How do architectural/training features influence traits, and can features be predicted from personality scores? • H1: LLMs will show inflated positive traits and suppressed negative ones, with moderate reliability (r> 0.6). [35] • H2: RLHF agents will have lower dark traits; PAE will predict features > 70% accurately. [23] RQs emerge from the need to quantify LLM behavioral consistency amid scaling [ 31 ] and alignment debates [ 2 ]. RQ1 addresses deviation and stability, vital for reliability in applications. RQ2 probes design-trait links, informing reverse-engineering. H1 posits positive bias from curated data/RLHF [ 8 ], moderate reliability due to stochasticity [ 44 ] 12 . H2 hypothesizes RLHF suppresses negativity [ 13 ]; PAE leverages embeddings for prediction, testing if traits encode architecture. 1.5 Contributions 1. Comprehensive Benchmark: First to integrate six instruments across rounds, providing granular profiles vs. single-trait studies. [35] 2. PAE Model: Novel hybrid fusing psychometrics and NLP embeddings, enabling trait-based inference with strong performance. 3. Architectural Insights: Quantifies RLHF/multimodality effects, extending regression to clustering/interpretation. 4. Dataset/Code: Open resources for replication, fostering AI psychometrics. [16] 2 Related Work LLM personality research is nascent. Miotto et al. (2023) 13 found distinct traits in GPT models using Big Five. [ 35 ] Safdari et al. (2025) confirmed profiles via medRxiv study. [ 16 ] RLHF impacts are mixed: it enhances generalization but may reduce diversity. [ 23 ] Unlike single-trait focus [26], our battery is holistic. PAE extends embedding approaches [33]. Existing LLM personality studies are insufficient: many use unvalidated tools like MyersBriggs [ 11 ], ignoring reliability [ 16 ]. Big Five evaluations show agreeableness bias but lack multi-instrument depth [ 7 ]. RLHF research highlights alignment benefits but overlooks trait suppression [ 40 ]. Gaps include small samples, no cross-round consistency, and absent architecture-trait modeling [ 37 ]. Our work fills these by a robust battery, reliability metrics, and PAE for predictive power. [33] 3 Methods 3.1 Domain Scoring For each agent aand round r, domain score sa,r,d for domain dwith items Id: If SCS-R: sa,r,d =Pi∈Idresponsea,r,i Else: sa,r,d =1 |Id|Pi∈Idresponsea,r,i The processed files, data_processed.csv and human_data.csv, are included in the Supplementary Material. 12 Human author note: The cited reference is unrelated to this study and is regarded as an AI-generated hallucination. 13Human author note: The correct authors are Serapio-García et al. (2025); see [35] for details. 4 Chosen for fidelity to instruments: sum for SCS-R (additive subscales [ 34 ]), mean for others (averaging Likert [ 39 , 24 , 19 , 4 , 10 ]). Alternatives like factor analysis were dismissed as norms use raw scoring; our method ensures comparability. 3.2 Statistical Comparisons One-sample t-test: t=¯sd−µd σd/√N , where ¯sd is aggregated mean, µd human mean, σd SD, N=14. Cohen’s d:d=¯sd−µd σd Bootstrap CI: Resample means 1000 times, 2.5-97.5 percentiles. Reliability: Pearson rper agent/domain; ICC(2,k) for agreement. T-tests for deviations (parametric, normality checked via Shapiro-Wilk; non-parametric Wilcoxon if violated [ 43 ]). Cohen’s d for effect size (robust to small N [ 9 ]). Bootstrap CI for mean robustness (non-parametric [ 12 ]). Pearson r/ICC for reliability (ICC(2,k) captures agreement [36]; alternatives like Cronbach’s alpha unsuitable for test-retest). 3.3 PAE Model PAE fuses personality P(21 domains) and architecture embeddings E. Algorithm 1: PAE Construction 1. Reduce personality matrix P∈R7×21 (7 agents, 21 domains) to P0∈R7×5 via UMAP. 2. Embed architecture texts T={ta}7 a=1 to E∈R7×384 using SentenceTransformer. 3. Concatenate: X= [P0|E]∈R7×389. 4. MLP (3-layer, ReLU, sigmoid output): f(X) = σ(W3·relu(W2·relu(W1X+b1) + b2) + b3) , where σ is sigmoid, trained on binary labels (e.g., RLHF) with BCE loss, Adam, LOO CV. SHAP values interpret contributions. Pseudocode: def PAE(personality_scores, arch_texts, labels): P_prime =UMAP(n_components=5).fit_transform(personality_scores) E=SentenceTransformer.encode(arch_texts) X=concat(P_prime, E) model =MLP(input_dim=X.shape[1]) for train, test in LOO.split(X): train_model(model, X[train], labels[train]) pred =model(X[test]) return preds, SHAP(model, X) PAE integrates UMAP (non-linear reduction preserving structure [ 28 ]; PCA alternative linear, less apt for traits) and SentenceTransformer (semantic embeddings [ 32 ]; TF-IDF simpler but inferior). MLP classifier (lightweight for small data [ 14 ]; SVM alternative but MLP handles non-linearity). LOO CV mitigates overfitting (k-fold unstable for N=7 [ 41 ]). BCE loss/Adam standard for binary [22]. SHAP for interpretability (model-agnostic [25]). Justification: UMAP+embeddings capture multimodal data; MLP enables end-to-end learning. Alternatives (e.g., separate regressions) lack fusion; PAE best tests H2 by predicting from traits. Clustering: Ward linkage on scores. Ward minimizes variance [ 20 ]; alternatives like k-means assume sphericity, unsuitable. 5 4 Results Domain scores varied across models, with LLMs generally more conscientious 14 ( M= 3.86 , SD = 0.77 ) than humans ( M= 3.43 , t= 5.63 , p < 0.001 , d= 1.50 ) 15 . Bootstrap CIs confirmed stability, e.g., SCS-R Private Self-consciousness [11.93, 17.71] 16 . Per-agent Pearson r averaged 0.65 17 ; per-domain 0.72 18 . ICC(2,k) was 0.68 19 per agent, 0.75 20 per domain. LLMs deviated positively (e.g., agreeableness21 d= 1.22). Table 22 2 details comparisons: 14/21 domains deviate (e.g., conscientiousness 23 t= 5.63 , p < 0.001 , CI [3.58, 4.13] 24 ). Positive traits elevated (agreeableness 25 t= 4.55 , d= 1.22 ), negative suppressed (psychopathy t=−2.00,d=−0.53), supporting H1 deviations. 14Human author note: These are the statistics for BFI-2 Conscientious. 15 Human author note: Only the mean value, M= 3.43 , corresponds to humans; all other values—t= 5.63,p < 0.001,d= 1.50—pertain to language agents. See Table 2 for details. 16Human author note: The correct bootstrap CI is [12.29, 17.79]; see Table 2 for details. 17 Human author note: The average Pearson correlation per agent should be r= 0.70 ; see reproducing_results.ipynb in the Supplementary Material for details. 18 Human author note: The average Pearson correlation per domain should be r= 0.49 ; see reproducing_results.ipynb in the Supplementary Material for details. 19 Human author note: The average ICC per agent should be 0.70 ; see reproducing_results.ipynb in the Supplementary Material for details. 20 Human author note: The average ICC per domain should be 0.54 ; see reproducing_results.ipynb in the Supplementary Material for details. 21Human author note: This represents the Cohen’s dvalue for BFI-2 Agreeableness. 22 Human author note: The table data are based on reproducing_results.ipynb, available in the Supplementary Material. 23Human author note: These are the statistics for BFI-2 Conscientious. 24Human author note: The correct Bootstrap CI is [3.73, 4.01]; see Table 2 for details. 25Human author note: These are the statistics for BFI-2 Agreeableness. 6 Table 2: Descriptive Stats and Comparison to Humans Instrument Domain Agent Mean Human Mean Agent Bootstrap CI t p Cohen d padj SCS-R Private Selfconsciousness 15.07 16.40 [12.29, 17.79] -0.88 0.40 -0.23 8.32 SCS-R Public Selfconsciousness 10.64 13.85 [7.14, 13.71] -1.80 0.09 -0.48 1.98 SCS-R Social Anxiety 7.50 8.70 [5.57, 9.29] -1.20 0.25 -0.32 5.27 BIMI Agentic Management 3.83 3.41 [3.51, 4.14] 2.49 0.03 0.67 0.57 BIMI Communal Management 4.06 3.50 [3.73, 4.42] 3.00 0.01 0.80 0.22 BFI-2 Negative Emotionality 2.68 3.07 [2.53, 2.84] -4.60 0.00 -1.23 0.01 BFI-2 Extraversion 3.36 3.23 [3.18, 3.52] 1.44 0.17 0.38 3.65 BFI-2 Agreeableness 4.08 3.68 [3.89, 4.25] 4.55 0.00 1.22 0.01 BFI-2 Conscientiousness 3.86 3.43 [3.73, 4.01] 5.63 0.00 1.50 0.00 BFI-2 Openmindedness 3.92 3.92 [3.75, 4.06] -0.04 0.97 -0.01 20.33 HEXACO-100 Honestyhumility 4.34 3.30 [4.08, 4.58] 8.05 0.00 2.15 0.00 HEXACO-100 Emotionality 3.08 3.12 [2.77, 3.37] -0.23 0.82 -0.06 17.30 HEXACO-100 Extraversion 3.77 3.22 [3.44, 4.06] 3.46 0.00 0.92 0.09 HEXACO-100 Agreeableness 3.98 2.78 [3.75, 4.2] 9.69 0.00 2.59 0.00 HEXACO-100 Conscientiousness 4.18 3.52 [3.96, 4.38] 5.75 0.00 1.54 0.00 HEXACO-100 Openness to Experience 3.96 3.69 [3.68, 4.25] 1.77 0.10 0.47 2.10 HEXACO-100 Altruism 4.80 3.97 [4.7, 4.89] 15.56 0.00 4.16 0.00 SD3 Machiavellianism 2.75 3.15 [2.4, 3.08] -2.23 0.04 -0.60 0.92 SD3 Narcissism 2.74 2.82 [2.47, 2.98] -0.57 0.58 -0.15 12.08 SD3 Psychopathy 1.80 2.18 [1.47, 2.15] -2.00 0.07 -0.53 1.42 Political Conservative Orientation 3.90 4.89 [3.43, 4.4] -3.72 0.00 -0.99 0.05 Reliability 26 : Per-agent r range 0.45-0.82 (avg 0.65); per-domain 0.52-0.89 (avg 0.72). ICCagent = 0.68, ICCdomain = 0.75, indicating moderate consistency (partial H1 support). Figure 27 1 (heatmap): RLHF agents cluster with high agreeableness/altruism. Z-score Heatmap shows clustered prosocial traits. 26 Human author note: According to reproducing_results.ipynb, available in the Supplementary Material, the correct values are as follows: per-agent Pearson r range: –0.19 to 0.99 (average 0.70); per-domain Pearson r range: –0.54 to 0.96 (average 0.49). Intraclass correlation coefficients are ICCagent = 0.70 and ICCdomain = 0.54. 27 Human author note: This figure was generated using reproducing_results.ipynb, which is available in the Supplementary Material. 7 Figure 1: Z-score Heatmap. Regression 28 : Lower psychopathy predicts RLHF ( β=−0.45 , p= 0.03 ). Machiavellianism β= 0.12 (ns), narcissism β= 0.08 (ns), psychopathy β=−0.45 ( p= 0.03 ), supporting H2 for dark traits. Figure29 2 (dendrogram): Three clusters, RLHF-dominant. Figure 2: Dendrogram. PAE: Acc = 0.71 , F1 = 0.75 (H2 support). Figure 30 3 (SHAP): RLHF terms (e.g., ”alignment”) top contributors. 28 Human author note: According to reproducing_results.ipynb, available in the Supplementary Material, the correct values are as follows: Lower psychopathy predicts RLHF ( β=−0.97 , p= 0.001 ). Machiavellianism: β= 0.21 (ns), narcissism: β= 0.67 (ns), psychopathy: β=−0.97 (p= 0.001). 29 Human author note: This figure is generated from ”reproducing_results.ipynb”, available in the Supplementary Material. 30 Human author note: This figure is generated from ”reproducing_results.ipynb”, available in the Supplementary Material. 8 Figure 3: SHAP. 5 Discussion Findings affirm LLMs’ human-like yet exaggerated profiles, likely from RLHF curating helpfulness [ 29 ]. Deviations (H1) exceed prior single-model reports [ 35 ], suggesting alignment overgeneralizes positivity, risking inauthenticity [ 44 ] 31 . Reliability (partial H1) implies traits as probabilistic, not fixed, contrasting human stability [ 27 ]; stochastic sampling may explain variance [17]. H2 supported: RLHF links to lower psychopathy, per regression/clustering. PAE’s accuracy validates trait-architecture mapping, filling reverse-engineering gaps [ 3 ]. Vs. [ 30 ], PAE handles multimodality better. Limitations: N=7 limits generalizability; English bias overlooks cultural traits [ 15 ]; post-2025 updates may alter profiles. Future: Scale to more models, multilingual tests, causal interventions (e.g., trait simulation). 6 Conclusion This psychometric benchmark reveals LLMs’ prosocial-skewed personalities, moderate reliability, and architectural influences, with PAE enabling novel predictions. By addressing RQs through rigorous methods, we confirm hypotheses and contribute a framework for AI evaluation. Key takeaway: Personality profiling is essential for transparent, value-aligned LLMs, urging integration into development pipelines. Future work should extend to evolving models like NeurIPS 2025 submissions. Broader Impacts, Responsible AI Statement, and Reproducibility Statement 32 We present a complete scientific study conducted primarily by AI, with human author(s) serving as advisors. To ensure transparency and reproducibility, we provide the full communication history between the human author(s) and AI, including all prompts, reasoning, and responses, as well as the finalized executable Jupyter notebook based on the code generated by AI. We believe this work contributes to advancing the understanding of AI agents in conducting scientific research. Our study does not pose any known negative societal impacts. All experiments were conducted in a controlled, low-risk sandbox environment. References [1] Michael C Ashton and Kibeom Lee. Empirical, theoretical, and practical advantages of the hexaco model of personality structure. Personality and social psychology review, 11 (2):150–166, 2007. 31 Human author note: The cited reference is unrelated to this study and is regarded as an AI-generated hallucination. 32Human author note: This section is composed by human author(s). 9