Evaluating the performance of large language models in predicting diagnostics for Spanish clinical cases in cardiology

Academic Editors: Douglas O’Shaughnessy and Suchao Xie Received: 25 November 2024 Revised: 16 December 2024 Accepted: 20 December 2024 Published: 25 December 2024 Citation: Delaunay, J.; Cusido, J. Evaluating the Performance of Large Language Models in Predicting Diagnostics for Spanish Clinical Cases in Cardiology. Appl. Sci. 2025,15, 61. https://doi.org/10.3390/ app15010061 Copyright: © 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/ licenses/by/4.0/). Article Evaluating the Performance of Large Language Models in Predicting Diagnostics for Spanish Clinical Cases in Cardiology Julien Delaunay 1,* and Jordi Cusido 1,2,* 1Top Health Tech, 08021 Barcelona, Spain 2Departament de Projectes i Construccio, Universitat Politecnica de Catalunya, 08034 Barcelona, Spain *Correspondence: [email protected] (J.D.); [email protected] (J.C.) Abstract: This study explores the potential of large language models (LLMs) in predicting medical diagnoses from Spanish-language clinical case descriptions, offering an alternative to traditional machine learning (ML) and deep learning (DL) techniques. Unlike ML and DL models, which typically rely on extensive domain-specific training and complex data preprocessing, LLMs can process unstructured text data directly without the need for specialized training on medical datasets. This unique characteristic of LLMs allows for faster implementation and eliminates the risks associated with overfitting, which are common in ML and DL models that require tailored training for each new dataset. In this research, we investigate the capacities of several state-of-the-art LLMs in predicting medical diagnoses based on Spanish textual descriptions of clinical cases. We measured the impact of prompt techniques and temperatures on the quality of the diagnosis. Our results indicate that Gemini Pro and Mixtral 8x22b generally performed well across different temperatures and techniques, while Medichat Llama3 showed more variability, particularly with the few-shot prompting technique. Low temperatures and specific prompt techniques, such as zero-shot and Retrieval-Augmented Generation (RAG), tended to yield clearer and more accurate diagnoses. This study highlights the potential of LLMs as a disruptive alternative to traditional ML and DL approaches, offering a more efficient, scalable, and flexible solution for medical diagnostics, particularly in the non-English-speaking population. Keywords: large language models; medical diagnosis; natural language processing; healthcare; Spanish language; prompt techniques; temperature settings 1. Introduction Symptom checkers based on traditional machine learning (ML) and deep learning (DL) approaches have become increasingly common in healthcare applications. Early systems, such as rule-based expert systems, relied on predefined algorithms and decision trees to suggest potential diagnoses based on input symptoms [ 1 , 2 ]. While these systems provided some degree of automation, they were often limited in terms of their flexibility and precision, particularly in complex or ambiguous cases. As computational power advanced, more complex ML and DL techniques were introduced to improve the accuracy of symptom checkers. These methods, which include random forests, support vector machines, and neural networks, were trained on large datasets to predict diagnoses more effectively. However, they still suffer from several key limitations, primarily related to the need for extensive, domain-specific training and the risks of overfitting [3,4]. The reliance on training data is one of the most significant constraints of ML and DL approaches. These models require large, high-quality labeled datasets to function Appl. Sci. 2025,15, 61 https://doi.org/10.3390/app15010061 Appl. Sci. 2025,15, 61 2 of 31 optimally, often necessitating domain-specific knowledge to ensure their accuracy. In many clinical settings, obtaining sufficiently detailed and labeled data can be a daunting task [ 5 ]. Moreover, these systems often demand the input of a complete set of symptoms to make an accurate diagnosis, which may not always be available in real-world scenarios [6]. Despite advancements in ML and DL, symptom checkers still struggle with generalization and adaptability. Each model must be retrained when faced with a new set of conditions or medical specialties, limiting their flexibility. In contrast, large language models (LLMs), which have shown significant promise in a variety of natural language processing tasks, offer a new approach to medical diagnosis. Unlike traditional ML/DL models, LLMs are capable of leveraging vast amounts of text data without the need for extensive retraining [ 7 – 10 ]. These notable improvements can be partly attributed to the adoption of methods that encode and manipulate text data using latent representations. Those methods embed text into high-dimensional vector spaces that capture the underlying semantics and structure of language. Thus, they can process incomplete information, adapt to various medical domains, and generate diagnostics in a conversational context, making them a promising alternative for developing more robust and generalizable symptom checkers [ 11 ]. These innovations have paved the way for novel patient care solutions, particularly in the diagnosis of diseases [ 12 ]. In this study, we focus on evaluating the effectiveness of LLMs in predicting diagnostics for cardiological diseases based on clinical cases in Spanish. The use of LLMs as symptom checkers has gained attention due to their potential to improve medical education and patient care. However, the performance of LLMs in diagnosing diseases in non-English languages and the impact of prompt techniques and temperature levels on their performance remain under-explored. In this paper, we aim to address these gaps by evaluating the impact of zero-shot, few-shot, and RetrievalAugmented Generation (RAG) prompt techniques as well as temperature levels (low vs. high) on the quality of diagnostic predictions made by LLMs in Spanish cardiology cases. We evaluate three LLMs, namely Gemini Pro [ 13 ], Mixtral 8x22b [ 14 ], and Medichat Llama3 [https://huggingface.co/sethuiyer/Medichat-Llama3-8B], using two sets of clinical cases in Spanish [ 15 , 16 ], accessed on 20 June 2024. Our results indicate that Gemini Pro and Mixtral 8x22b generally performed well across different temperatures and techniques, while Medichat Llama3 showed more variability, particularly with the few-shot prompting technique. Low temperatures and specific prompt techniques, such as zero-shot and RAG, tended to yield clearer and more accurate diagnoses compared to high-temperature and few-shot prompting. These findings highlight the importance of selecting appropriate models, prompt techniques, and temperature settings to optimize diagnostic performance. Our methodology consists of presenting the LLMs with symptom descriptions and test results in Spanish. We then evaluate the completeness, precision, recall, and quality of the diagnostic predictions made by each model using LLM-based evaluators [ 17 ]. Our study contributes to a deeper understanding of the role of artificial intelligence in medical diagnosis and the impact of prompt techniques and temperature levels on the performance of LLMs in non-English languages. The results of our study can inform the strategic use of LLMs in healthcare, suggesting a balance between sensitivity and realism to optimize patient outcomes. The remainder of this paper is organized as follows. We begin by defining in Section 2 the key concepts necessary to understand our methodological approach. Section 3provides an overview of related work on the use of LLMs in medical diagnosis and the impact of prompt techniques and temperature levels on their performance. Section 4describes the datasets, LLMs, prompt techniques, and temperature levels used in our study. Section 5 presents our methodology for evaluating the diagnostic predictions made by the LLMs. Appl. Sci. 2025,15, 61 3 of 31 Section 6reports the results of our study, while Section 7discusses the implications of our findings for the use of LLMs in healthcare. Finally, Section 8concludes the paper and outlines directions for future research. 2. Key Concepts in Natural Language Processing This section provides an overview of the key natural language processing (NLP) concepts and technologies used in this study to ensure that readers from the healthcare domain are familiar with the foundational concepts. We begin by discussing tokenization and embeddings, which are fundamental to NLP. We then present Large Language Models (LLMs), which are central to our approach, covering advanced techniques such as Mixture of Experts (MoE) and fine-tuning of LLMs to specific domains. Finally, we discuss the impact of prompts and temperature settings on their performance. 2.1. Tokenization and Embeddings Tokenization is a crucial preprocessing step in NLP that converts text into tokens, which are smaller units of meaning. In the context of LLMs, tokenization is essential for transforming clinical case descriptions into a format that the model can understand and process. Effective tokenization ensures that the context and semantic meaning of the clinical data are preserved. Mathematically, tokenization can be described as: T= Tokenizer(X) (1) where Xis the input text, and Tis the resulting tokenized representation. Tokenization is particularly important in symptom checkers, where the accuracy of the diagnostic predictions can directly impact patient outcomes. For clinical cases, where medical jargon and complex terminology are common, this method ensures that words like “myocardial infarction” or “hypertension” are adequately handled by breaking them down into subword units that the model can understand and process efficiently. The process involves breaking down the text into individual words or subwords, which are then converted into numerical representations called embeddings. Embeddings capture the semantic meaning of the text, allowing the LLM to understand the context and relationships between different parts of the clinical description. There are several types of embeddings, including word embeddings (Word2Vec [ 18 ] and GloVe [ 19 ]), subword embeddings (Byte Pair Encoding and WordPiece), and contextual embeddings. In BERT [ 7 ] and RoBERTa [ 8 ], contextual embeddings are generated through a process that maps tokens to high-dimensional vectors etusing a pre-trained lookup table: et=Embeddings(t)(2) where et is the vector representation of token t. These vectors are used by the model to understand and generate text by considering the context in which a word appears, allowing the same word to have different embeddings based on its usage. For instance, the representation of a word like “pain” will change depending on the surrounding terms (e.g., “chest pain” vs. “muscle pain”). Mixture of Experts (MoE) is an advanced technique used to enhance the capabilities of LLMs by leveraging a combination of specialized sub-models, or “experts,” each trained to handle specific aspects of a task. In an MoE architecture, the model dynamically selects the most appropriate experts to process different parts of the input, allowing for more efficient and effective handling of complex tasks. This approach has been shown to improve the performance and scalability of LLMs, especially in scenarios requiring specialized knowledge or fine-grained processing [ 20 ]. Subword tokenization is commonly used in Appl. Sci. 2025,15, 61 4 of 31 MoE, where specialized embeddings are often computed based on the “expert” selected by the gating mechanism, enhancing the model’s ability to specialize in specific domains. Thus, the embedding process can be formalized as: et= N ∑ i=1 αi·ei,t(3) where ai represents the attention weights from the gating mechanism selecting which experts iwill contribute to the final embedding et, and Nis the number of experts. 2.2. Large Language Models Large Language Models are a class of artificial intelligence models designed to understand and generate human language. These models are trained on vast amounts of text data and can perform a wide range of NLP tasks, such as text generation, translation, summarization, and question-answering [ 13 , 14 , 21 , 22 ]. LLMs leverage deep learning techniques, particularly transformer architectures [ 10 ], to capture complex linguistic patterns and contextual information. Once the tokens are tokenized and embedded, they pass through layers of attention and neural activations, where the model focuses on the most relevant parts of the input. This activation is achieved using self-attention layers, which allow each token to attend to every other token in the input. The attention mechanism is defined mathematically as: Attention(Q,K,V)=softmax QKT √dk!V(4) where Qis the query matrix (representing the current token), Kis the key matrix (representing all tokens), Vis the value matrix (containing the content of the tokens), and dk is the dimension of the key matrix (scaling factor). This attention mechanism allows BERT to learn contextual relationships between tokens, which is crucial in complex medical diagnoses, where understanding the relationship between “chest pain” and “shortness of breath” is essential. In MoE, the gating mechanism selects the most appropriate subset of experts to process the tokens. Each token tis processed by a subset of experts Et: zt=∑ i∈Et Wi·et(5) where zt is the activated output for token t, and Wi represents the weights associated with each expert i . The gating mechanism ensures that only the most relevant experts are activated, optimizing the network’s efficiency and domain-specific performance. During propagation, information moves through multiple layers, refining the embeddings to generate a final diagnostic prediction. This process involves a series of transformations, with each layer applying a transformation f: hl=f(hl−1)(6) where hl is the hidden state of layer l, and fis the transformation function, which could be a feed-forward neural network or another attention layer. This iterative process allows the model to refine its understanding of the clinical case, producing a more accurate diagnosis. Fine-tuning is a process used to adapt pre-trained LLMs to specific domains or tasks by further training the model on domain-specific data. This approach allows the model to learn the unique linguistic patterns, terminology, and contextual nuances of the target domain, Appl. Sci. 2025,15, 61 5 of 31 enhancing its performance and accuracy. Fine-tuning has been successfully applied in various domains, including healthcare, where models are fine-tuned using specialized clinical data to improve their understanding and generation of medical information [ 23 , 24 ]. By adapting the model to the unique characteristics of the target domain, fine-tuning can significantly improve its accuracy and reliability. Fine-tuned models are better equipped to understand and generate contextually relevant responses, making them more effective in specialized applications. In the context of this study, LLMs are evaluated for their ability to predict medical diagnoses from Spanish textual descriptions of clinical cases. Techniques such as examplebased prompting and Retrieval-Augmented Generation (RAG) are explored to enhance the quality of predictions. The impact of temperature settings and prompt strategies on diagnostic accuracy is assessed, contributing to the understanding of the strengths and limitations of LLMs in this domain. 2.3. Impact of Prompts and Temperature Prompts and temperature settings play a crucial role in the performance of LLMs, particularly in tasks requiring nuanced understanding and accurate predictions. The way input prompts are crafted significantly influences the model’s output. Techniques such as zero-shot prompting, few-shot prompting, and Retrieval-Augmented Generation (RAG) allow for diverse ways of guiding the model’s reasoning process. For instance, zero-shot prompting requires no additional examples and relies entirely on the model’s general knowledge, making it efficient for straightforward cases. Few-shot prompting incorporates a small set of examples Kwithin the input prompt, enabling the model to infer patterns and produce more contextually accurate results: DK={(Xi,yi)}K i=1(7) where Xi represents the input text (clinical case), and yi is the corresponding diagnosis. This set helps the model learn the structure of the input and apply it to a new case. RetrievalAugmented Generation leverages external knowledge sources to supplement the model’s responses, improving precision in complex scenarios. The temperature Tis a hyperparameter that controls the randomness of the model’s output. Temperature adjusts the probability distribution of the model’s predictions, with lower temperatures constraining the model to produce more deterministic and focused responses, which is beneficial for tasks demanding accuracy, such as medical diagnoses. Conversely, higher temperatures introduce variability, which may lead to creative but less consistent outputs. Mathematically, the softmax function, used in generating predictions, is modified by temperature as follows: P(y|X)=expzy T ∑y′expzy′ T(8) where Zy is the score (logit) for the class y,P(y|X) is the probability of a particular class y given input X. When Tis small (e.g., 0.1), the softmax output becomes more concentrated, meaning the model will tend to choose the most likely outcome. This is useful for precision when a clear diagnosis is present. When Tis large (e.g., 2), the output becomes more uniform, allowing the model to explore a wider range of potential diagnoses, which is helpful for ensuring sensitivity to less obvious but relevant diagnoses, particularly in cases with rare or complex symptoms. Appl. Sci. 2025,15, 61 6 of 31 By carefully selecting the input and tailoring the prompt techniques (zero-shot, fewshot, and RAG), we can maximize both precision and sensitivity in the diagnostic predictions. The model’s ability to handle diverse and complex clinical cases, especially rare diseases, is enhanced by the combination of RAG, few-shot learning, and the dynamic adjustment of temperature. Zero-shot learning allows the model to make predictions from a baseline understanding, while few-shot learning helps it generalize from prior examples, and RAG enriches the model’s knowledge with external, case-specific information. By leveraging these techniques, we ensure that the model remains flexible and capable of delivering accurate and reliable diagnoses. Temperature plays a key role in fine-tuning the balance between precision (focusing on the most probable diagnoses) and sensitivity (exploring less common but relevant possibilities), making the overall diagnostic system more robust and adaptable to various clinical scenarios. This study sheds light on the optimal configurations for leveraging LLMs in medical diagnosis prediction, particularly for Spanish-language clinical cases. 3. Related Works Machine learning methods have been applied to electronic health records (EHRs) for various clinical predictions [ 25 – 27 ]. These methods can handle high-dimensional data and find new features or nonlinear relationships in the data. However, issues related to data quality, such as missingness, misclassification, and measurement error, can impact the performance of these models [ 26 ]. Careful evaluation of the capabilities and limits of these models can help mitigate some of these concerns [26]. While traditional machine learning algorithms and rule-based systems have been used for diagnostic tasks, they often require specific training on datasets with clear target classes and structured data [ 28 , 29 ]. In contrast, LLMs offer the advantage of processing unstructured textual data directly, making them well-suited for handling the complexities of clinical reports. LLMs can capture nuanced linguistic patterns and contextual information, which are essential for accurate diagnostic predictions in cardiology [30,31]. Large language models (LLMs) have shown significant potential in various healthcare applications, including the analysis of clinical text for predicting patient outcomes [32–35] . These models can analyze large amounts of data, identify patterns, and assist in the analysis and understanding of risk factors for diseases. The use of LLMs in diagnostic predictions has gained attention, particularly with the advent of models like GPT-4, Gemini, and Llama-3 [ 24 , 36 ]. Domain-specific LLMs have demonstrated exceptional performance on multiple natural language processing tasks, surpassing the performance of general LLMs [ 37 ]. Research on the application of LLMs to non-English languages has been growing. For instance, clinical pre-trained language models can be used to analyze text from admission and medical reports to predict the probability of no coverage in a labor insurance process [ 38 ]. However, research on the performance of LLMs in diagnostic predictions in non-English languages, such as Spanish, remains limited. There are various parameters that impact the quality of the LLMs; among these, the use of prompt techniques and temperature have emerged as powerful approaches to leverage the capabilities of LLMs [39]. Prompt-based learning enables LLMs to perform prediction tasks by modifying the input using a template and filling in the unfilled information [ 39 ]. Temperature is a hyperparameter used in LLMs to control the randomness of the generated outputs [ 40 ]. Adjusting the temperature can impact the diversity and quality of the generated text. A recent study [40] investigated the effect of sampling temperature on the performance of LLMs on various problem-solving tasks. Although this study did not focus specifically on healthcare applications, its findings suggest that temperature may affect the performance of LLMs in diagnostic predictions. Appl. Sci. 2025,15, 61 7 of 31 Despite the growing interest in applying LLMs to healthcare and non-English languages, there is a lack of research on the impact of prompt techniques and temperature on the performance of LLMs in diagnostic predictions in Spanish. This study aims to address this gap by evaluating the accuracy and reliability of LLMs in predicting medical diagnoses based on Spanish textual descriptions of symptoms while exploring the effects of prompt techniques and temperature on their performance. Our findings will contribute to the understanding of the strengths and limitations of LLMs in diagnostic predictions in Spanish and provide insights for future research in this area. 4. Method In this section, we first introduce the hypotheses that guide our investigation into the capabilities of LLMs in predicting medical diagnoses from Spanish textual descriptions of clinical cases. We then present the datasets we used to evaluate the capacities of diverse LLMs. Following this, we outline the various models we used to predict the diagnosis and evaluate the capacities of each model. Finally, we present technical details, including prompt techniques and temperature values. 4.1. Hypotheses This study aims to investigate the capacities of LLMs in predicting medical diagnoses based on Spanish textual descriptions of clinical cases. We hypothesize that LLMs demonstrate a high level of adaptability and generalization to diverse and complex cardiology cases. We explore the impact of different prompt techniques and temperature settings on the diagnostic accuracy of LLMs. Specifically, we hypothesize that few-shot prompting techniques will enhance diagnostic accuracy (H1.1) and that RAG will further improve predictions by incorporating external knowledge sources (H1.2). Additionally, we expect that lower temperature settings will result in more deterministic and focused predictions (H2.1), while higher temperature settings will introduce variability and creativity (H2.2). Finally, we aim to compare the performance of different LLMs, hypothesizing that models with advanced techniques such as MoE and fine-tuning will show superior performance (H3.1, H3.2). Hypothesis 1. Impact of Prompt Techniques. • H1.1: Few-shot prompting techniques will enhance the diagnostic accuracy of LLMs by providing contextual guidance and reducing the need for extensive training data. • H1.2: RAG will further improve diagnostic predictions by incorporating external knowledge sources, particularly in cases with ambiguous or complex symptoms. Hypothesis 2. Influence of Temperature Settings. • H2.1: Lower temperature settings will result in more deterministic and focused diagnostic predictions, leading to higher accuracy in straightforward cases. • H2.2: Higher temperature settings will introduce variability and creativity in the predictions, which may be beneficial for exploring alternative diagnoses in ambiguous cases but could also lead to less consistent results. Hypothesis 3. Variability Among LLMs. • H3.1: Different LLMs (e.g., Gemini Pro, Mixtral 8x22b, Medichat Llama3) will exhibit varying performance levels due to differences in their architectures, training data, and parameter settings. • H3.2: Models with advanced techniques such as MoE and fine-tuning will show superior performance in handling the complexities of cardiology diagnoses. Appl. Sci. 2025,15, 61 8 of 31 4.2. Datasets In the field of cardiology, clinical cases often include detailed descriptions of patient history, current symptoms, physical examination findings, diagnostic test results, and clinical course. These descriptions are crucial for making accurate diagnoses and developing effective treatment plans. In this study, we extracted the clinical cases from two datasets in the field of cardiology published in 2018 [ 15 ] and 2020 [ 16 ]. The datasets were obtained from the “Sociedad Española de Cardiología”, and they included a total of 73 and 94 clinical cases, respectively, both written in Spanish. Although all cases are related to some form of heart disease, they were selected to have a variety of diagnoses, ensuring that the models are tested on a broad spectrum of diagnostic challenges. Each clinical case consisted of the following sections: 1. Introduction: Present the case and provide background information. 2. Medical History: Include the patient’s medical history, such as previous illnesses, surgeries, and medications. 3. Current Illness and Physical Examination: Contains information about the patient’s current illness and physical examination findings. 4. Diagnostic Tests: List the results of any diagnostic tests performed on the patient, such as laboratory tests, imaging studies, and electrocardiograms. 5. Clinical Course: Describe the patient’s clinical course, including any treatments administered and their response to treatment. Predicting cardiology diagnoses is particularly challenging due to the complexity and diversity of heart diseases. Cardiology encompasses a wide range of conditions, from congenital heart defects to acquired heart diseases such as coronary artery disease, heart failure, and arrhythmias. Each of these conditions presents unique symptoms, diagnostic criteria, and treatment options, making accurate diagnosis a multifaceted task. The diversity of possible diagnoses in cardiology adds another layer of complexity. For instance, chest pain, a common symptom, can be indicative of various conditions ranging from stable angina to acute myocardial infarction, pulmonary embolism, or even gastrointestinal disorders. The overlap in symptoms among different cardiac conditions requires a nuanced understanding of clinical presentations and diagnostic tests. Table 1provides additional information about the datasets, including the mean number of characters per clinical case and the average number of diagnoses per case. As shown in the table, the mean number of characters per case was higher in the 2020 dataset compared to the 2018 dataset. This is likely due to the fact that the 2020 cases were more complex and required more detailed descriptions. The average number of diagnoses per case was also higher in the 2020 dataset, indicating that these cases involved more comorbidities and complex medical conditions. Table 1. Information about the experimental datasets. We indicate the information about the number of diagnoses in the row “Diag.” while information about the number of characters per document is provided in the row “Charac”. The “Mean” and “Median” columns denote, respectively, the average and median number of characters and diagnoses per document. The column “ σ ” represents the standard deviation. Dataset Type Mean Median σ 2018 Diag. 3.12 3 1.9 Charac. 5882 5513 2019 2020 Diag. 4.13 4 1.92 Charac. 8019 7648 2416 Appl. Sci. 2025,15, 61 9 of 31 It is worth noting that the datasets were anonymized to protect patient privacy. All personal identifying information was removed, and each case was assigned a unique identifier. This ensured that the data were used ethically and in compliance with relevant regulations . In this study, we did not train the LLMs on these datasets; instead, we used them solely for evaluation purposes. The datasets serve as real-world clinical cases to test the diagnostic capabilities of the LLMs. Therefore, we did not split the dataset into training, validation, and testing subsets, nor did we monitor for overfitting or underfitting. The primary task of this study is to measure how varying prompt techniques (zero-shot, fewshot, and Retrieval-Augmented Generation (RAG)) and temperature levels (low and high) affect the quality of diagnostic predicted by three LLMs. 4.3. Models In this study, we selected three cutting-edge LLMs to assess their proficiency in predicting diagnoses based on clinical cases. Each model possesses distinct features and capabilities that render them well-suited for this task and merit further exploration. To facilitate the reproducibility of our experiments, we leverage the Langchain library [https://www.langchain.com/], accessed 20 July 2024 , which serves as the foundation for the LLMs employed in our evaluation. By using the Langchain library, researchers can replicate our experiments and build upon our findings to advance the state of the art in LLM-based diagnosis prediction. The LLMs evaluated in this study were accessed on 20 August 2024 and are presented in the text and in Table 2, in order to increase transparency, ranging from the more opaque to the most open-source model. Table 2. This table includes information on the LLMs evaluated in this study, such as the number of parameters, whether the model was fine-tuned, whether it is open source and the company that developed it. * Google has not publicly disclosed the estimated count, which is the exact number of parameters. ** Mixtral 8x22b is a SMoE model with 22 distinct models at 7B, using 39B active parameters out of 141B. Model Name Parameters Fine-Tune Open Source Company Gemini Pro 540B * No No Google Mixtral 8x22b 141B ** No Yes Mistral AI Medichat-Llama3 8B Yes Yes Meta 4.3.1. Gemini Pro Gemini Pro is a highly compute-efficient multimodal model capable of recalling and reasoning over fine-grained information from millions of tokens of context [ 13 ]. It achieves near-perfect recall on long-context retrieval tasks across modalities and improves the stateof-the-art in long-document QA, long-video QA, and long-context ASR. Gemini Pro has been shown to match or surpass the performance of other state-of-the-art models on a broad set of benchmarks. 4.3.2. Mixtral 8x22b In this study, we have selected Mixtral 8x22b, a powerful variant of Mistral AI’s previous model [ 14 ], Mixtral 8x7b. Mixtral 8x22b is a sparse Mixture-of-Experts (SMoE) model [ 20 ] with 39B active parameters out of 141B, making it a significantly larger model size with 22 times more parameters than its predecessor. This increased parameter count aims to enhance Mixtral’s capabilities in handling complex tasks and understanding nuanced language patterns. Mixtral 8x22b offers high performance and efficiency and is fluent in multiple languages, including Spanish. Its strong mathematics and coding capabilities make it well- Appl. Sci. 2025,15, 61 16 of 31 Several key insights can be drawn from these results. First, the zero-shot technique consistently results in a lower number of input tokens compared to the few-shot and RAG techniques. This indicates that zero-shot prompting is more efficient in terms of costs and input requirements, making it suitable for straightforward cases requiring quick and concise diagnostics. Second, the RAG technique shows a more balanced approach, with a moderate number of input tokens and consistent output token counts across different temperatures. Third, the significant increase in output tokens for the evaluator (GPT-4o) with zero-shot and few-shot techniques at higher temperatures highlights the diagnostic model’s tendency to generate more detailed and exploratory responses. Finally, the pricing information indicates that the cost of using these models varies with the prompt technique and temperature setting. Higher temperatures and more context-rich prompts generally result in higher costs due to the increased number of output tokens. This underscores the importance of carefully selecting prompt techniques and temperature settings to balance diagnostic accuracy and computational cost. 6.5. Qualitative The qualitative analysis involved evaluating 90 generated diagnoses by examining five diagnostics produced by each LLM (Gemini Pro, Mixtral 8x22b, and Medichat Llama 3) across every combination of temperature settings (low vs. high) and prompt techniques (zero-shot, few-shot, and RAG). This evaluation focused on key aspects such as clarity, precision, justification, diagnostic scope, and actionability. Importantly, the analysis was conducted without comparing the generated outputs to actual diagnoses, allowing us to assess the style and characteristics of the diagnostics based solely on the model and parameter settings. This approach facilitated a comprehensive understanding of how different configurations influence the quality of diagnostic outputs. We provide an example of diagnoses from each model in Appendix B. Details about how we conducted this evaluation and in-depth results are available in Appendix C. 6.5.1. General Impact of Temperature Across all LLMs and prompt techniques, low temperatures consistently produced clearer, more precise, and well-justified diagnostics. These outputs are highly structured and grounded in clinical data. However, in some cases (e.g., Medichat Llama 3), there may be an occasional over-elaboration or redundancy, which can slightly detract from overall clarity but does not diminish clinical accuracy. Higher temperatures, on the other hand, increase variability, creativity, and the range of diagnostic possibilities. Across all models, high temperatures introduce more speculative and overlapping diagnoses, leading to verbosity and less focus. Justifications become weaker, with some diagnoses lacking strong clinical support. 6.5.2. Impact of Prompt Techniques Zero-shot prompting offers clarity and precision but lacks depth, making it less effective for handling rare or complex conditions. Conversely, Few-Shot prompting enhances diagnostic accuracy by incorporating contextual examples. It is highly effective in complex cases where examples can guide the model towards more nuanced and precise outputs. However, at high temperatures, this technique risks over-expansion and redundancy. Finally, RAG prompting offers the most comprehensive technique, offering in-depth, wellsupported diagnostics by integrating external data. It is highly effective for complex clinical scenarios but can become dense and speculative, particularly at high temperatures. Appl. Sci. 2025,15, 61 17 of 31 7. Discussion The findings of this study provide valuable insights into the accuracy and reliability of LLMs in predicting medical diagnoses based on Spanish textual descriptions of clinical cases. The results highlight both the potential and the limitations of LLMs in healthcare applications, particularly for non-English speaking populations. 7.1. Influence of Prompt Techniques The impact of different prompt techniques on the diagnostic accuracy of LLMs was a key focus of this study. Gemini Pro and Mixtral 8x22b demonstrated consistently high performance across various metrics and prompt techniques, suggesting their robustness and reliability in generating accurate and comprehensive diagnoses. This consistency is particularly notable given the complexity and variability of the clinical cases used in this study. The performance of Medichat Llama3, however, was more variable. Its lower precision and recall scores with the few-shot prompting technique indicate that this model may struggle with certain types of prompts or require more specific tuning, which contradicts Hypothesis 1.1. The improvement in its performance with the RAG technique underscores the potential benefits of incorporating external data to enhance diagnostic accuracy, validating Hypothesis 1.2. From the qualitative analysis, we noted that zero-shot prompting was best suited for straightforward cases requiring quick and concise diagnostics. Few-shot prompting enhanced diagnostic accuracy by incorporating contextual examples, but Medichat Llama3’s variable performance suggests that this technique may not always be effective, especially with smaller and fine-tuned LLMs. RAG prompting offered the most comprehensive technique, providing in-depth and well-supported diagnostics by integrating external data. It is, therefore, ideal when detailed evidence and broader exploration are necessary. However, RAG prompting requires careful tuning to maintain focus and avoid excessive diagnoses, as it can become dense and speculative, particularly at high temperatures. 7.2. Influence of Temperature Settings The impact of temperature settings on the models’ performance is also noteworthy. Lower temperatures generally resulted in more consistent and accurate diagnoses, while higher temperatures introduced more variability. This highlights the importance of carefully selecting temperature settings to balance precision and creativity in diagnostic predictions, aligning with Hypothesis 2. Low temperatures consistently produced clearer, more precise, and well-justified diagnostics, making them reliable for actionable decision-making in healthcare settings, as per Hypothesis 2.1. The focus is primarily on well-supported, probable conditions, avoiding speculative diagnoses. Higher temperatures increase variability and creativity, which can be valuable for exploring rare or nuanced conditions but often comes at the cost of precision and clarity. High temperatures are better suited for brainstorming or exploratory tasks rather than definitive diagnostics, confirming Hypothesis 2.2. 7.3. Variability Among LLMs The study also highlighted the variability in performance among different LLMs. Gemini Pro and Mixtral 8x22b demonstrated consistently high performance across various metrics, suggesting their robustness and reliability in generating accurate and comprehensive diagnoses, as per Hypothesis 3.1. However, the performance of Medichat Llama3 was more variable, indicating that this model may struggle with certain types of prompts or require more specific tuning. Appl. Sci. 2025,15, 61 18 of 31 While we hypothesized in Hypothesis 3.2 that both MoE and fine-tuning models would show superior performance, our results highlight the limits of smaller and finetuned models as the performance of Medichat Llama3 was more variable. This might be due to the limited number of parameters in the model or the fine-tuning of specific healthcare datasets that are too far from the one we tested (e.g., only English reports). This underscores the importance of carefully selecting and adapting models to the specific context in which they will be used. 7.4. Limitation This study has several limitations that should be considered. One limitation of this study is the lack of control over the data used to train the LLMs. Since these models were trained on large corpora of text, it is impossible to know exactly what information they have learned or how it may influence their performance. Additionally, by using publicly available datasets to evaluate the models, there is a risk of data contamination, as the models may have already been exposed to some or all of the data during training [ 46 ]. This could potentially inflate their performance metrics and lead to an overestimation of their true capabilities. Ideally, a different dataset that is completely independent of the evaluated dataset would be included to further validate the models’ performance. However, due to the limited availability of such data, we were unable to include an external dataset in this study. Therefore, to mitigate this risk, we used Spanish PDF documents in our evaluation, which may be less frequently encountered in the datasets used by the LLMs during training. Another limitation is the potential for evaluator bias. Using an external evaluator like GPT-4o helps mitigate some of this bias, but it is still possible that the evaluator’s assessments could be influenced by its own biases or limitations [ 47 ]. To address this, future research should explore the use of multiple evaluators or human experts to provide a more comprehensive and unbiased evaluation of the LLMs’ performance. The generalizability of the findings is also a concern. The datasets used in this study focus on cardiac cases with specific years and languages. This limits the applicability of the results to a broader range of clinical scenarios. Future research should aim to include a more diverse set of clinical cases and evaluate the LLMs’ performance across different languages and healthcare systems. 7.5. Research Advice and Future Directions To address the limitations identified in this study and further enhance the potential of LLMs in healthcare applications, future research should focus on improving interpretability and conducting future evaluations. 7.5.1. Interpretability The interpretability of LLMs is critical for medical applications. Understanding the reasoning behind the diagnoses generated by LLMs is essential for building trust and ensuring accountability. One notable benefit of the RAG technique is that it can serve as a tool for transparency by indicating which parts of the input documents are used as context by the LLM [ 48 ]. This capability enhances interpretability by allowing users to trace the source of the information used in generating the diagnosis. While this study did not explicitly evaluate the quality of the explanations provided by the LLMs, it is important to acknowledge the growing body of work emphasizing the need for interpretability in LLMs, especially in healthcare. Recent works such as Cohen-Wang et al. [49] and research by Anthropic [ 50 , 51 ] on the explainability of LLMs highlight the importance of interpretability. These studies demonstrate the potential of attributing model generations to specific contexts and providing detailed rationales for the generated outputs. This presents a novel avenue for future research, where the focus Appl. Sci. 2025,15, 61 19 of 31 could be on developing and assessing methods to enhance the interpretability of LLMs in healthcare applications. Evaluating the quality and clarity of the explanations provided by LLMs depending on the prompt techniques, temperature levels, and model architectures could significantly improve their utility and reliability in clinical settings. 7.5.2. Future Evaluations To further advance the field, future evaluations should consider incorporating a more diverse set of clinical cases from different years, languages, and healthcare systems to evaluate the LLMs’ performance in a broader range of scenarios. This diversity will help ensure that the models are robust and generalizable to various clinical contexts. Moreover, using multiple evaluators or human experts to provide a more comprehensive and unbiased evaluation of the LLMs’ performance can help mitigate evaluator bias and provide a more robust assessment of the models’ capabilities. Next, establishing robust evaluation frameworks that can assess the performance of LLMs in real-world clinical settings and ensure continuous improvement will help in understanding the long-term reliability and adaptability of the models in dynamic healthcare environments. Finally, exploring other areas of healthcare, such as patient education, clinical decision support, and healthcare administration, can uncover new applications and benefits of LLMs in enhancing overall healthcare delivery and patient outcomes. By addressing these limitations and exploring these future directions, researchers can further enhance the potential of LLMs in healthcare applications and ensure their reliable and effective use in clinical settings. 8. Conclusions This study provides valuable insights into the accuracy and reliability of LLMs in predicting medical diagnoses based on Spanish textual descriptions of clinical cases. The findings highlight the potential of LLMs in healthcare applications but also underscore the need for careful selection of models, prompt techniques, and temperature settings to optimize diagnostic performance. Across all models and prompt techniques, low-temperature settings consistently produce higher-quality diagnostics that are clear, precise, and supported by strong clinical evidence, making them ideal for routine clinical use. High temperatures, while introducing more creative and speculative diagnostic possibilities, often compromise precision, clarity, and focus. The choice between low and high temperatures ultimately depends on the clinical context: low temperatures are best suited for focused, evidence-based diagnostics, while high temperatures are more appropriate for exploratory tasks requiring broader diagnostic consideration. The balance between these factors varies slightly by model, but the overall trends remain consistent across Gemini Pro, Mixtral 8x22b, and Medichat. The choice of prompt technique significantly impacts the quality of diagnostic outputs, with zero-shot excelling in simplicity and clarity and RAG providing the most in-depth and evidence-based diagnostics. Each technique shows clear advantages and trade-offs, with temperature settings further amplifying these effects. Future research should focus on developing advanced models and evaluation frameworks to further enhance the application of LLMs in healthcare. Additionally, exploring hybrid models, refining prompt techniques, and investigating the integration of additional contextual information could provide further insights into optimizing the use of LLMs in clinical practice. By addressing these areas, we can continue to advance the field and harness the full potential of LLMs in improving patient outcomes and healthcare delivery. Author Contributions: Conceptualization, J.D. and J.C.; methodology, J.D. and J.C.; software, J.D.; validation, J.D. and J.C.; investigation, J.D.; resources, J.C.; data curation, J.D.; writing—original draft Appl. Sci. 2025,15, 61 20 of 31 preparation, J.D.; writing—review and editing, J.C.; supervision, J.C.; project administration, J.C. All authors have read and agreed to the published version of the manuscript. Funding: J.C. work was supported by the Torres Quevedo grant “Asistente Virtual para la mejora del sistema de salud—Virtual Assistant for Better HealthCare-VA4BHC” (ref. PTQ2021-012147). Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Data Availability Statement: The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding authors. Conflicts of Interest: All authors were employed by the company Top Health Tech. All authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. Abbreviations The following abbreviations are used in this manuscript: MDPI Multidisciplinary Digital Publishing Institute DOAJ Directory of open access journals LLM Large Language Model NLP Natural Language Processing ML Machine Learning RAG Retrieval-Augmented Generation Appendix A. Prompt Template In this section, we detail the diverse prompts we used to ask the Large Language Models (LLMs) to generate a task. We first present in Appendix A.1, the prompts used to make the models generate diagnostics based on a clinical case. Then, in Appendix A.2, we outline the prompts used to evaluate the precision, recall, completeness, and quality of the diagnosis generated by a physician and an LLM. Appendix A.1. Diagnosis Prompts In order to generate a diagnosis based on a clinical case, we used two types of prompts. The first one is a system prompt (cf Figure A1), which provides the model with context and instructions to generate a diagnosis. The second one is a diagnostic prompt (Figure A2), which contains the specific clinical case and asks the model to generate a diagnosis based on it. Appl. Sci. 2025, 15, x FOR PEER REVIEW 21 of 31 RAG Retrieval-Augmented Generation Appendix A. Prompt Template In this section, we detail the diverse prompts we used to ask the Large Language Models (LLMs) to generate a task. We first present in Appendix A.1, the prompts used to make the models generate diagnostics based on a clinical case. Then, in Appendix A.2, we outline the prompts used to evaluate the precision, recall, completeness, and quality of the diagnosis generated by a physician and an LLM. Appendix A.1. Diagnosis Prompts In order to generate a diagnosis based on a clinical case, we used two types of prompts. The first one is a system prompt (cf Figure A1), which provides the model with context and instructions to generate a diagnosis. The second one is a diagnostic prompt (Figure A2), which contains the specific clinical case and asks the model to generate a diagnosis based on it. Figure A1. This prompt template is used to instruct the LLM to generate possible diagnoses based on patient information. For the system prompt, we tested several variations to determine which one would yield the best results. We found that providing the model with a brief explanation of the task helped improve the model’s performance. We also experimented with different levels of detail in the instructions, ranging from very specific to more general. Figure A2. This prompt template is used to ask the LLM to list the most probable diagnoses based on a given clinical case and explain why each diagnosis is a possibility. Text in red is the emotional stimuli added, while text in blue might be replaced depending on the parameters evaluated (zeroshot, few-shot, RAG). For the diagnostic prompt, we used a template-based approach, where we filled in the details of the clinical case into a predefined template. This allowed us to ensure consistency across different prompts and reduce the risk of introducing bias. We also tested different variations of the template, such as changing the order of the symptoms or using different phrasing, to see how these changes would affect the model’s performance. Text in red is the emotional stimulus we have added to the prompt diagnosis, in line with Li et al. [52]. We replaced the <examples> with examples of how to evaluate the metric, depending on whether we were using zero-shot, few-shot, or Retrieval-Augmented Generation (RAG) approaches. We repeated this process for each metric. Figure A1. This prompt template is used to instruct the LLM to generate possible diagnoses based on patient information. For the system prompt, we tested several variations to determine which one would yield the best results. We found that providing the model with a brief explanation of the task helped improve the model’s performance. We also experimented with different levels of detail in the instructions, ranging from very specific to more general. Appl. Sci. 2025,15, 61 21 of 31 Appl. Sci. 2025, 15, x FOR PEER REVIEW 21 of 31 RAG Retrieval-Augmented Generation Appendix A. Prompt Template In this section, we detail the diverse prompts we used to ask the Large Language Models (LLMs) to generate a task. We first present in Appendix A.1, the prompts used to make the models generate diagnostics based on a clinical case. Then, in Appendix A.2, we outline the prompts used to evaluate the precision, recall, completeness, and quality of the diagnosis generated by a physician and an LLM. Appendix A.1. Diagnosis Prompts In order to generate a diagnosis based on a clinical case, we used two types of prompts. The first one is a system prompt (cf Figure A1), which provides the model with context and instructions to generate a diagnosis. The second one is a diagnostic prompt (Figure A2), which contains the specific clinical case and asks the model to generate a diagnosis based on it. Figure A1. This prompt template is used to instruct the LLM to generate possible diagnoses based on patient information. For the system prompt, we tested several variations to determine which one would yield the best results. We found that providing the model with a brief explanation of the task helped improve the model’s performance. We also experimented with different levels of detail in the instructions, ranging from very specific to more general. Figure A2. This prompt template is used to ask the LLM to list the most probable diagnoses based on a given clinical case and explain why each diagnosis is a possibility. Text in red is the emotional stimuli added, while text in blue might be replaced depending on the parameters evaluated (zeroshot, few-shot, RAG). For the diagnostic prompt, we used a template-based approach, where we filled in the details of the clinical case into a predefined template. This allowed us to ensure consistency across different prompts and reduce the risk of introducing bias. We also tested different variations of the template, such as changing the order of the symptoms or using different phrasing, to see how these changes would affect the model’s performance. Text in red is the emotional stimulus we have added to the prompt diagnosis, in line with Li et al. [52]. We replaced the <examples> with examples of how to evaluate the metric, depending on whether we were using zero-shot, few-shot, or Retrieval-Augmented Generation (RAG) approaches. We repeated this process for each metric. Figure A2. This prompt template is used to ask the LLM to list the most probable diagnoses based on a given clinical case and explain why each diagnosis is a possibility. Text in red is the emotional stimuli added, while text in blue might be replaced depending on the parameters evaluated (zero-shot, few-shot, RAG). For the diagnostic prompt, we used a template-based approach, where we filled in the details of the clinical case into a predefined template. This allowed us to ensure consistency across different prompts and reduce the risk of introducing bias. We also tested different variations of the template, such as changing the order of the symptoms or using different phrasing, to see how these changes would affect the model’s performance. Text in red is the emotional stimulus we have added to the prompt diagnosis, in line with Li et al. [ 52 ]. We replaced the <examples> with examples of how to evaluate the metric, depending on whether we were using zero-shot, few-shot, or Retrieval-Augmented Generation (RAG) approaches. We repeated this process for each metric. Appendix A.2. Evaluator Prompts Appl. Sci. 2025, 15, x FOR PEER REVIEW 22 of 31 Appendix A.2. Evaluator Prompts Figure A3. A scale from 1 to 5 was used to evaluate the precision of the predicted diagnoses compared to the true diagnosis. For precision, we asked the models to compare the list of generated diagnoses to the ground truth diagnosis to determine how many of the model’s diagnoses were correct. Figure A4. This prompt is used to evaluate the recall of the predicted diagnoses compared to the true diagnosis on a scale from 1 to 5. For recall, we asked the models to compare the list of generated diagnoses to the ground truth to determine how many of the relevant symptoms and findings were identified by the model. Figure A5. This scale is used to help an LLM evaluate the completeness of diagnoses predicted by a model compared to the true diagnosis. For completeness, we asked the models to evaluate the completeness of the diagnosis compared to the ground truth. For quality, we asked the models to provide a qualitative assessment of the diagnosis by the other models, including its strengths and weaknesses, as well as any potential limitations or sources of uncertainty. The model then evaluates the quality of the assessment based on the strengths and weaknesses identified. Overall, our evaluation prompts were designed to provide a comprehensive and nuanced assessment of the models’ performance, taking into account both quantitative and qualitative factors. We refrained from providing f ew - sh ot ex a mp l es in th e prompts, as it has been shown to have no significant impact on performance improvement [17]. By using a standardized set of prompts, we were able to ensure consistency and comparability across different models and clinical cases. Figure A3. A scale from 1 to 5 was used to evaluate the precision of the predicted diagnoses compared to the true diagnosis. For precision, we asked the models to compare the list of generated diagnoses to the ground truth diagnosis to determine how many of the model’s diagnoses were correct. Appl. Sci. 2025, 15, x FOR PEER REVIEW 22 of 31 Appendix A.2. Evaluator Prompts Figure A3. A scale from 1 to 5 was used to evaluate the precision of the predicted diagnoses compared to the true diagnosis. For precision, we asked the models to compare the list of generated diagnoses to the ground truth diagnosis to determine how many of the model’s diagnoses were correct. Figure A4. This prompt is used to evaluate the recall of the predicted diagnoses compared to the true diagnosis on a scale from 1 to 5. For recall, we asked the models to compare the list of generated diagnoses to the ground truth to determine how many of the relevant symptoms and findings were identified by the model. Figure A5. This scale is used to help an LLM evaluate the completeness of diagnoses predicted by a model compared to the true diagnosis. For completeness, we asked the models to evaluate the completeness of the diagnosis compared to the ground truth. For quality, we asked the models to provide a qualitative assessment of the diagnosis by the other models, including its strengths and weaknesses, as well as any potential limitations or sources of uncertainty. The model then evaluates the quality of the assessment based on the strengths and weaknesses identified. Overall, our evaluation prompts were designed to provide a comprehensive and nuanced assessment of the models’ performance, taking into account both quantitative and qualitative factors. We refrained from providing f ew - sh ot ex a mp l es in th e prompts, as it has been shown to have no significant impact on performance improvement [17]. By using a standardized set of prompts, we were able to ensure consistency and comparability across different models and clinical cases. Figure A4. This prompt is used to evaluate the recall of the predicted diagnoses compared to the true diagnosis on a scale from 1 to 5. For recall, we asked the models to compare the list of generated diagnoses to the ground truth to determine how many of the relevant symptoms and findings were identified by the model. For completeness, we asked the models to evaluate the completeness of the diagnosis compared to the ground truth. For quality, we asked the models to provide a qualitative assessment of the diagnosis by the other models, including its strengths and weaknesses, as well as any potential Appl. Sci. 2025,15, 61 22 of 31 limitations or sources of uncertainty. The model then evaluates the quality of the assessment based on the strengths and weaknesses identified. Appl. Sci. 2025, 15, x FOR PEER REVIEW 22 of 31 Appendix A.2. Evaluator Prompts Figure A3. A scale from 1 to 5 was used to evaluate the precision of the predicted diagnoses compared to the true diagnosis. For precision, we asked the models to compare the list of generated diagnoses to the ground truth diagnosis to determine how many of the model’s diagnoses were correct. Figure A4. This prompt is used to evaluate the recall of the predicted diagnoses compared to the true diagnosis on a scale from 1 to 5. For recall, we asked the models to compare the list of generated diagnoses to the ground truth to determine how many of the relevant symptoms and findings were identified by the model. Figure A5. This scale is used to help an LLM evaluate the completeness of diagnoses predicted by a model compared to the true diagnosis. For completeness, we asked the models to evaluate the completeness of the diagnosis compared to the ground truth. For quality, we asked the models to provide a qualitative assessment of the diagnosis by the other models, including its strengths and weaknesses, as well as any potential limitations or sources of uncertainty. The model then evaluates the quality of the assessment based on the strengths and weaknesses identified. Overall, our evaluation prompts were designed to provide a comprehensive and nuanced assessment of the models’ performance, taking into account both quantitative and qualitative factors. We refrained from providing f ew - sh ot ex a mp l es in th e prompts, as it has been shown to have no significant impact on performance improvement [17]. By using a standardized set of prompts, we were able to ensure consistency and comparability across different models and clinical cases. Figure A5. This scale is used to help an LLM evaluate the completeness of diagnoses predicted by a model compared to the true diagnosis. Overall, our evaluation prompts were designed to provide a comprehensive and nuanced assessment of the models’ performance, taking into account both quantitative and qualitative factors. We refrained from providing few-shot examples in the prompts, as it has been shown to have no significant impact on performance improvement [ 17 ]. By using a standardized set of prompts, we were able to ensure consistency and comparability across different models and clinical cases. Appendix A.3. Few-Shot Examples In this section, we present the few-shot examples selected for our study and justify their inclusion. These examples were chosen to provide a comprehensive and representative set of rare cardiovascular diseases, ensuring the robustness and reliability of our findings. 1. Chronic ischemic heart disease. Old anterior infarction, a single-vessel disease with complete revascularization. Severe systolic dysfunction of the left ventricle. No current angina. 2. Complete atrioventricular block paroxysmal. Definitive DDDR pacemaker implant. Post-procedure Takotsubo syndrome. Psychological stress due to unforeseen hospital admission and administration of isoproterenol as possible inducers. Transient moderate systolic dysfunction of the left ventricle at the expense of akinesia of the middle and apical segments. Coronary arteries without significant lesions. 3. Neuroendocrine tumor of intermediate grade stage IV. Carcinoid heart disease. Cardiac insufficiency. Critical tricuspid stenosis. Severe tricuspid insufficiency. Severe pulmonary stenosis. Moderate pulmonary insufficiency. Surgical replacement of tricuspid and pulmonary valves. Complicated postoperative period with carcinoid crisis. Death. 4. Marfan syndrome: carrier of a pathogenic mutation in FBN1. Aneurysm of the ascending aorta and severe aortic insufficiency intervened. David’s procedure and aortic tube. Dilation of the left ventricle and secondary systolic dysfunction, severe at discharge, currently moderate (LVEF 44%) with NYHA class I. The selection process for these examples involved several key steps to ensure their representativeness and relevance to our study objectives. First, we consulted with a panel of medical experts to identify a diverse set of rare cardiovascular diseases that are representative of the broader spectrum of such conditions. The experts provided insights into the clinical significance, prevalence, and diagnostic challenges associated with each disease. Next, the examples were selected based on specific criteria aimed at ensuring diversity, complexity, and relevance. The selected cases cover a range of rare cardiovascular diseases, including ischemic heart disease, stress-induced cardiomyopathy, carcinoid heart disease, and genetic syndromes like Marfan syndrome. These cases exhibit complex clinical presentations and diagnostic challenges, ensuring that the LLMs are tested on a variety of difficult Appl. Sci. 2025,15, 61 23 of 31 scenarios. Additionally, the examples are relevant to the study’s objectives of evaluating the LLMs’ ability to handle rare and complex cardiovascular diseases.\ By following this rigorous selection process, we ensured that the few-shot examples are representative of the diversity and complexity of rare cardiovascular diseases. This approach enables a comprehensive evaluation of the LLMs’ ability to handle such cases, providing robust and reliable insights into their potential to assist medical professionals. Appendix A.4. Retrieval-Augmented Generation In this section, we justify the specific choices made for the Retrieval-Augmented Generation (RAG) technique used in our study. These choices were based on scientific reasoning and aimed to optimize the performance of the LLMs in generating accurate and contextually relevant diagnoses for rare cardiovascular diseases. Appendix A.4.1. Retrieval Model We selected Chroma [https://docs.trychroma.com/] [ 53 ], accessed on 20 July 2024, as our retrieval model due to its advanced capabilities in handling large-scale data retrieval tasks. Chroma is designed to efficiently index and search through vast amounts of data, making it well-suited for retrieving relevant medical information from a large corpus. Its ability to handle high-dimensional embeddings and perform similarity searches ensures that the retrieved information is both accurate and relevant to the clinical case at hand. Appendix A.4.2. Model Embedding For the embedding model, we chose the text-embedding-3-large model from OpenAI [ 54 ]. This model was selected for its ability to capture the nuanced semantics of the medical language, ensuring that the embeddings generated are highly representative of the underlying clinical information. The text-embedding-3-large model has been pre-trained on a diverse range of texts, making it robust and capable of handling the complexities of medical terminology. The use of a pre-trained model also helps to mitigate the risk of overfitting and improves the generalizability of the retrieval process. Appendix A.4.3. Chunk Size Given the complexity and interconnected nature of medical information, we decided to cut the retrieval by clinical case rather than within each case. This approach preserves the entire context of each case, ensuring that the LLM has access to all relevant information for a given patient. This is particularly important for rare cardiovascular diseases, where multiple pieces of information are often interconnected. Appendix A.4.4. Implementation Details In our implementation, we used the Chroma retrieval model to index a large corpus of medical data from the 2018 dataset. The embedding model was used to generate highdimensional embeddings for each clinical case and corresponding diagnoses in the corpus. The retrieval process involved performing similarity searches to identify the most relevant chunks of information for a given clinical case from the 2020 dataset. The LLM then used the retrieved information to generate comprehensive and contextually relevant diagnoses. Appendix B. Diagnoses Examples In this section, we provide examples of the diagnoses generated by the three studied models, along with the ground truth diagnosis generated by a healthcare expert. Each subsection presents the diagnoses and explanations provided by different models, allowing for a comparative analysis of their outputs. Appl. Sci. 2025,15, 61 24 of 31 Appendix B.1. Ground Truth Diagnosis The ground truth diagnosis is provided by healthcare experts based on their clinical evaluation and diagnostic tests. This serves as the benchmark against which the modelgenerated diagnoses are compared. • MINOCA (infarto agudo de miocardio sin enfermedad coronaria aterosclerótica obstructiva) en relación con miocarditis aguda con fibrosis y edema extensos. • Función sistólica del ventrículo izquierdo ligeramente deprimida (FEVI 47%) secundaria a lo anterior. •Fibrilación ventricular idiopática: Red coronaria angiográficamente normal. •No cardiopatía estructural. •Buena función biventricular. Appendix B.2. Gemini The Gemini model provides the following diagnoses and explanations based on the patient’s clinical data and medical history. Appendix B.2.1. Diagnósticos Más Probables 1. Miocarditis aguda inducida por fármacos (especialmente lenalidomida): Este es el diagnóstico más probable dado el inicio reciente de lenalidomida, un fármaco conocido por sus posibles efectos secundarios cardíacos. La presentación clínica de la paciente, incluyendo dolor torácico, malestar general, marcadores cardíacos elevados y anomalías en las imágenes (miocarditis con fibrosis y edema), respalda fuertemente este diagnóstico. 2. Miocarditis viral: Aunque menos probable debido al inicio del tratamiento con lenalidomida, la miocarditis viral sigue siendo una posibilidad. Las infecciones virales son una causa común de miocarditis y pueden presentarse con síntomas similares a los de la paciente. 3. Progresión del mieloma múltiple: Si bien la miocarditis es una posibilidad, la progresión del mieloma múltiple en símisma podría contribuir a los síntomas de la paciente. El mieloma múltiple puede causar daño a órganos, incluido el corazón, y podría explicar los marcadores cardíacos elevados. Sin embargo, la resonancia magnética que muestra miocarditis hace que esta posibilidad sea menos probable. Appendix B.2.2. Explicación de los Diagnósticos • Miocarditis inducida por fármacos: La lenalidomida, al igual que otros agentes quimioterapéuticos, puede causar toxicidad cardíaca, incluida la miocarditis. La paciente comenzóa tomar lenalidomida recientemente, y la aparición de sus síntomas coincide con el inicio del tratamiento. Además, la resonancia magnética muestra signos claros de miocarditis, lo que respalda aún más este diagnóstico. • Miocarditis viral: Las infecciones virales son una causa común de miocarditis y pueden causar síntomas similares a los experimentados por la paciente. Sin embargo, la falta de antecedentes recientes de infección viral y el inicio del tratamiento con lenalidomida hacen que este diagnóstico sea menos probable. • Progresión del mieloma múltiple: El mieloma múltiple puede causar daño a varios órganos, incluido el corazón. La progresión de la enfermedad podría explicar los marcadores cardíacos elevados y algunos de los síntomas de la paciente. Sin embargo, la resonancia magnética que muestra miocarditis sugiere que la miocarditis, y no la progresión directa del mieloma, es la causa principal de sus síntomas. Appl. Sci. 2025,15, 61 25 of 31 Appendix B.3. Medichat The Medichat model provides a detailed clinical summary and treatment plan based on the patient’s medical history and current condition: El paciente es una mujer de 69 años con antecedentes de enfermedad mixta (MM) en etapa IIIA según el sistema de ISS. Presentan síntomas compatibles con infarto agudo de miocardio, sin elevación del ST segmento en la ECG. La resonancia magnética cardiaca revelóuna miocarditis aguda con edema y fibrosis extensas, asícomo un deterioro leve de la función ventricular izquierda. El paciente actualmente estáen tratamiento con lenalidomida, pero debido a la escasa literatura disponible sobre la cardiotoxicidad inducida por este fármaco, se decidió suspender el ciclo hasta una revisión posterior en consultas de hematología. Durante el ingreso, se inicióun tratamiento ardioprotector limitado debido a la hipotensión, pero al alta se derivóa consultas de cardio-onco-hematología para seguimiento y titulación del tratamiento cardioprotector. En el último control ecocardiográfico realizado un mes después del ingreso, la función sistólica del ventrículo izquierdo se había normalizado. Sin embargo, la literatura disponible sugiere que la lenalidomida puede causar cardiotoxicidad, aunque no se pudo establecer una relación clara entre el tratamiento y el cuadro de la paciente. En resumen, el paciente tiene miocarditis aguda con edema y fibrosis extensas, así como un deterioro leve de la función ventricular izquierda. Además, se encuentra en tratamiento con lenalidomida, aunque la cardiotoxicidad inducida por este fármaco no se ha establecido de manera clara en la literatura. Se recomienda una revisión posterior en consultas de hematología para evaluar el tratamiento y monitorear la función cardíaca de la paciente. Appendix B.4. Mixtral The Mixtral model provides the following diagnoses and explanations based on the patient’s clinical data and medical history. Appendix B.4.1. Diagnósticos Más Probables, en Orden de Probabilidad 1. Miocarditis aguda secundaria a cardiotoxicidad por lenalidomida en el contexto de un cuadro de enfermedad activa por mieloma múltiple. 2. Cardiopatía isquémica con síndrome coronario agudo sin elevación del ST (SCASEST). 3. Otras posibles causas de miocarditis, como infección o inflamación sistémica. Appendix B.4.2. Explicación De Los Posibles Diagnósticos 1. Miocarditis aguda secundaria a cardiotoxicidad por lenalidomida en el contexto de un cuadro de enfermedad activa por mieloma múltiple: Debido a que la paciente tiene un antecedente de cardiotoxicidad por bortezomib y estáactualmente en tratamiento con lenalidomida, también asociada con toxicidad cardiaca, es probable que el cuadro clínico actual se deba a miocarditis aguda inducida por esta terapia. La presencia de hipotensión arterial en este paciente indica un efecto real del agente citotóxico en la función cardíaca y es compatible con la hipotensión inducida por medicamentos. 2. Cardiopatía isquémica con SCASEST: Aunque la paciente presenta dolor torácico e incremento significativo de marcadores de daño miocárdico como la creatin-kinasa y la troponina-T, la ausencia de lesiones coronarias obstructivas sugiere una causa primaria no isquémica para el dolor torácico de esta paciente. Además, los parámetros de movilidad segmentaria en los estudios de imágenes (ecocardiografía y resonancia magnética) indican una disfunción sistólica de grado leve limitada a ciertas regiones del miocardio, lo cual desfavorece una patología aterosclerótica aguda. La normal-