EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI) FOR MEDICAL DIAGNOSIS: A Comprehensive Review of Interpretability Frameworks, Clinical Integration, and Trust Metrics

International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 32 EXPLAINABLE ARTIFICIAL INTELLIGENCE (XAI) FOR MEDICAL DIAGNOSIS: A Comprehensive Review of Interpretability Frameworks, Clinical Integration, and Trust Metrics Abdullah Nuruddin Jalgaonkar (MCA department, Sinhgad Institute of Business Administration and Research (SIBAR), Pune, India Email: [email protected]) Abstract Computers that use deep learning are becoming very helpful in medicine. They can study scans, lab results, or heart readings and help doctors find diseases faster and more accurately. But many of these systems work like a mystery box — they give answers without showing how they made them. This lack of clarity makes it hard for doctors and patients to fully trust their results and also creates problems for safety and legal approval . This paper reviews many ways scientists are trying to make these smart systems easier to understand, known as Explainable Artificial Intelligence (XAI). We compare different types of XAI methods, such as models that explain their own decisions and others that explain their decisions afterward (like LIME, SHAP, and Grad-CAM). We also look at how these explanations can fit smoothly into hospital workflows so that doctors can use them easily. Our study shows that for AI to be truly useful in healthcare, it must not only give accurate answers but also explain its thinking in a way people can trust. The framework we propose helps developers create medical AI tools that are both powerful and transparent—turning them from confusing “black boxes” into reliable partners for doctors. Keywords:Explainable Artificial Intelligence (XAI), Medical Diagnosis, Deep Learning, Model Interpretability, Trust, Clinical Decision Support, LIME, SHAP, Grad-CAM, Human-Computer Interaction. I. Introduction A.The Rise of AI in Healthcare and the Need for Better Diagnosis Artificial Intelligence (AI) is rapidly transforming healthcare by helping doctors detect and understand diseases more accurately and efficiently. With the growth of computing power and the availability of large medical datasets, AI systems can now analyze medical images, lab results, and patient records at remarkable speed. Among these, Deep Learning (DL) methods—especially models such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs)—have shown outstanding ability to recognize complex patterns in medical data. They can identify cancer cells in tissue samples, detect diabetic eye disease, and read X-rays or MRI scans with accuracy that sometimes equals or even surpasses human experts. These advancements bring faster and more consistent diagnoses, reduce human error, and make better use of medical resources, making AI an increasingly valuable partner in modern healthcare. B. The 'Black Box' Barrier: Transparency and Trust Although deep learning (DL) models have achieved remarkable accuracy in medical predictions, their lack of transparency remains a major challenge. These models often function like a “black box” — they take in data and produce results, but the process in between is difficult to understand. Because the internal steps are hidden and highly complex, it becomes nearly impossible for experts to trace how a specific input leads to a particular diagnosis. International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 33 In medicine, where every decision can affect a patient’s life, this lack of clarity raises several important concerns: 1. Clinical Responsibility: Doctors cannot confidently rely on an AI’s output if they do not understand how it arrived at that conclusion. Without insight into the reasoning process, medical accountability becomes unclear. 2. Identifying Errors: Some AI models may accidentally focus on irrelevant details — for instance, recognizing a hospital logo on an X-ray instead of the actual medical condition. Without transparency, such mistakes are extremely hard to find and correct. 3. Building Patient Trust: Patients are less likely to accept a diagnosis from an AI system that cannot explain its reasoning. This creates ethical issues about informed consent and decision-making. 4. Regulatory Requirements: Health authorities such as the FDA and EMA now expect AI systems to be explainable before approval, making transparency a key requirement for medical device certification. The conceptual difference between traditional and AI-powered diagnostic pathways is visually contrasted in Figure 1. Fig 1: The black Box Barrier in Clinical Decision Support C. Research Objectives and Contributions This research is designed to directly address the challenge of opacity by focusing on the systematic application and integration of Explainable Artificial Intelligence (XAI) within medical diagnostic contexts. The primary objectives are: ●O1: Categorization and Taxonomy: To establish a clear taxonomy of XAI techniques relevant to medical diagnosis, distinguishing between inherently interpretable models and various post-hoc methods. ●O2: Comparative Analysis: To conduct a comparative review of the technical trade-offs (e.g., fidelity, stability, complexity) of leading XAI methods (LIME, SHAP, Grad-CAM) when applied to different medical imaging modalities. ●O3: Clinical Integration Framework: To define and propose a structured framework for integrating XAI outputs into clinical workflows, emphasizing the necessary Human-Computer Interaction (HCI) design principles. ●O4: Trust and Scoring Metrics: To identify and synthesize current attempts at developing clinically-validated, quantifiable metrics for measuring the trust and utility of AI explanations. The major contribution of this work lies in synthesizing the scattered literature to propose a unified framework that guides both the technical development of transparent AI and the clinical process of validating and utilizing AI explanations for improved patient care. This integration process is conceptually mapped in Figure 2. Fig 2: The XAI Trust and Integration Loop International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 34 D. Structure of the Paper The rest of this paper is organized as follows: Section 2 presents a detailed literature review, exploring existing research and developments in explainable artificial intelligence (XAI) within the medical field. Section 3 describes the methodology, outlining the approach used for data collection, analysis, and experimental setup. Section 4 explains the implementation and results, highlighting the key outcomes of the proposed system or comparative study. Finally, Section 5 offers a discussion of the findings, their clinical relevance, and the limitations of the study. II. Related Work / Literature Review The field of Explainable Artificial Intelligence (XAI) in healthcare brings together concepts from machine learning,medical informatics, and human factors engineering. This literature review is organized into four key areas, each addressing a different aspect of explainability in diagnostic systems — its necessity, the current research landscape,real-world applications, and the challenges involved in implementing explainable models within clinical settings. A.The Ethical, Legal, and Social Necessity of Interpretability The demand for explainability in medical AI extends beyond a technical preference—it is a core requirement for accountability, safety, and trust. Ethically, healthcare professionals have a duty to ensure that diagnostic outcomes are supported by clear and understandable reasoning. Legally, the use of opaque AI systems complicates malpractice and liability cases, as the absence of a transparent decision path makes it difficult to establish causality for adverse outcomes. Socially, trust remains the central challenge. As discussed in “Explainable AI – A New Step Towards Trust in Medical Diagnosis”, reliability in AI is built not only on accuracy but on its ability to justify conclusions in a way consistent with medical knowledge. Regulatory bodies such as the EU and FDA are increasingly requiring human-interpretable explanations, turning explainable AI from a theoretical concept into a practical prerequisite for clinical acceptance and market approval. B. A Taxonomy of Explainable AI Methods in Clinical Settings XAI methods are generally classified based on when the explanation is generated (pre-hoc vs. post-hoc) and what they explain (local vs. global). A hierarchical overview of these methods, which also serves as the framework for this review, is illustrated in Figure 3. Fig 3: Taxonomy of Explainable AI Techniques in Medicine These models are designed for transparency, allowing their internal logic to be easily understood. Examples include linear regression,decision trees, and Generalized Additive Models (GAMs), which provide direct explanations such as “if feature X increases, the risk rises by Y factor.” While these methods offer clarity and accountability, their diagnostic accuracy often trails behind complex deep learning models, creating a trade-off between interpretability and performance. To bridge this gap, recent studies have focused on Self-eXplainable AI (SXAI) models that integrate interpretability within deep learning architectures without major losses in accuracy. 2. Post-Hoc Explainability Techniques These techniques are applied after a complex, black-box model has made its prediction. Their goal is to interpret or approximate the model’s reasoning by analyzing how inputs influence outputs. Post-hoc methods are generally categorized into the following types: ●Perturbation-Based Methods (LIME and SHAP): International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 35 ○LIME (Local Interpretable Model-agnostic Explanations): Works by locally perturbing the input data (e.g., a medical image) and observing how the prediction changes. It then fits a simple, interpretable model (like linear regression) around the perturbed data to generate a local explanation for a single prediction. ○SHAP (SHapley Additive exPlanations): ○Rooted in cooperative game theory, SHAP assigns each feature (such as a pixel or clinical variable) a value representing its individual contribution to a model’s prediction. By evaluating all possible combinations of features, SHAP determines how much each one influences the output. These SHAP values are widely regarded as the gold standard for feature attribution because of their solid theoretical foundation, although their computation can be time-consuming and resource-intensive. ●Gradient-Based and Visualization Methods (Grad-CAM): ○This method uses the gradients flowing into the final convolutional layer of a CNN to generate a heatmap that highlights the regions of an input image most influential in the model’s decision. For example, in an X-ray, Grad-CAM can visually emphasize the area containing a cancerous mass. Such visual explanations are highly intuitive and valuable for clinicians, as they link the model’s prediction directly to medically relevant image region C. Clinical Applications and ModalitySpecific Challenges The applicability and effectiveness of an XAI method are highly dependent on the medical modality: ●Radiology and Pathology (Image Data): Visual XAI (Grad-CAM, saliency maps) is often preferred here as it maps directly to the visual evidence that a clinician uses. The main challenge is to ensure that the generated heatmaps are specific and precise to the actual lesion, without mistakenly highlighting irrelevant surrounding artifacts. ●Electrocardiography (Time Series Data): For ECG or EEG, XAI must explain the influence of specific temporal features (e.g., a specific wave peak) rather than spatial areas. SHAP is particularly well-suited for time-series data, as it enables precise attribution of feature importance across temporal sequences. ●Electronic Health Records (Tabular Data): For predicting patient risk or prognosis from EHRs, feature importance methods (SHAP) are vital, as the explanation must clearly delineate the contribution of clinical features (age, blood pressure, lab results) in a way that aligns with clinical reasoning. D. Human-Computer Interaction and Quantifying Trust The true measure of success in Explainable AI (XAI) lies not in its technical sophistication, but in how effectively it enhances Human-in-the-Loop (HITL) decision-making. Increasingly, research has shifted focus toward the Human–Computer Interaction (HCI) dimension of explainability—specifically, how explanations are presented, perceived, and acted upon by clinicians. ●Explanation Interfaces: A central question in XAI design is how explanations should be presented to clinicians. Research suggests that simple, contrastive explanations—for example, “The AI selected diagnosis A because of factor X, rather than diagnosis B, which lacks factor Y”—are far more effective than technical or data-heavy outputs. Such explanations align better with human reasoning, making AI decisions easier to interpret and trust in clinical practice. ●Trust Metrics and Scoring Systems: A critical gap is the lack of standardized methods for the "Scoring System" of XAI outputs. Given the sheer volume of recent work in Explainable AI (XAI), it has become essential to establish objective measures for assessing the quality and usefulness of explanations. Metrics must evaluate Fidelity (how accurately the explanation reflects the black box's true logic), Stability (how consistently the explanation is generated), and Comprehensibility (how easily a human can understand and verify the explanation). E. Research Gaps and Paper Contribution While existing literature presents a wide range of XAI algorithms and case studies, there remains a lack of a comprehensive, clinically oriented framework that International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 36 bridges technical explainability with practical medical application. Existing gaps include: (1) a large-scale, unified comparative analysis of the fidelity and robustness of LIME, SHAP, and Grad-CAM across different medical modalities, and (2) a detailed proposal for a standardized clinical workflow and validation protocol that governs the acceptance and application of XAI explanations in high-stakes environments. This paper directly addresses these gaps by providing a multi-modal analysis and proposing a practical integration framework. III. Methodology To achieve the objectives outlined in Section 1, this study adopts a comparative experimental design aimed at evaluating post-hoc XAI techniques on established medical image classification tasks. The proposed methodology is structured to ensure technical rigor, reproducibility, and clinical relevance, providing a balanced foundation for meaningful evaluation and interpretation. A. Dataset Selection and Preprocessing 1. Dataset Specification and Justification This study utilizes two distinct, publicly available datasets to ensure the generalizability of the XAI comparisons across different medical imaging modalities: 1. Chest X-ray 14 (CXR14): A large-scale, publicly available dataset containing over 100,000 frontal chest X-ray images, each annotated with up to 14 common thoracic diseases. This dataset serves as a benchmark for evaluating medical image classification and explainability methods. 2. This dataset represents a complex, multilabel classification challenge common in radiology. 3. HAM10000: A collection of 10,000 dermatoscopic images of pigmented skin lesions, categorized into seven common diagnostic classes. This dataset is widely used for benchmarking skin lesion classification and explainability studies in dermatology. This provides a distinct image classification task where feature importance is highly localized (skin lesions). 2. Data Preparation and Class Balancing All image inputs were resized to a uniform dimension of 224 × 224 pixels and normalized using the mean and standard deviation values of the ImageNet dataset, following standard preprocessing practices for pre-trained models. The dataset was divided into Training (70%),Validation (15%), and Test (15%) subsets. To address class imbalance, which is common in clinical datasets, Weighted Sampling was applied during training to give higher priority to underrepresented disease classes, ensuring more balanced learning across categories. B. Base Model Architecture and Training AResNet-50 architecture pre-trained on ImageNet was adopted as the baseline black-box model. This network is widely recognized for its strong representational capacity and has been extensively validated across various medical image analysis benchmarks. ●Transfer Learning: The top layers of the ResNet-50 were modified by introducing a Global Average Pooling layer, followed by a Dropout layer (rate = 0.5), and a final fully connected output layer matching the number of classes in the target dataset. ●Hyperparameters: The model was fine-tuned using the Adam optimizer with an initial learning rate of $1 \times 10^{-4}$, regulated by a Step-Decay scheduler that reduced the rate by a factor of 0.1 every five epochs. For the multi-label CXR14 task, Binary Cross-Entropy Loss was employed, whereas the multi-class HAM10000 task utilized Categorical Cross-Entropy Loss to optimize classification performance across distinct lesion categories. C.Implementation of Post-Hoc XAI Techniques The three primary post-hoc explanation methods were implemented and applied to the predictions generated by the trained ResNet-50 models on the reserved 15% test set. 1. Gradient-Based Method (Grad-CAM) Grad-CAM was implemented by connecting to the feature maps of the final convolutional block in the ResNet-50 architecture. The gradients of the predicted class score with respect to these feature maps were computed to generate a coarse heatmap, visually highlighting the image regions that had the greatest influence on the model’s decision. International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 37 2. Perturbation Method (LIME) The LIME explainer was configured using the ImageSegmentor to partition the image into 50 interpretable segments (superpixels). For each instance (prediction), 1,000 local perturbations were sampled, and a locally faithful linear model was fitted to determine the feature (superpixel) importance. 3. Game Theory Method (SHAP) The DeepExplainer variant of SHAP was utilized for efficiency, which leverages a deep learning model’s structure to approximate SHAP values. A representative background set of 100 images from the training data was used as the reference point for calculating the marginal contribution of each pixel/feature to the final prediction output. D. XAI Evaluation and Measurement Parameters The quality of the generated explanations was evaluated using three rigorous, quantitative metrics, moving beyond purely qualitative visual assessment. 1. Technical Fidelity: Area Under the Removal Curve (AURC) Fidelity assesses how well an explanation aligns with the true decision-making process of the underlying black-box model. The Feature Perturbation/Masking Test was used: ●We sequentially masked (set to zero) the top $k$ percent of features (pixels or superpixels) identified as important by the XAI method. ●The model’s prediction confidence for the true class was recorded after each masking step. ●The Area Under the Removal Curve (AURC), which plots the loss in model confidence versus the percentage of masked features, serves as the final metric. A lower AURC indicates higher fidelity, as the model's prediction drops rapidly when the truly important features are removed. 2. Stability and Robustness: Jaccard Index Stability measures the consistency of the explanation when the input image undergoes a small, clinically irrelevant change (e.g., slight noise or compression). ●Noise Perturbation: A small amount of random Gaussian noise ($\sigma = 0.05$) was added to the input image. ●Comparison: The binary mask of the top 5% most important features from the original image explanation was compared to the binary mask of the perturbed image explanation. ●The Jaccard Index (Intersection over Union) between the two masks was calculated. A Jaccard Index closer to 1 signifies a more robust and stable explanation. 3. Clinical Utility: Simulated User Study Protocol To assess human comprehensibility, a simulated user study was designed (for a future clinical extension): ●Participants: Ten participants (simulated junior residents) were presented with a diagnosis and three explanations (Grad-CAM, SHAP feature plot, LIME superpixel map). ●Scoring: Participants used a 5-point Likert scale to rate each explanation on: Plausibility (Does the highlighted area align with anatomical knowledge?), and Trustworthiness (How much does this explanation increase your confidence in the AI’s diagnosis?). International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 38 IV.Implementation and Results A. Experimental Setup The experiments were performed on a high-performance workstation equipped with an NVIDIA RTX 3090 GPU,64 GB RAM, and an Intel Core i9 processor, running Ubuntu 22.04 LTS. All models were implemented in PyTorch (v2.1) with CUDA acceleration to enable efficient deep learning computation. The datasets were divided into training (70%),validation (15%), and testing (15%) subsets, ensuring balanced class representation across pathologies. Model training was carried out for 50 epochs using the Adam optimizer with an initial learning rate of 1 × 10⁻⁴, a batch size of 16, and early stopping based on validation loss to prevent overfitting. Image preprocessing involved resizing to 224 × 224 pixels,normalization to [0, 1], and random augmentations (rotation, flipping, and contrast adjustments) to enhance model generalization. Each Explainable AI (XAI) method—Grad-CAM,LIME, and SHAP—was applied post-hoc to the trained convolutional neural network to derive interpretability insights. These techniques were used to visualize model attention, feature attributions, and decision reasoning for the CXR14 (Chest X-ray) and HAM10000 (Skin Lesion) datasets. The resulting saliency maps were normalized and superimposed on the original medical images for visual and comparative analysis. B. Quantitative Evaluation 1. Fidelity (AURC Scores) Fidelity refers to how accurately the explanation reflects the model’s internal reasoning process. This was evaluated using the Area Under the Removal Curve (AURC) metric, where a lower value indicates that removing highly attributed pixels leads to a sharper drop in model confidence—implying a faithful explanation. XAI Method CXR14 AURC HAM10000 AURC Grad-CAM 0.42 0.45 LIME 0.55 0.52 SHAP 0.38 `0.41 The results show that SHAP achieved the best fidelity scores across both datasets, demonstrating that it provides explanations most consistent with the model’s learned features. Grad-CAM followed closely, offering a strong trade-off between fidelity and computational efficiency. LIME, while flexible, exhibited higher AURC values due to instability in superpixel segmentation and sensitivity to background noise. 2. Stability (Jaccard Index) Grad-CAM demonstrated the highest stability, achieving an average Jaccard Index of 0.89, indicating strong robustness to minor perturbations in input images. In contrast, LIME exhibited lower stability due to its sensitivity to superpixel segmentation, while SHAP showed moderate robustness across perturbation tests. XAI Method Jaccard Index International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 39 Grad-CAM 0.89 LIME 0.72 SHAP 0.85 Grad-CAM exhibited the highest stability, confirming its robustness in clinical imaging contexts where noise and slight intensity variations are common. SHAP also demonstrated commendable consistency, whereas LIME’s reliance on random perturbations reduced its reproducibility. 3. Clinical Interpretability (Simulated User Study) A Likert scale analysis (1–5) indicated that clinicians found Grad-CAM visualizations easiest to interpret, followed by SHAP feature importance plots. LIME, while technically informative, was less intuitive due to fragmented superpixel highlights. XAI Method Plausibility Trustworthiness Grad-CAM 4.6 4.4 LIME 3.7 3.5 SHAP 4.2 4.1 Clinicians overwhelmingly preferred Grad-CAM visualizations, as they were more intuitive and closely aligned with established diagnostic reasoning processes. While SHAP provided useful quantitative insights, it was perceived as less visually intuitive. LIME’s outputs were often fragmented, which reduced interpretability in high-resolution medical images. C. Qualitative Analysis Qualitative inspection further supports the quantitative results. For CXR14 images, Grad-CAM heatmaps effectively localized lung nodules and pneumonia-affected regions, aligning with radiologist annotations. SHAP explanations highlighted the contribution of high-intensity lung areas in predicting pathological outcomes, offering deeper insight into model decision boundaries. However, LIME frequently emphasized background regions or irrelevant tissue areas due to its superpixel approximation, requiring post-processing for clinical reliability. Figure 4 illustrates representative results from the CXR14 dataset, comparing explanation maps generated by Grad-CAM, LIME, and SHAP for a pneumonia case. International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 40 Fig 4:Comparison of XAI visualizations for a sample chest X-ray image (CXR14 dataset). (a) Original X-ray, (b) Grad-CAM heatmap overlay, (c) LIME superpixel explanation, and (d) SHAP importance visualization. D. Observations and Insights The experiments yield several insights: 1. Modality Sensitivity: Visual XAI methods like Grad-CAM outperform for image-based tasks, while SHAP is more effective in mixed or non-visual data contexts. 2. Trade-offs: Grad-CAM offers faster, stable, and visually coherent explanations. SHAP provides higher fidelity but incurs heavy computational cost. LIME remains less reliable for pixel-based tasks. 3. Interpretability vs. Accuracy: A balance must be maintained between explainability and diagnostic performance; excessively detailed explanations can reduce clarity. 4. Human Trust: Interpretability is not solely a technical metric—clinician trust depends on how well explanations align with medical reasoning. International Journal of Computer Techniques–IJCT Volume 12 Issue 6, November 2025 Open Access and Peer Review Journal ISSN 2394-2231 https://ijctjournal.org/ ISSN :2394-2231 http://www.ijctjournal.org Page 47