Explainability and Causality in Machine Learning through Shapley values

BACHELOR’S DEGREE FINAL PROJECT Explainability and Causality in Machine Learning through Shapley values Presented by: Alba Carballo Castro Supervised by: DR. EMILIO CARRIZOSA PRIEGO FACULTY OF MATHEMATICS Statistics and Operational Research Department Sevilla, June 2022 Abstract Explainability and causality are becoming increasingly relevant in Machine Learning research. On the one hand, given the growing use of models in decision-making processes, the way in which they make predictions needs to be more thoroughly understood. On the other hand, a rising interest exists in formalising and introducing the causal relationships present in the real world into those same models. This work addresses both aspects through the use of Shapley values, a concept that is at the origin of SHAP, one of the most popular explainability techniques. Different methods for calculating Shapley values to explain predictions are introduced that take into account the dependence and the causal structure of the data. These methods are illustrated and compared through a series of experiments using a database whose causal structure is known. They show that differences can be observed when taking causality into account. 3 Resumen La explicabilidad y la causalidad son áreas cada vez más relevantes en la investigación en Aprendizaje Automático. Por un lado, dado el creciente uso de los modelos en los procesos de toma de decisión, es necesario comprender mejor la forma en que realizan las predicciones. Por otro lado, existe un creciente interés por formalizar e introducir en esos mismos modelos las relaciones causales presentes en el mundo real. Este trabajo aborda ambos aspectos mediante el uso de los valores de Shapley, concepto que está en el origen de SHAP, una de las técnicas de explicabilidad más populares. Se exponen diferentes métodos de cálculo de valores de Shapley para explicar las predicciones que tienen en cuenta la dependencia y la estructura causal de los datos. Estos métodos se ilustran y comparan mediante una serie de experimentos que utilizan una base de datos cuya estructura causal se conoce. De ellos se pueden observar que existen diferencias cuando se tiene en cuenta la causalidad. 5 A Rosita, Javi y Mariloli para que estéis siempre. 7 Contents 1 Introduction 11 2 Preliminars 13 2.1 Explainable Machine Learning . . . . . . . . . . . . . . . . . . . . . 13 2.2 The problem of causality . . . . . . . . . . . . . . . . . . . . . . . . 16 2.3 Predictionmodels............................ 19 3 SHAP 23 3.1 Shapleyvalues ............................. 23 3.2 SHAP .................................. 25 3.3 SHAPvalues .............................. 26 3.4 Methods for feature dependency . . . . . . . . . . . . . . . . . . . . 27 3.5 Methods for causal structures . . . . . . . . . . . . . . . . . . . . . . 31 3.6 Othermethods.............................. 32 4 Computational implementation 37 4.1 Shapley values computation . . . . . . . . . . . . . . . . . . . . . . 37 4.2 Data preparation, model training and plotting . . . . . . . . . . . . . 39 9 16 2.2. The problem of causality 3. According to how the explanation is made: •Model-agnostic: the method is applicable to different Machine Learning models, since it does not inspect specific parameters. •Model-specific: the method is specifically built for a given Machine Learning model since it uses its internal structure and parameters to provide the explanations. 2.2 The problem of causality At a time when data and the information they provide us with are taking on a central role, some scientists started to wonder if they could help establish cause-effect relationships, allowing to understand or even predict effects. The relevance that this issue has acquired is exemplified by the fact that the Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel (commonly known as the Nobel Prize in Economics) was awarded in 2021 to Joshua Angrist and Guido Imbens for their contributions on the analysis of causality. In Statistics, the field of causal inference is gaining importance, with scientists aiming to formally establish existent and predict future causal relationships. For this purpose, they are using different mathematical tools. In this section we will describe Judea Pearl’s approach [25] to the problem of causality which combines a symbolic language, do-calculus, with the use of graphs. 2.2.1 Causal diagrams Causal diagrams are a powerful tool in causal inference used to represent causeeffect relationships graphically but also to facilitate formalisation [23]. The main diagram used is the directed acyclic graph (DAG). Definition 2.2.1 (Directed acyclic graph).In causal inference, a directed acyclic graph (DAG) is a graph in which nodes represent variables and each arrow a cause-effect relationship between two of them. A DAG does not contain cycles, and it is only complete if variables that are common cause for any other two are included. It is important to note that DAGs are acyclic since variables should not cause themselves. This implies that bidirected arrows are not permitted, as two variables cannot cause each other [30]. Chapter 2. Preliminars 17 There are three basic causal structures [25, 27]: 1. Chain: A →B→C. B is a mediator between the cause A and the effect C. 2. Fork: A ←B→C. B is a common confounder for A and B. 3. Collider or inverted fork: A →B←C. Both A and C are causes for B. Example 2.2.2 (DAG).In the following DAG we can see a chain (A →B→C), a fork (D ←B→C) and a collider (A →E←C): A B C D E Figure 2.2: Directed acyclic graph Another type of causal diagram that will be particularly useful throughout this work is the causal chain graph. They are useful when the complete causal structure is not known but only a partial ordering is available. Definition 2.2.3 (Causal chain graph).A causal chain graph consists of chain components that contain a certain number of variables and are linked by directed edges without forming cycles [12]. In each chain component, the relationship between variables can be that of a common confounder, mutual interactions, etc. Example 2.2.4 (Causal chain graph).In a causal chain graph, only causal relationships between different chain components are known. The relationships between the variables of a given chain component are assumed. In this case, we assume a common confounder for A, B, C and mutual interactions for G, H, I. A BC D E F G H I Figure 2.3: Causal chain graph 18 2.2. The problem of causality 2.2.2 Pearl’s do-calculus The do-calculus is an axiomatic system developed by Judea Pearl in 1995 to enable identification of causal effects in models [24]. The basis of the do-calculus is the operator do(X=x). It represents an intervention in which a function Xof the model is replaced by a constant X=x, therefore actively setting the value of the variable. The idea is to be able to manipulate expressions of the form p(y|do(x), z, w)so that we arrive to classic probabilities without the do-operator. There are three basic rules that, applied successively, allow to do so. These rules are dependent the causal structure given by the diagrams, and in fact allow for them to be simplified. We will denote GXthe operation of deleting edges from graph Gthat point to variable X, and GXthe one of deleting edges that come out from X. The rules are therefore the following [24, 25]: 1. Rule 1 (Insertion/deletion of observations): p(y|do(x), z, w) = p(y|do(x), w) if the variables Wand Xblock all possible paths from Yto Zin GX. 2. Rule 2 (Action/observation exchange): p(y|do(x), do(z), w) = p(y|do(x), z, w) if the variables Wand Xblock all possible paths from Yto Zin GXZ. 3. Rule 3 (Insertion/deletion of actions): p(y|do(x), do(z), w) = p(y|do(x), w) if the variables Wand Xblock all possible paths from Yto Zin GXZ(W), where GXZ(W)accounts for the graph GXwithouth the edges that go into Zif Zis not an ancestor of W. For further explanations on the do-calculus the reader is referred to [24]. Chapter 2. Preliminars 19 2.3 Prediction models In this section we will introduce briefly the different prediction models supported by the shapr package, as explained in Chapter 4. These models can be used both in classification and regression problems. 2.3.1 Generalized Linear Models Generalized Linear Models (GLM) emerged as a generalisation of linear models. These models assume that the response variable Yfollows a distribution in the exponential family. GLM have three components [19]: •Random component: probability distribution of the response variable Y. •Systematic component: linear predictor produced by the covariates (x1, . . . , xn), noted as η=Pn i=1 βixi. •Link function: associates the random and systematic components, E[Y] = η. In GLM models, the linear relationship is not between the response variable and the covariates, but between the transformed response variable by means of the link function and the covariates. Example 2.3.1 (Logistic regression).The logistic regression is an example of GLM which can be used in binary classification problems. The link function is η= log π 1−π where π=P(Y= 1 |X=x)and 1−π=P(Y= 0 |X=x). Generalized Additive Models with integrated smoothness estimation Another example of GLM are Generalized Additive Models (GAM). In this case, the linear predictor is obtained by adding the result of applying smooth functions over the covariates and a conventional parametric component of the linear predictor (α): η=α+f1(X1) + · · · +fn(Xn)(2.3.1) Smooth functions over the covariates can be chosen freely by the user. However, there are selection techniques based on likelihood that penalize its maximization in order to avoid over-fitting. 20 2.3. Prediction models 2.3.2 Random Forests Random Forests are a supervised learning algorithm that was firstly introduced by L. Breiman in [2] in 2001. The idea is to build a large number of non-correlated decision trees and then averaging the prediction of each of them, which will therefore be the Random Forest prediction. This is known as ensemble learning, since multiple algorithms are combined to obtain more accurate predictions. The algorithm as described in [11] is the following: Algorithm 1 Random Forest 1. For b= 1 to B: (a) Draw a bootstrap sample of size Mfrom the training data. (b) Create a decision tree Tbof size nmin (minimum node size) by repeating: i. Randomly select pvariables from the ninitial variables. ii. Pick the best separation variable among the pselected. iii. Split the node into two. 2. Output the ensemble of trees {Tb}B 1 3. The prediction for an individual xwill be CB rf (x) = majorityvote{Cb(x)}B 1, where Cb(x)is the class prediction for the b-th random forest. Fitting Bmodels using Bdifferent bootstrap samples and then averaging the prediction is a technique called bootstrap aggregation or bagging. Another characteristic of Random Forests is feature randomness. Randomly selecting a subset of variables from the original space of features allows for more independence across trees, since the separation variable at each step will be different. Finally, this method can also be used for regression. In that case, the prediction for an individual xcan be calculated as fB rf =1 BPB b=1 Tb(x). 2.3.3 Extreme Gradient Boosting (XGBoost) The eXtreme Gradient Boosting (XGBoost) algorithm is another supervised learning algorithm that can be used both in regression and classification problems. It gained notoriety as it was widely used and achieved great results in Machine Learning competitions [10]. Chapter 2. Preliminars 21 The idea behind it is building a large number of trees using boostrap aggregation, but in this case trees are built sequentially trying to minimize errors from previous ones. This technique is known as tree boosting. The error is minimized using a gradient descent optimization algorithm, with gives name to XGBoost. The adjective extreme comes from the fact that it is optimized to treat missing values and use few computational resources to provide accurate predictions. A detailed explanation of the algorithm can be found in [3]. 22 2.3. Prediction models Chapter 3 SHAP SHAP (SHapley Additive exPlanations) is one of the most commonly used model agnostic techniques to explain predictions. In a Machine Learning model, it allows to quantify the contribution of a particular feature in the prediction for an observation by assigning it an importance value [17]. This measure of importance is calculated through the use of Shapley values, designed in the first instance to solve a problem in Game Theory. Later on, the underlying idea was applied in the area of Explainable Machine Learning. 3.1 Shapley values Shapley values were proposed by Lloyd Shapley in 1953 in the context of Game Theory [28]. The purpose was to numerically determine the contribution or marginal gain of each player in a cooperative game. Definition 3.1.1 (Characteristic function).Let N={1, . . . , n}be a set of players. The function v: 2N→Rdefined over this set is known as characteristic function. It measures the value or contribution of a given subset Sof Nwith v(∅)=0. The characteristic function can therefore measure the marginal gain of an individual player or a subgroup of them. As players may have overlapping skills, it is needed to average the contribution of a player in all possible formations, i.e. in all possible subsets of N. A Shapley value is therefore the calculation of the marginal value of all possible subsets weighted over all possible permutations. 23 24 3.1. Shapley values Definition 3.1.2 (Shapley value).Let N={1, . . . , n}be a set of nplayers, Sa subset with size sand vthe characteristic function. The Shapley value of player iis defined as: ϕi(v) = X S⊆N∖{i} s!(n−s−1)! n![v(S∪ {i})−v(S)] (3.1.1) With nbeing the number of players, S⊆N∖{i}indicates all the possible subsets in which we can add player ito the set. The term s!accounts for all the possible permutations of players in Sand (n−s−1)! for those belonging to the residual players that are not in it. Finally, n!is the total number of possible sets, taking order into account. It acts therefore as a normalizing constant as it is in the denominator of all weights. Proposition 3.1.3 (Properties of Shapley values).Shapley values satisfy the following properties [12]: 1. Symmetry: If v(S∪{i}) = v(S∪{j})for all S⊆N∖{i, j}, then ϕi(v) = ϕj(v). 2. Null-player (Dummy): If v(S∪{i}) = v(S)for all S⊆N∖{i}, then ϕi(v)=0. A player that never contributes to the game either directly or indirectly has null Shapley value. 3. Linearity: If we have two possible value functions v1and v2, then they satisfy ϕi(α1v1+α2v2) = α1ϕi(v1) + α2ϕi(v2)for any α1, α2∈R. 4. Efficiency: The marginal gain of each player of the game equals the total contribution of the set of players: v(N) = PN i=1 ϕi(v). 3.1.1 Model explanation using Shapley values In Machine Learning, Shapley values are at the core of some local, post-hoc, model-agnostic interpretation methods. Let us assume that we have a prediction model fover a set Nof nfeatures that we aim to explain. Shapley values allow to measure the contribution of a feature ito a prediction made for a specific observation x= (x1, . . . , xn). Chapter 3. SHAP 25 For linear models, Shapley values can be calculated as [17] ϕi(f) = X S⊆N∖{i} s!(n−s−1)! n![fS∪{i}(S∪ {i})−fS(S)] (3.1.2) where the characteristic function is set as v(S) = fS(S)and is the result of evaluating the model fStrained only with the features present in the subset S. In general, Shapley values measure the difference between the estimate of the model for features that are not in Sand the averaged predicted value of the model. The estimate of the model over a subset of parameters is the result of integrating the parameters that are not in the subset, i.e. marginalizing over features that are not in the model. As a result we obtain the following expression of the characteristic function: v(S) = Zf(x1, . . . , xn)dPi/∈S−EX[f(x)] (3.1.3) When substituting this characteristic function in Equation 3.1.1, since the expected value of the model EX[f(x)] will be the same for any subset S, it will cancel. The problem that remains then is how to approximate the integral of the parameters that are not in the subset. 3.2 SHAP SHAP (SHapley Additive exPlanations) was introduced by Lundberg and Lee as a framework that allows prediction interpretation [17]. The idea is explaining the prediction of a given instance by calculating the marginal contribution of each feature. It is an additive feature attribution method. Proposition 3.2.1. If xis a given input and x′its coalition vector, SHAP verifies the following properties: 1. Local accuracy: The explanation model and the original model agree f(x) = g(x′) = ϕ0+ n X i=1 ϕix′ i This property is consistent with the efficiency property of the Shapley values (3.1.3) by setting ϕ0=E[f(x)] and x′ i= 1 for all i= 1, . . . , n [20]. 32 3.6. Other methods Frye et al. [7] named these Shapley values for which symmetry is lost as Asymmetric Shapley values, and proposed them as a way to incorporate causal knowledge into model explainability. They argue that the property of symmetry may confuse causal relationships in the data, and that some choices of w(π)incorporate causal understanding into the explanation. If there are causal dependencies between features, this can be introduced by setting w(π) = 0 for those permutations in which a known ancestor idoes not precede a descendant j. As such, the Asymmetric Shapley value of feature xiwill measure its effect by assuming feature xjis unknown (since it is its descendant), while the Asymmetric Shapley value of feature xjwill assume its predecesor xihas already been specified. 3.5.2 Causal Shapley values Heskes et al. [12] try to provide explanation that takes into account causal dependencies through a different method. They introduce the notion of Causal Shapley value by choosing the following characteristic function: v(S) = E[f(z)|do(zS=xS)] = Zf(z¯ S, xS)p(z¯ S|do(zS=xS)) dz¯ S(3.5.2) This equation is just the result of replacing p(z¯ S|zS=xS)in Equation 3.3.2 by p(z¯ S|do(zS=xS)). The use of Pearl’s do-calculus allows to measure the contribution of a given feature if we actively set its value compared to not knowing it. The practical calculation of these causal Shapley values is made by means of the causal diagrams presented in Section 2.2.2. It is possible to retrieve the formulas for p(z¯ S|do(zS=xS)) from both DAGs and causal chain graphs, since the latter can be considered as a DAG where instead of variables there are chain components. Details on these formulas can be found in [12]. 3.6 Other methods Lundberg and Lee [17] proposed numerous methods that supposed an improvement to previous approaches to explainability, such as Kernel SHAP, Deep SHAP, Linear SHAP and Max SHAP. We will present Kernel SHAP, Tree SHAP [16] and the Monte- Carlo sampling method. Chapter 3. SHAP 33 3.6.1 Kernel SHAP Kernel SHAP combines ideas from Shapley values and LIME (Local Interpretable Model-agnostic Explanations) [26], another local explainability method. LIME creates local linear explainable models by training a dataset of perturbed samples from the original instance. The originality of this method is that it is possible to retrieve the Shapley values from the coefficients of a weighted linear model. The weights are obtained from a SHAP kernel proposed by Lundberg and Lee in [17]: πx(z′) = n−1   n |z′| |z′|(n− |z′|) (3.6.1) where |z′|is the number of features present in the coalition vector z′, i.e. the number of components of the vector that are equal to 1. They prove then that if the explanation model gis trained by optimizing the loss function L(f, g, πx) = X z′∈Z [f(hx(z′)) −g(z′)]2πx(z′)(3.6.2) then the Shapley values can be recovered. Kernel SHAP algorithm is as follows: Algorithm 2 Kernel SHAP 1. For k= 1 to K: (a) Sample a coalition vector z′ k∈ {0,1}n. (b) Convert z′ kto the original feature space by using hx, with xthe instance of interest, and then apply the model to get f(hx(z′ k)). (c) Use the SHAP kernel (Equation 3.6.1) to calculate the weight πx(z′ k)for z′ k. 2. Fit the weighted linear model by optimizing the loss function described in Equation 3.6.2. 3. The coefficients of this linear model will be the Shapley values ϕk. 34 3.6. Other methods This method is an improvement since it offers a solution to the problems that arise from the exponential complexity of the Shapley values formula and the challenge of estimating SHAP values. However, it still assumes feature independence, and therefore presents the same issues when dealing with dependencies between variables. 3.6.2 Tree SHAP Tree SHAP allows to compute SHAP values for decision tree models in a more computationally efficient way [16]. The characteristic function is defined by means of the conditional expectation instead of the marginal expectation. This results in a reduction the complexity of computing from O(TLDN)to O(T L2N), where Tis the number of trees, Lthe maximum number of leaves in any tree, Dthe maximum depth and Nthe number of features. The aim is to calculate E[f(z)|zS=xS]in a tree model. The prediction for zcan fall into different leaves since only known values for zare zS=xS. The idea is then the following [20]: • If S=Nthe set of all features, the expected value will be the expected prediction of the model, the only leave into which zcan fall. • If S=∅, the expected value will be the weighted average of all leaves, where the weights are calculated by node size (number of training samples that fall in that node). • If Scontains some features, the expected value will be the weighted average of the leaves that can be reached following the decision path given by zS=xS. This last calculation can be performed in polynomial time by calculating the outcome of considering all possible subsets at the same time, descending in the tree. A detailed explanation of the algorithm can be found in [16] and [18]. Finally, as Shapley values are additive, this method is also useful when working with tree ensemble models, since the SHAP values will be the average of the SHAP values for the single trees. Chapter 3. SHAP 35 3.6.3 Monte-Carlo sampling When dealing with large sets of features, problems of computational nature may arise. All possible subsets S⊆Nneed to be evaluated, thus becoming an intractable problem. A possible solution lies in the use of the Monte-Carlo sampling algorithm [20], which allows to retrieve an approximation to the Shapley value of a feature ifor a given individual x. Algorithm 3 Monte-Carlo sampling method to approximate Shapley values 1. For m= 1 to M: (a) Obtain a second individual yby sampling randomly from the dataset. (b) Create two new instances from xand yby replacing a number of feature values by others from the individual y: xm +i= (x1, . . . , xi−1, xi, yi+1, . . . , yn) xm −i= (x1, . . . , xi−1, yi, yi+1, . . . , yn) In the first case, the original value of feature ifor the individual xis kept while in the second it is also taken from the sampled individual y. 2. The Shapley value of feature iwill therefore be the result of averaging the difference of those two instances: ˆ ϕi=1 M M X m=1 f(xm +i)−f(xm −i)(3.6.3) 3.6.4 Feature importance Shapley values as described in this section allow to calculate importance of a feature in a particular prediction (local interpretability). In order to estimate the importance of a feature for a Machine Learning method, it is possible to average Shapley values across instances. If our prediction model has been trained over nfeatures, the relevance of feature j can be calculated as follows Φj=1 M n X i=1 |ϕi j|(3.6.4) where ϕi jrepresents the Shapley value of feature jfor the instance i. 36 3.6. Other methods This method would give a measure of global interpretability, that is, an explanation of the expected model behaviour [21]. Chapter 4 Computational implementation In this chapter, the computational tools used to design and carry out the different experiments in Chapter 5 will be described. The packages for Shapley value computation, model training and plotting will be described. 4.1 Shapley values computation One of the main packages for the calculation of Shapley values in R is shapr. It is based on the method for computation of Shapley values described in [1]. The package is composed of three functions [15]: •shapr: gets Shapley weights for test data, creating an explainer object. •explain: computes Kernel SHAP values for test data. •plot.shapr: plots the individual prediction explanations. The first step is training the model. Afterwards, the function shapr() can be used to obtain the Shapley weights for test data, that is, the predictions we want to explain. Arguments include the data, the trained model and optionally the number of combinations. By default, when omitting the argument for the number of combinations, it will calculate the weights for the 2npossible combinations, where n=|N|is the number of features considered. The data should have a numeric form, with categorical features encoded and converted into integer format. 37 38 4.1. Shapley values computation This function currently supports explanation of the following models: • Generalized linear models: stats::glm. • Linear models: stats::lm. • Random Forests through the package "ranger": ranger::ranger. • Extreme Gradient Boosting (XGBoost): using both xgboost::xgboost or xgboost::xgb.train. • Generalized additive models with integrated smoothness estimation using the function mgcv::gam. After the explainer object is created in the previous step, individual predictions can be explained using the function explain(). The main arguments are test data corresponding to the individuals to be explained, the explainer object, the approach to be used and the prediction value for unseen data, which is usually the mean of the response. There are different approaches to the computation of Shapley values as seen in Chapter 3. Some options are empirical for the empirical approach, gaussian for the multivariate Gaussian distribution approach and copula for the Gaussian copula approach, amongst others. If multiple individuals are explained, a different approach for each individual can be chosen by giving a vector of the length of the number of predictions to be explained as an argument. 4.1.1 Causal Shapley values To compute causal Shapley values as described in Chapter 3, authors in [12] developed an extension of the shapr package that allows to do so. The code is available at their GitLab repository. This extension modifies both functions presented before introducing new arguments. The causal structure of data, also known as partial ordering is needed to calculate the causal Shapley values. This partial ordering is given in the form of a list of vectors, each of them representing a chain component of the causal chain graph. The vectors themselves contain the numbers corresponding to the columns of the variables in the dataset that are included in the chain component. Chapter 4. Computational implementation 39 The first function receives two new arguments: shapr(..., asymmetric = FALSE, ordering = NULL). The argument for asymmetry allows to compute both symmetric or asymmetric Shapley values. The ordering argument receives the partial ordering list. As for the explain() function, the approach causal is introduced in addition to those mentioned above. Additionally, three new arguments are to be set for the function: the previously seen for asymmetry and ordering and another for confounding. The confounding argument accounts for whether the variables in each of the chain components are confounded or not. It is therefore a vector of logicals, with the same length as the partial ordering list. 4.2 Data preparation, model training and plotting Data preparation The dataset to be used in the experiments has been prepared using functions from the tidyverse package to merge, filter and declare types of variables. A descriptive analysis of the features has also been made using this package. Partitioning data into train and test sets is possible using the library caret (Classification And REgression Training) and its function createDataPartition(). This function receives a vector with the response variable, the number of partitions to create and the percentage of training data, among other arguments. It returns a list of the row positions that will be included in the training data. Random Forest Different libraries allow Random Forest implementation. The one which is currently suported by the shapr package is the library ranger. The function to train the Random Forest is ranger(). It receives as input the formula (description of the model to fit) and the training data. Other arguments that can be set are the number of trees to calculate, whether sampling with replacement or not, weights for sampling or whether the forest to grow is a classification forest. More information is available on the online documentation. XGBoost XGBoost models can be implemented through the xgboost package by using the function xgb.train() or the simpler version xgboost(). These functions receive the training data in matrix form as an input and a set of parameters. 40 4.2. Data preparation, model training and plotting Other arguments such as the maximum number of iterations, printing information about model performance or saving the model can also be set. A detailed description of the functions and its arguments can be found in the online documentation. Plotting results Plots have been constructed using ggplot2, the package for graphical representation included in tidyverse. The code for the individual explanation plots and the sina plots has been adapted from the ones available in the shaprr package and its causal extension respectively. Chapter 5 Experiments In this chapter, a series of experiments will be performed on real data in order to illustrate the concepts introduced in this work. Data used has been obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI). We will first describe the dataset and its causal structure, which we will compare with the structure of associations between variables. Later, we will contrast the different ways of calculating Shapley values described in Chapter 3. Finally, we will see if there are differences in Shapley values when working with different models. All the code used for the different experiments can be found on this GitHub repository. 5.1 ADNI database Data used were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). The ADNI was launched in 2003 as a publicprivate partnership. Its primary goal has been to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (MCI) and early Alzheimer’s disease (AD). Features considered are described in Figure 5.1. They are divided into three groups according to their type. The demographic variables account for the sex, years of education and age at baseline (at the beginning of the study) of the patients. Biomarkers are indicators in the human body that empirical studies have associated with the presence of Alzheimer’s disease. Finally, the gene ApoE4 has also been linked to an increased risk of development of the disease. 41 48 5.3. Differences in methods These Shapley values have been calculated setting the arguments of the different functions as follows: • Symmetric Shapley values: without including the argument for asymmetry. –Symmetric causal Shapley values: causal approach and partial order. –Marginal Shapley values: causal approach with all variables confounded. • Asymmetric Shapley values: including the argument for asymmetry. –Asymmetric Shapley values: causal approach with all variables confounded. –Asymmetric causal Shapley values: causal approach and partial order. We can see that there are differences across the Shapley values calculated, especially for categorical features. Both asymmetric Shapley values clearly separate the categories of APOE4 and PTGENDER, whereas the symmetric ones do not do so well. In the case of continuous features, there are hardly any differences between methods. In some of the variables, such as FDG and ABETA, low values of the feature might be associated to negative Shapley values and high values to positive in all methods. For the remaining variables no evident patterns are observed in any of the four types of Shapley values. 5.3.3 Analysis of individual explanations Additionally, we can plot individual explanations using the four Shapley values dealing with causality for given individuals to further illustrate the experiment. We have chosen three different observations, one corresponding to the positive class (Disease Present, with a low predicted probability), other classified in the negative class (Cognitive Normal, high predicted probability) and the third having a predicted probability near 0.5. The different plots show the Shapley value for each of the features alongside their value. Positive Shapley values are considered to push to increase the predicted value, that is, the predicted probability, whereas features with negative Shapley values will be the ones pushing to decrease the prediction. Chapter 5. Experiments 49 For the first individual the model has predicted the presence of Alzheimer’s disease. In all four Shapley value prediction explanations we can see that the variable that influences the decision the most is the age of the individual, which pushes to decrease the predicted probability, therefore contributing to the classification in the positive class. As for the rest of the variables, it is here where we see differences across methods. In the symmetric one, the other variable that seems to have an influence in the prediction is ABETA, with a positive contribution. However, in both asymmetric methods this variable has nearly no influence, and in the case of the asymmetric Shapley value we see that instead of contributing to increase it does so to decrease the predicted probability. Figure 5.7: Individual plots for the different Shapley values in the XGBoost model The second individual has a predicted class with no disease present since its predicted probability is nearly 1. In this case the variables that have the greatest influence is FDG, contributing towards an increment of the predicted probability. For the rest of features, we see differences across methods. For the causal Shapley value, the second variable that influences the prediction the most is the age, as in the marginal Shapley values. For the asymmetric Shapley values we see that we see that both PTAU and AGE have an positive influence in the decision. 50 5.3. Differences in methods Finally, in the case of the asymmetric causal Shapley values both AGE a PTAU have a positive influence as in the precedent case, but it is also remarkable that ABETA pushes towards decreasing the predicted probability, something that does not happen in the rest of cases. Figure 5.8: Individual plots for the different Shapley values in the XGBoost model Finally, the third individual is assigned to the class with disease present, although it has been chosen because the predicted probability is close to the threshold, established in 0.5. The variable that contributes the most towards the prediction is FDG in a positive way for all methods, but we find discrepancies on the contributions of the other features. In the causal Shapley value nearly no other variable has influenced the decision. As for the marginal Shapley value, age seems to be of some relevance. Differences are found in both asymmetric Shapley values, since PTAU pushes to decrease the predicted probability but seems to have no influence when using symmetric methods. We also see that there are discrepancies in the way the rest of the variables contribute across all methods, but since they are not very relevant this can be disregarded. Chapter 5. Experiments 51 Figure 5.9: Individual plots for the different Shapley values in the XGBoost model 5.4 Differences in models This second experiment aims at showing if there are differences in Shapley values across models. To this end, we will compare the four Shapley values dealing with the causal structure obtained before for the XGBoost model with the obtained for a logistic regression model, a type of Generalized Linear Model. The other methods are not presented since they posed computational problems. Predicted DP CN Actual DP 130 8 CN 32 5 Table 5.10: Confusion matrix for the logistic model 52 5.4. Differences in models In the model evaluation metrics for the logistic regression we get an accuracy of 77.1%, a sensitivity of 80.3% and a specificity of 38.5%. See also Table 5.10. The summary plots of the four Shapley values are shown in Figure 5.11 Figure 5.11: Sina plots for the different Shapley values in the logistic model In general, we see that in this case a relationship between the Shapley values and those of the variables is best observed compared to what happened in the XGBoost model. For the categorical variables, both symmetric and asymmetric Shapley values separate the different categories, whereas this only happened in the asymmetric Shapley values of the XGBoost model. In the logistic model, the three categories of APOE4 are better differentiated in the asymmetric case. Additionally, the two genders are clearly separated in the PTGENDER variable, and all individuals corresponding to one of the categories obtain the same Shapley value in this variable. Chapter 5. Experiments 53 As for the continuous variables, in the XGBoost model it was difficult to tell whether a given value of a feature would correspond to a low or high Shapley values. For some variables, such as PTAU or PTEDUCAT this was impossible. In the case of the logistic model, the PTAU variable is still difficult to read, whereas for the rest, it is intuited that low feature values will correspond to negative Shapley values and viceversa. It is also observed that the variables were more spread out on the Shapley value axis for the XGBoost whereas in this case they are more clustered. This difference is particularly noticeable in the case of the AGE variable. Finally, within the logistic model there are no significant differences between Shapley values for continuous variables. 54 5.4. Differences in models Chapter 6 Conclusions In this work we have shown how Shapley values, which were born 70 years ago in the context of Game Theory, can be an useful tool when addressing a recent issue in Machine Learning, that of model explainability. They are thus the idea behind SHAP, a technique that allows local interpretability but whose results can also be extrapolated to explain models globally. At first, SHAP values were developed assuming feature independence. In this text, we present the original technique, as well as later developments that focused on removing this assumption with the objective of getting accurate explanations efficiently. Three methods are described, which work with Gaussian distributions, Gaussian copulas and empirical conditional distributions - a non-parametric approach. Additionally, we aimed at introducing the problem of causality. With models being increasingly used in different areas of society and taking part in decision-making processes, it is essential that they accurately reflect and model the reality of the world we live in. In that sense, we present two methods to calculate Shapley values when knowing the underlying causal struture of the variables, thus introducing causality into the realm of explainability. One of this methods relies on Pearl’s axiomatic system, the do-calculus, whereas the other works with the symmetry property of Shapley values. In order to illustrate these concepts, we chose the ADNI (Alzheimer’s Disease Neuroimaging Initiative) dataset to carry out different experiments. The choice for this dataset was motivated by the fact that some authors had proposed a causal structure for its features, thus allowing to calculate Shapley values that need this information. We have compared seven different techniques for calculating Shapley values for the XGBoost model. 55 56 In general, it can be seen that the main discrepancies are found between symmetrical and asymmetrical Shapley values. These differences are best noticed when analysing prediction explanations for specific individuals, as it can be observed that the methods sometimes differ in highlighting the variables that contribute the most towards the prediction. Finally, it is also observed that the explanations change with the models. In this case, we have compared XGBoost with logistic regression, finding that the latter separates better the variables. The question that arises now is how to go beyond. We have seen that taking into account the causal structure of the data changes explanations. The problem is that this causal relationships cannot be formally found directly from data so far. Some attemps have been made in this sense [29], but it seems there is still a long way to go in the quest of translating reality of the real world into that of formality. Acknowledgements Data collection and sharing for this project was funded by the Alzheimer’s Disease Neuroimaging Initiative (ADNI) (National Institutes of Health Grant U01 AG024904) and DOD ADNI (Department of Defense award number W81XWH-12-2-0012). ADNI is funded by the National Institute on Aging, the National Institute of Biomedical Imaging and Bioengineering, and through generous contributions from the following: AbbVie, Alzheimer’s Association; Alzheimer’s Drug Discovery Foundation; Araclon Biotech; BioClinica, Inc.; Biogen; Bristol-Myers Squibb Company; CereSpir, Inc.; Cogstate; Eisai Inc.; Elan Pharmaceuticals, Inc.; Eli Lilly and Company; EuroImmun; F. Hoffmann-La Roche Ltd and its affiliated company Genentech, Inc.; Fujirebio; GE Healthcare; IXICO Ltd.; Janssen Alzheimer Immunotherapy Research & Development, LLC.; Johnson & Johnson Pharmaceutical Research & Development LLC.; Lumosity; Lundbeck; Merck & Co., Inc.; Meso Scale Diagnostics, LLC.; NeuroRx Research; Neurotrack Technologies; Novartis Pharmaceuticals Corporation; Pfizer Inc.; Piramal Imaging; Servier; Takeda Pharmaceutical Company; and Transition Therapeutics. The Canadian Institutes of Health Research is providing funds to support ADNI clinical sites in Canada. Private sector contributions are facilitated by the Foundation for the National Institutes of Health (www.fnih.org). The grantee organization is the Northern California Institute for Research and Education, and the study is coordinated by the Alzheimer’s Therapeutic Research Institute at the University of Southern California. ADNI data are disseminated by the Laboratory for Neuro Imaging at the University of Southern California. 57