Deducing neighborhoods of classes from a fitted model

Gerharz, Alexander; Groll, Andreas; Schauberger, Gunther Article — Published Version Deducing neighborhoods of classes from a fitted model AStA Advances in Statistical Analysis Provided in Cooperation with: Springer Nature Suggested Citation: Gerharz, Alexander; Groll, Andreas; Schauberger, Gunther (2024) : Deducing neighborhoods of classes from a fitted model, AStA Advances in Statistical Analysis, ISSN 1863-818X, Springer, Berlin, Heidelberg, Vol. 108, Iss. 2, pp. 395-425, https://doi.org/10.1007/s10182-024-00502-5 This Version is available at: https://hdl.handle.net/10419/315086 Standard-Nutzungsbedingungen: Die Dokumente auf EconStor dürfen zu eigenen wissenschaftlichen Zwecken und zum Privatgebrauch gespeichert und kopiert werden. Sie dürfen die Dokumente nicht für öffentliche oder kommerzielle Zwecke vervielfältigen, öffentlich ausstellen, öffentlich zugänglich machen, vertreiben oder anderweitig nutzen. Sofern die Verfasser die Dokumente unter Open-Content-Lizenzen (insbesondere CC-Lizenzen) zur Verfügung gestellt haben sollten, gelten abweichend von diesen Nutzungsbedingungen die in der dort genannten Lizenz gewährten Nutzungsrechte. Terms of use: Documents in EconStor may be saved and copied for your personal and scholarly purposes. You are not to copy documents for public or commercial purposes, to exhibit the documents publicly, to make them publicly available on the internet, or to distribute or otherwise use the documents in public. If the documents have been made available under an Open Content Licence (especially Creative Commons Licences), you may exercise further usage rights as specified in the indicated licence. http://creativecommons.org/licenses/by/4.0/ Vol.:(0123456789) AStA Advances in Statistical Analysis (2024) 108:395–425 https://doi.org/10.1007/s10182-024-00502-5 ORIGINAL PAPER Deducing neighborhoods ofclasses fromafitted model AlexanderGerharz1 · AndreasGroll1· GuntherSchauberger2 Received: 22 November 2022 / Accepted: 16 February 2024 / Published online: 8 May 2024 © The Author(s) 2024, corrected publication 2024 Abstract In this article, a new kind of interpretable machine learning method is presented, which can help to understand the partition of the feature space into predicted classes in a classification model using quantile shifts, and this way make the underlying statistical or machine learning model more trustworthy. Basically, real data points (or specific points of interest) are used and the changes of the prediction after slightly raising or decreasing specific features are observed. By comparing the predictions before and after the shifts, under certain conditions the observed changes in the predictions can be interpreted as neighborhoods of the classes with regard to the shifted features. Chorddiagrams are used to visualize the observed changes. For illustration, this quantile shift method (QSM) is applied to an artificial example with medical labels and a real data example. Keywords Interpretable machine learning· Explainable artificial intelligence· Classification task· Feature space partition· Chorddiagrams 1 Introduction With the increasing demand for very complex models in the areas of data analysis and predictive modeling, the number of blackbox models is growing steadily. The problem with these models is that by raising the predictive power of a model or an algorithm by adding more complexity or flexibility to it, the loss of interpretability can be tremendous. While mostly it is fairly easy to understand the fitting algorithm, understanding the fitted prediction model is pretty hard. In a random forest with 500 trees, for example, it is easy to understand a single classification tree, but to completely understand the whole ensemble model it is necessary to look at every * Alexander Gerharz [email protected] 1 Chair ofStatistical Methods forBig Data, Faculty ofStatistics, TU Dortmund University, Dortmund, Germany 2 Technical University ofMunich, TUM School ofMedicine andHealth, Chair ofEpidemiology, Munich, Germany 396 A.Gerharz et al. split in every tree, which gets too expensive if the corresponding classification task was very huge and complex (Breiman 2001). The world of interpretable machine learning (IML) methods tries to open a door to understand the internals of these complex models without having to understand every single internal detail of them. Altogether, this way IML methods try to increase the trustworthiness of such complex models. A famous IML method is the computation of the permutation feature importance as described by Breiman (2001). Here, the input is randomly permuted feature by feature and it is measured how much worse a model performs after permutation in order to determine the features’ importance in the model. In contrast, the partial dependence plot (Friedman 2001), for example, does not calculate the importance of a feature in a model, but it is a well-known method to estimate the mean effect of a specific feature on the target value by shifting the inputs of some data and to observe how the output changes. A typical structure for this kind of IML methods is displayed in Fig.1. Another interpretable machine learning method that is based on this structure is the individual conditional expectations (ICE) plot, which, similar to the partial dependence plot, describes the effect of a specific feature on the target value, but instead of displaying a mean effect it presents the individual changes for every observation (Goldstein etal. 2015). Another completely different IML method is the usage of anchors (Ribeiro etal. 2018). Anchors are used to find specific features and their respective feature values that determine the prediction of an observation, while the other features could be randomly altered without affecting the prediction too much. An overview of the still verylimited range of IML methods can be found in the publicly available book of Molnar (2019), which lists evenmore IML methods and explains their usages on every day examples. Most of those methods are applicable Fig. 1 Basic concept behind most interpretable machine learning methods/explainable artificial intelligence 397 Deducing neighborhoods ofclasses fromafitted model on both regression and classification tasks (or even more), while the method proposed in this article is specifically designed for classification tasks only. The quantile shift method (QSM) presented in this work is based on the basic concept of IML (see Fig.1) and is used to determine which classes are modeled as neighbors by a fitted model with regard to specific features of interest. QSM is used to determine, which small changes in the features lead to a substantial change in the predictions in the sense that the predicted class labels change. These changes can then be interpreted as neighborhoods for the different classes of an observation before and after the shift. In contrast, the anchors method is used to find the features and their respective values, which determine a specific prediction and interpret them as substantial for this specific prediction. While both methods observe whether slight changes in the features change a prediction, the interpretation is substantially different. In an application setting, QSM can be used whenever the influence of a feature on the prediction of a target class within a fixed classification model is of interest. This could be useful, e.g., in medicine, when a physician uses a classification model for pain levels and wants to investigate what influence an increase of a feature, e.g.,the heart rate, can have on the pain level of the patient. For some classic statistical models, this could be derived from the regression coefficients. However, if the model has many target classes and many features, the finally predicted label is hard to derive from the coefficients. Also, if the underlying model is a blackbox model with no easy way for interpretation, it is a challenge to investigate the influence of a specific feature. Here, QSM can help as it will indicate how the prediction will change if a specific feature (or even multiple features) of interest are modified. The remainder of this article is structured as follows: In Sect.2, we introduce the mathematical details of the method and derive the corresponding migration matrix containing these changes, which will later be presented as a chorddiagram. Additionally, we illustrate the method’s relevance with an artificially created example with labels from the field of medicine and also provide an in-depth discussion and explanation of how to generally interpret the method’s results. In Sect.4, a real data example is used to illustrate how the method works, and different ways how to use QSM are shown. Finally, Sect.5 concludes and discusses advantages and disadvantages of the proposed method. 2 Methodology In this section, we first set the mathematical background for QSM and explain how to interpret it. As there are certain conditions, which have to be kept in mind to assure a meaningful interpretation of the results, we will then explain some possible pitfalls and how they can be solved. 398 A.Gerharz et al. 2.1 Motivation The general idea of the proposed method is to take the predictions of a model for a given data set, then increase or decrease the value of one or multiple feature(s) for each observation and interpret the changes in the predicted classes. With the proposed method, it can be investigated, which classes of a categorical target are predicted next to each other based on the partition of the feature space by a specific classification model. This way, it can be learned which target classes differ with regard to the feature(s) of interest, but have very similar (or equal) values in their other features. It can also be investigated, which of the target classes have larger or lower values of the feature(s) of interest. Furthermore, this method can also be used to investigate whether there is an ordinal structure in the categorical target variable. For this method, an intuitive way to conduct the shifts for a specific feature is to choose a fixed absolute amount and just add or subtract it to/from the feature for all observations. However, there are two major issues with this process, which we will describe in the following. If the distribution of a feature is very complex and contains some low-density areas and some high-density areas, then a model can usually learn more decision boundaries in a high-density area than in a low-density area (Fig.2). Here, it is already clear, why changing the feature values by a fixed amount is not always a good idea, because a small shift size might be enough to change the prediction in a high-density area, but not in a low-density area. If the shift size is chosen to be large, then it might be appropriate for the low-density area, but the shifts might be too big for the high-density area and some important neighbors might be skipped. The yellow arrows in this figure display a possible shift size. In the high-density area, the data points could already be shifted far enough to possibly skip two decision boundaries, while in the low-density area some data points do not even skip a single decision boundary. Another issue is that by modifying the feature values the marginal distribution gets changed a lot, leading to values that have not existed before and might even be impossible to exist. This can easily lead to extrapolation, where the model is used to predict the class of observations with unlikely or even impossible feature values (see Hooker 2004), which will result in interpretations which are not meaningful. Fig. 2 Example for a feature with highand low-density areas and possible decision boundaries; yellow arrows display a common possible shift size for all feature 399 Deducing neighborhoods ofclasses fromafitted model The proposed method tackles also these issues by using the underlying data to dictate the size of the shift for each area of the range of a feature. This can be achieved by conducting the shifts based on the empirical cumulative distribution function. This way, smaller shifts will be conducted in high-density areas and larger shifts in low-density areas. Principally, QSM can help to answer the following questions: • What happens to the predicted label, when a specific feature increases or decreases? This question can be relevant in the field of Medicine. QSM could help a physician who is concerned what effect a specific pharmaceutical, which has an impact on a feature of the classification model, could have, e.g., on the pain level of a specific group of patients. Then, QSM can indicate the effect. Although the method is designed to work globally, a trained classification model can be used with focus on a very specific group of observations to explain the model locally. Researchers of other fields, e.g., Sports or Politics, could also be interested in this kind of question as often a new training method (Sports) or a new law or political guideline (Politics) can have an impact on a feature within a classification model. If one can estimate the impact on the features, then with QSM the impact on the prediction of the target classes can be investigated. • How are the target classes related? This question can be relevant in, e.g., Social Sciences. If one models the impact of social-economic features on, e.g., the happiness in life (target variable - measured in categories), then QSM can help to investigate what kind of impact an increase in, e.g., the income or the living expenses has. Relying on quantiles to investigate the impact, as proposed in this method, might improve the interpretation. • Is there an ordinal structure within the target classes? Here, there are two types of cases in which this can be relevant: – In the first case, if one assumes that there might be some kind of ordinal structure within the target classes based on a specific feature of interest, one could investigate if increasing or decreasing that feature confirms the direction of the assumed order. E.g., if there is a model for the relation of the physical fitness of a football player and the league the player is playing in, one could assume that faster players are playing in higher leagues. Of course, there might be some more complex structures that for some players being faster might not necessarily help to play in a higher league (goalkeepers?). – The second case is when the target actually has an ordinal structure, but the method used can only do classification without taking the ordinal structure into account. In Economics, the credit rating of a country or a company is usually indicated with ordinal categories. If the classification method used can only treat this target variable as categorical and not ordinal, then QSM can help to investigate this structure. If QSM indicates that slightly tweaking one of the features makes the predicted credit rating actually jump by multiple levels, then the fitted model might actually need further investigation. For this case, by no means does QSM indicate how the model needs to be modified, but it might reveal some pitfalls of the model at hand. 400 A.Gerharz et al. 2.2 Mathematical background In the following, we will set the mathematical background for QSM. The aim is to slightly increase or decrease the value of the features of interest and observe the changes in the predicted classes. Suppose  f(x) is a final model fitted for a classification task on a sample of size n with K different classes, K≥2 , and L be the set of all the features used for this classification with a specific set size p=|L| . Then, 𝜋  f,k(x) denotes the estimated probability by the model  f( ⋅ ) for an observation x to belong to a specific class k∈{1, …,K} . Next, we determine such that k∗  f (x) is the class with the highest probability as estimated by the model  f( ⋅ ) for the observation x (from here on we will always talk about the same fitted model, which is why we drop index  f in the following for better readability). Next, we choose a subset M⊆L containing the feature(s) of interest. Mostly, the subset M has a size of |M|=1 , i.e.,we focus on a single specific feature. Let now  xi represent the feature-vector for observation i , i=1, ..., n , where those features from M each were shifted componentwisely by a small amount. The shift is done by slightly increasing or decreasing the result of the empirical cumulative distribution function (ecdf) of the subset M containing the features of interest before applying the quantile-function (see Fig.3). For this purpose, a small value ql , the quantile shift size, is added componentwisely to  Fl (⋅ ) denoting the ecdf for all features l=1, ..., p , with k∗  f (x) = arg max k 𝜋  f,k (x) q l= { ul,for Ll∈M,with ul∈ [−1, 1 ] 0, else. Fig. 3 Example of a feature shift for a single feature (continuous case) 401 Deducing neighborhoods ofclasses fromafitted model The empirical cumulative distribution function can be built either on the original data set, used for fitting the underlying model, or on a very specific data set which can be a subset of the original data or a completely new data set. Depending on the choice of the data used for interpretation, the user can choose between a global or a very targeted local explanation of the model. To prevent extrapolation in the quantile function  F − 1 l (𝛼 ) , 𝛼 is limited to the interval [0, 1] . Then, for a positive shift with ql∈(0, 1] , we define: The modifying values ql for each l∈M are set by the user. As this is a crucial point for the method, in the following we provide some examples and recommendations for a reasonable choice of q . The inverse of the ecdf  F − 1 l does not necessarily exist, as  Fl typically is not continuous. Hence, for a positive shift we have to define Equation(2) determines each value of feature l after the shift as one out of the truly observed values of the respective feature, which were used to estimate the ecdf. Due to the definition of the inverse of the ecdf as defined in Eq.(2), a positive shift is generally not comparable to a negative shift, if it is done the exact same way. While even a slight positive shift results in a change of the corresponding feature’s values, slight negative shifts typically change nothing at all. In Fig.4, both a positive (left) and negative (right) shift are shown for a feature l with values xl=(1, 2, 2, 3, 4, 4, 5)T . Now, QSM is applied with 0 < | ql | < 1 7 . In the ecdf as defined above, for the point xl=2 for example, we have  F l(2)= 3 7 (bluearrow in the left part of Fig.4). If now ql is added, this results in 𝛼 where (1)  Fl(xi,l)=min{  Fl(xi,l)+ql,1} ⟹ x i,l =  F−1 l (min{  F l (x i,l )+q l ,1}) . (2)  F−1 l (𝛼)=inf {x∶  F l (x)≥𝛼} . Fig. 4 Comparison of positive (left) and negative (right) shift with the same |ql| 402 A.Gerharz et al. 3 7 <𝛼< 4 7 (blue arrow ). Due to the definition of Eq. (2), the positive shift results in  F −1 l (𝛼)= 3 (bluearrow ), which means that a very small positive shift results in a change of the feature’s value. If an equally small, but negative shift is used for the same value xl=2 , then  F l(2)= 3 7 (red arrow in the right part of Fig. 4), which is the same as for the positive shift. Next, the amount |ql| is subtracted, which results in 𝛼 where 2 7 <𝛼< 3 7 (red arrow ). When the shifted value is calculated now, due to the definition from Eq.(2),  F −1 l (𝛼)= 2 (redarrow ), which means the value has not changed at all. In fact, in the present example not a single value would change when performing the negative shifts, which shows that negative and positive shifts of the same absolute amount q are not necessarily comparable as conducted right now. Hence, a negative shift has to be defined in another way. A negative shift can be done by using a positive shift in the same manner as before after flipping the whole distribution of the corresponding feature. In particular, in a first step a whole feature l is flipped by multiplying all of its values with (−1) . Next, the ecdf is constructed for the flipped feature l and the absolute amount of ql is added to its values. Last, the resulting quantiles give the new values of the flipped feature l which have to be transformed back to get meaningful values on the original scale. Therefore, for any ql from ql∈ [−1, 0) Eq.(1) changes to with  F−l( ⋅ ) as the empirical cumulative distribution function of the flipped feature l and  F−1 −l (⋅ ) being the corresponding quantile function. With this small difference in the process of the shifts the shift sizes become comparable in both directions. Combining all equations from above, the shifts become well defined as Now,  xi is the new shifted observation, which has the same value for those covariates from L⧵M as xi , but different values for the features from M . These features were increased or decreased by the value that corresponded to the component wise raise or reduction of the respective ecdf by the amount ql , not exceeding the minimum or maximum of the empirical distribution of the features from M . Principally, this modus operandi does not only work for metric features, but also for (ordered or nominal) categorical features of the form xl∈{1, …,c} , where c is the number of categories. For this kind of features, a shift from one group to another has to be chosen manually, e.g., switching from group r to another group s within a specific feature l , i.e.,changing from xl=r to xl=s . Finally, for observation i,i=1, ..., n , M⊆L and q =(q 1 , ..., q p ) T (possibly including zeros if M⊂L ) let Cq,M(xi) define the pair of the original and the (potentially) new class prediction resulting from this shift, i.e., (3)  xi,l =−  F −1 −l (min{  F −l (−x i,l )+ | q l| ,1}) , xi,l= { F −1 l(min{  Fl(xi,l)+ql,1}),for ql∈(0, 1] −  F−1 −l (min{  F −l (−x i,l )+ | q l| ,1}),for q l ∈ [−1, 0) . 409 Deducing neighborhoods ofclasses fromafitted model Note that in Fig. 6, features x1 and x2 were intentionally presented without a scale. The intention is to illustrate by this artificial data example why using quantile-based feature shifts can be advantageous compared to using plain (rather arbitrary) numbers. The main reason is that it is often hard to determine what size a “slight” increase or decrease might be in the typical case when the exact distribution of the features is unknown. This is particularly relevant when the task is to find direct neighborhoods. Moreover, if many predictors are present, a fast and automatic method for the computation of the corresponding chorddiagrams is essential, instead of determining for every feature and observation manually, what a slight shift might be. Different features usually have different scales and, depending on their location in the feature space, a “slight” shift could have a different meaning for different observations, especially if a feature of interest has a very complex distribution (e.g., a multimodal distribution). To avoid the problem of different scales, it could be a good idea to determine the shift sizes based on the feature’s standard deviation, but especially for features with high-density and low-density areas, this still does not solve the problem (see Fig.2). All of these cases can be handled by using small amounts on the quantile scale, which are comparable for all metric features. 3.1 Remarks aboutthemethod’s interpretation In the following, we give some important remarks regarding the interpretation of the results. 1. If we define the preference order then due to the fact that particularly complex models can produce also complex partitions of the feature space, it follows This expresses that the results of QSM can not be interpreted transitively. In particular, a rather complex model could classify a specific class spotwise in the feature space, in which case one could get results that seem transitive, but in fact are not (for more details, see Sect.3.2). 2. If the classification method used for modeling is only designed to estimate (linear) monotonous effects, then migrations between two classes can only be found in one direction, but not the other. If, instead, the underlying method can also estimate more complicated effects, then migrations from a certain class A into another class B, but also in the other direction, from B into A, can be found at the same time. This indicates that the values of the original feature of interest, but also the values of the covariates, can be a deciding factor on what is happening if the feature of interest is increased. In these cases, a local application of QSM can help to improve the understanding on what might happen if the feature of interest is increased. A≻B∶= class A is directly (or generally) next to class B in the direction of the shift, A≻B∧B≻C⇏A≻C. 410 A.Gerharz et al. 3. As indicated, QSM is built to find neighborhoods as described by the model, but not to proof that there is no neighborhood between two classes regarding the shift of xM . If the goal is to proof that a certain class has no direct neighborhoods within the fitted model, then one would have to fill the complete modeled space of the class of interest with data points and then had to shift xM with infinitely small steps from the starting points until the limits of the feature range. If one also considers the model for extrapolation, then these infinitely small steps would need to be done either until +∞ or −∞ , respectively, in direction of the shift, or until all points have switched classes. 4. The shift of xM could be too big, such that an intermediate class was skipped, and consequently, no direct neighborhood was found. Hence, the “neighborhoods” from above should be regarded more generally as an “exists above” (if the shift was done by raising xM ) or as an “exists below” (if the shift was done by decreasing xM ). To find direct neighborhoods one would have to start with a very small shift of xM and increase (or decrease) it continuously. In contrast to this, if one has a specific shift in mind, one could just use this specific shift and then the resulting migration matrix shows the corresponding class changes (if any). 5. If specific features are shifted and a neighborhood is found between two classes with respect to the magnitude of the shift, this neighborhood can be interpreted as a neighborhood between the original class and the class after the shift, if the task at hand is to find out, how the model describes the neighborhoods (of course, keeping in mind that an intermediate class could have been skipped). But if the task at hand is to find realistic and practical neighborhoods between modeled classes, then these neighborhoods should always be investigated in two ways. If a neighborhood is found by the intended shift between two classes, this means that there exist data points on one side near the border between these two classes. This does not necessarily mean that on the other side of this border data points can also exist. If similar shifts are carried out in the opposite direction and this neighborhood is not confirmed, then this might mean that due to the shift impossible feature combinations have been created and, thus, the found neighborhood has no practical use. Consider, e.g., a classification model for the position of football players containing shots on target and scored goals as features. If the player had a single shot on target, which resulted in a goal, then increasing the amount of goals leads to impossible data points, but a classification border, and thus a neighbor, could still be found if the model does extrapolation in this area of the feature space. A reason for this might be that the model simply extrapolates into this area of the feature space (general problem of extrapolation, which can lead to unreasonable interpretations; Hooker 2004). 6. In many cases, multiple data points could occur with equal values of a possible feature of interest. If those points are directly at a border between two classes, different problems can be observed, as shifting all data points changes the underlying distribution not just at the edges. For most applications, this is typically not a problem, but if the actual number of observations changing class is investigated, then having ties might lead to misleading results. In cases with ties, a bulk of data points tied in the feature of interest could find a neighborhood, while if the data 411 Deducing neighborhoods ofclasses fromafitted model points would not be tied, then just a small amount of data points would find this neighborhood. A detailed discussion can be found in Sect.3.3. 7. Finally, a rather straightforward and fundamental remark: If the model at hand is rather bad and inappropriate, the found neighborhoods between the classes are correct for describing and understanding the model, but would not reflect the reality. Hence, it is important to properly evaluate the model first, before it is interpreted. 3.2 Transitive interpretations To show possible problems regarding a transitive interpretation of this method, here is a small artificial data example. In Fig.8, a feature space with two features x1 and x2 is shown. A model now labels most of this feature space as class B, while a small area with a low value of both features is labeled as class C and a small area with high values of both features is labeled as class A. In addition to that there are 10 red and gray data points, which are used to describe the partition of the feature space with QSM. Now, the feature space partition should be determined by choosing q x 1 = 1 11 and q x 2 =0 (keeping x2 constant). Hence, only neighborhoods with regard to x1 are looked for. With 10 different data points, this means that holding x2 constant each data point gets assigned the next larger value of x1 contained in the data set. The data point with the highest x1 does not change, as it is already at the upper bound of the range of x1 . This shift results in the migration matrix shown in Table5. In Fig.8, the two points, which change their prediction, are marked in red and switch the modeled class through the dashed lines. These are the two points shown Fig. 8 Example for transitivity problem in a 2-dimensional feature space 412 A.Gerharz et al. in the migration matrix in Table 5 as one has switched from class A to class B and the other one from class B to class C. As there is an area of class B modeled in the direction of the shift above an area of class A, and then there is an area of class C modeled in the direction of the shift above an area of class B, based on the corresponding migration matrix one could conclude that in the direction of the feature shift there is some kind of “hierarchy”. Particularly, here one might conclude that along the direction of this shift class A is below class B, which itself is below class C, but as shown in Fig.8 this is not the case. Even if the dimensionality would be too large to graphically visualize it, just by checking the respective feature of interest for the groups separately would likely confirm the non-transitivity for this example. In particular, the respective chord diagram as shown in Fig. 9 can (wrongly) suggest this already mentioned kind of “hierarchy”. The chorddiagram shows the Table 5 Migration matrix for QSM with the shift of q =( 1 11 ,0 ) from Fig.8 Aafter Bafter Cafter Abefore 1 1 0 Bbefore 0 5 1 Cbefore 0 0 2 Fig. 9 Chorddiagram for migration matrix in Table5 413 Deducing neighborhoods ofclasses fromafitted model migration of one observation from class A to class B and the migration of one observation from class B to class C, which looks like a hierarchical structure that actually does not exist. To conclude, QSM results should not be interpreted with regard to transitivity. In this specific case, QSM just describes that there is in fact an area of class B modeled in the direction of the shift above an area of class A and there is an area of class C modeled in the direction of the shift above an area of class B. Thus, with this specific shift these two neighborhoods were found. When using other shifts in other directions, then other neighborhoods could be found. 3.3 Multiple data points withthesame value When multiple data points with the same value occur in a data set and QSM is used, some unexpected problems can occur. In the left graph of Fig.10, there are 3 data points with an equal value for x1 . When choosing the shift size q x 1 = 1 n+1 , so the smallest possible number that allows the data points to shift, then all 3 data points would change their predictions. In this case a neighborhood between class A and class B and another neighborhood between class A and class C would be found. As shown in the right graph of Fig. 10, if the same three data points would not be exactly equal but just slightly different, only one of these data points would change its prediction and the neighborhood between class A and class B would not even be found in this case, which is another problem. If the shift size q x 1 = 2 n+1 is chosen, such that every data point is shifted by 2 unique values of x1 , then both neighborhoods would be found even in the situation of the right graph in Fig.10, when the data points are not exactly equal, as one of the data points changes its prediction from class A to class B. For any continuous (and random) feature, the probability for a specific value is zero. Hence, multiple data points with the same covariate value theoretically should never occur. But in real world applications, for example due to rounding, equal covariate values are possible and could substantially change the interpretation of QSM (see Fig.10). Fig. 10 QSM with qx1 = 1 n+1 with some equal (left) and slightly jittered (right) data points 414 A.Gerharz et al. If some observations have exactly the same value for a feature of interest (e.g., due to rounding), then the marginal distribution of the corresponding feature gets changed substantially, which is not desired. To avoid this problem, there are some ways to adjust QSM to treat observations with equal values more fairly compared to those, which have unique values for a specific feature. 1. Shift all ties: One possibility is to shift all of the data points that share a specific value of a feature of interest and observe the changes in the predictions (left graph in Fig.10). In this example, this guarantees that all data points are shifted, and neighborhoods can be found more easily. As mentioned above, this method might lead to disproportionately large shifts compared to observations with unique values. Also, the marginal distribution of the feature of interest gets changed a lot by this very fast, which is a high indicator of creating improbable covariate combinations. 2. Repeatedly shift ties randomly: Another alternative in the case of multiple observations with equal values for a specific feature of interest, is to repeatedly shift the observations after determining an artificial random order. As shown in the right graph of Fig.10, when the observations were just slightly jittered, in this example two observations were changed by almost no amount and one by a larger amount. As a consequence, these observations are treated more fair compared to the other observations than in the shift all ties method, but very unequally among themselves. For the artificial example, the prediction of just one of these three observations changes. The repeatedly shift ties randomly approach does exactly that, but instead of jittering and thus adding some blurredness to the data, it repeatedly executes QSM and randomly determines an order of the tied observations. While all observations with unique values of the feature of interest are shifted in the same way for all the repeated shifts, the observations sharing their value of the feature of interest with other observations are shifted in just a fraction of the repetitions by the full shift as in the shift all ties approach. Hence, observations with equal values of the specific feature of interest are in most repetitions shifted less, and if |q| is rather small, they might not be shifted at all. Thus, in comparison to the shift all ties method, this method treats the observations with the same values of the specific feature of interest more fair compared to the observations, which have unique values of this feature of interest. Additionally, the parameter q is more sensitive as a smaller change of q has more impact on the output of the method. Thus, it can be used more flexible by the user. The differences of these methods are illustrated in Sect.4 in a real data example. 415 Deducing neighborhoods ofclasses fromafitted model 4 Real data application In this section, QSM is applied to a real data set in order to provide some insight on the method’s potential. For this purpose, a data set for classifying the location of proteins in yeast is used. First, the data set is used to show how QSM should be applied and how to interpret the results. Next, the two proposed approaches to handle ties from Sect.3.3 are compared. 4.1 Application andinterpretation ofQSM The chosen data set is the yeast data set (Horton and Nakai 1996), which was taken from the OpenML database (Vanschoren etal. 2013). The data set contains proteins from yeast and the target of the classification task is to localize the protein into one of ten possible locations within the yeast. The ten locations are further described by Horton and Nakai (1996) and contain cytoplasmic, including cytoskeletal (CYT); nuclear (NUC); vacuolar (VAC); mitochondrial (MIT); peroxisomal (POX); extracellular, including those localized to the cell wall (EXC); proteins localized to the lumen of the endoplasmatic reticulum (ERL); membrane proteins with a cleaved signal (ME1); membrane proteins with an uncleaved signal (ME2); and membrane proteins with no N-terminal signal (ME3), where ME1, ME2 and ME3 proteins may be localized to the plasma membrane, the endoplasmatic reticulum membrane or the membrane of a Golgi body. The data set also contains eight different features, which are used to classify the target variable. These features are also further described in Horton and Nakai (1996) and in the UCI Repository (Dua and Graff 2017) and represent: • mit: score of discriminant analysis of the amino acid content of the N-terminal region (20 residues long) of mitochondrial and non-mitochondrial proteins. • erl: presence of “HDEL” substring (thought to act as a signal for retention in the endoplasmic reticulum lumen); binary attribute. • pox: peroxisomal targeting signal in the C-terminus. • vac: score of discriminant analysis of the amino acid content of vacuolar and extracellular proteins. • nuc: score of discriminant analysis of nuclear localization signals of nuclear and non-nuclear proteins. • mcg: McGeoch’s method for signal sequence recognition. • gvh: von Heijne’s method for signal sequence recognition. • alm: score of the ALOM membrane spanning region prediction program. For this classification task, a multinomial regression model using the logit link function is now computed with the nnet package (Venables and Ripley 2002). When using a multinomial regression model we obtain coefficients as well as an intercept for every class of the target except of the chosen reference class for every feature. Overall, this results in 81 different coefficients for this exemplary data set. 416 A.Gerharz et al. As interpreting a multinomial logistic regression model can be very complex, there exist techniques to help with the interpretation. One method is the visualization with Effect Stars (Tutz and Schauberger 2012). Effect Stars condense the potentially huge amount of coefficients into one very clearly arranged graphic. The aim is to give a nice overview of the coefficients for all the target classes into one single star plot per feature. This overview still has its limitations as the coefficients of every feature are visualized without considering the complex relations between all the features including the intercept. Another way to assist in the interpretation of a multinomial logistic regression model is the use of Effect graphics (Fox and Weisberg, 2019; Fox and Hong, 2009). Effect graphics display how the predicted probabilities for all the target classes change for a single specific feature or a small amount of features at the same time. In this process, all other features are fixed to specific values, while the features of interest are varied over their range of values. This does not consider the complex structure between the features as this process can produce feature combinations that are very unlikely to exist or are even impossible. Also, as the additional features are fixed to specific values, the graphic only visualizes the behavior of the model in a very narrow range of the feature space ignoring everything around it. Consider a 2-dimensional example as, e.g., in Sect.3. If the effect of x1 should be investigated, then x2 is fixed to its mean (default), and the effect of x1 is investigated on a thin horizontal line in the middle of Fig.6. All the other areas are not considered for investigating the effect. Imbalanced target classes may also lead to problems in the interpretation of both methods. Effect Stars are visualized without linkage between all the features, but especially the different intercepts for the imbalanced target classes can have a huge impact on whether a class has a high likelihood to be predicted. The Effect visualizations may also suggest that a specific small target class would not be predicted for any value of the feature(s) of interest, but the cause for this could be the fixed values of the remaining features. QSM has the advantage that the interdependence between the features is considered as it uses real data points. Also, the imbalance of classes does not affect QSM, as real data points with feature combinations of the smaller classes are used instead of one fixed combination like in the Effect graphics. For the multinomial logistic regression model, QSM can help to visualize which classes will realistically be predicted if a specific feature or a small amount of features of the empirical data would have slightly larger or lower values based on the underlying model. In the following, QSM is applied to this multinomial model to learn which target classes are modeled closely and similar to each other with slight differences concerning a specific feature. For exemplary purpose, we choose to use just the single feature mit , but also other single features or even multiple features at the same time could be used. Here, it is investigated how the predicted target class would change if the computed mit score would be slightly larger. This helps to grasp the relations between the predicted target classes regarding this mit score. An important reason why this data set is used here is that it contains a lot of observations with equal feature values. Even though the data set contains n=1484 observations, xmit contains just 78 different values. For the fitted model, 417 Deducing neighborhoods ofclasses fromafitted model the quantile shift size of q mit = 75 1484+1 ≈ 0.05 is chosen and ( q mcg ,q gvh ,q alm ,q erl ,q pox ,q vac ,q nuc ) ⊤ = 0 . This means that all feature values of xmit are increased by a slight amount. By this choice we simulate a data set, where the score of the respective discriminant analysis underlying the mit value would have been slightly larger. The marginal distribution of the feature xmit is shown in Fig.11. This indicates that larger values might lead to a classification of target class MIT. Recall that feature xmit is representing a score of a discriminant analysis of the amino acid content of the N-terminal region (20 residues long) to divide mitochondrial and non-mitochondrial proteins. The coefficients of the multinomial model can be used to check if according to the model a larger value of xmit does indeed result in a larger probability of being predicted as target class MIT. QSM can now add the information, which other target classes might be similar to the target class MIT regarding the unchanged other features, but with a lower value of xmit according to the model. Of course, depending on the structure of the data set and the complexity of the model, even the target class MIT might have another target class close to it in which observations might migrate if xmit is increased. Also, it is globally investigated if other target classes are modeled similar to each other regarding the other features, but with slightly different values in xmit . Two target classes that are found to be similar in the described manner are what we refer to as a neighbors. Fig. 11 Marginal distributions of xmit for each of the target classes 418 A.Gerharz et al. In Table 6, it is shown that by using the shift all ties approach a lot of observations change their predicted class and indicate neighborhoods. These neighborhoods are visualized in the chorddiagrams in Figs.12 and 13. We find that 21 observations change their prediction from the target class CYT to MIT. This means that for 21 of the proteins the model predicts them to be mitochondrial Table 6 Migration matrix for prediction changes when raising xmit by q mit = 75 1485 with the shift all ties approach; the observed class changes are in bold font CYT ERL EXC ME1 ME2 ME3 MIT NUC POX VAC CYT 608 0 0 0 0 0 21 1 0 0 ERL 0 5 0 0 0 0 0 0 0 0 EXC 0 0 27 0 0 0 40 0 0 ME1 0 0 0 42 0 0 0 0 0 0 ME2 0 0 0 0 41 0 60 0 0 ME3 0 0 0 0 2170 20 0 0 MIT 0 0 0 0 0 0 227 0 0 0 NUC 0 0 0 0 0 0 21 293 0 0 POX 0 0 0 0 0 0 0 0 13 0 VAC 0 0 0 0 0 0 10 0 0 Fig. 12 Changes in prediction when increasing xmit with q mit = 75 1485 ; all observations portrayed 425 Deducing neighborhoods ofclasses fromafitted model Declarations Conflict of interest The authors declare there is no conflict of interest. The code used for the examples can be found in a public github repository: https://github.com/habbeda1/QSM. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. References Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001) Dua, D., Graff, C.: UCI machine learning repository (2017) Fox, J., Hong, J.: Effect displays in r for multinomial and proportional-odds logit models: extensions to the effects package. J. Stat. Softw. 32(1), 1–24 (2009) Fox, J., Weisberg, S.: An R Companion to Applied Regression, 3rd edn. Sage, Thousand Oaks (2019) Friedman, J.H.: Greedy function approximation: a gradient boosting machine. Ann. Stat. 29(5), 1189– 1232 (2001) Goldstein, A., Kapelner, A., Bleich, J., Pitkin, E.: Peeking inside the black box: visualizing statistical learning with plots of individual conditional expectation. J. Comput. Graph. Stat. 24(1), 44–65 (2015) Gu, Z., Gu, L., Eils, R., Schlesner, M., Brors, B.: circlize implements and enhances circular visualization in r. Bioinformatics 30, 2811–2812 (2014) Hooker, G.: Diagnostics and extrapolation in machine learning. Ph.D. thesis, Stanford, CA, USA (2004) Horton, P., Nakai, K.: A probablistic classification system for predicting the cellular localization sites of proteins. Intell. Syst. Mol. Biol. 8, 109–115 (1996) Keller, F., Muller, E., Bohm, K.: HiCS: high contrast subspaces for density-based outlier ranking. IEEE (2012) Molnar, C.: Interpretable machine learning: a guide for making black box models explainable (2019) R Core Team.: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2021) Ribeiro, M.T., Singh, S., Guestrin, C.: Anchors: high-precision model-agnostic explanations. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32(1) (2018) Tutz, G., Schauberger, G.: Visualization of categorical response models: from data glyphs to parameter glyphs. J. Comput. Graph. Stat. 22(1), 156–177 (2012) Vanschoren, J., van Rijn, J.N., Bischl, B., Torgo, L.: Openml: networked science in machine learning. SIGKDD Explor. 15(2), 49–60 (2013) Venables, W.N., Ripley, B.D.: Modern Applied Statistics with S, 4th edn. Springer, New York (2002) Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.