The impact of class imbalance in classification performance metrics based on the binary confusion matrix

Pattern Recognition 91 (2019) 216–231 Contents lists available at ScienceDirect Pattern Recognition journal homepage: www.elsevier.com/locate/patcog The impact of class imbalance in classification performance metrics based on the binary confusion matrix Amalia Luque a , ∗, Alejandro Carrasco b , Alejandro Martín a , Ana de las Heras a a Dpto. Ingeniería del Diseño. Escuela Politécnica Superior. Universidad de Sevilla. Virgen de África, 7. 41011 Sevilla, Spain b Dpto. Tecnología Electrónica. Escuela Politécnica Superior. Universidad de Sevilla. Virgen de África, 7. 41011, Sevilla, Spain a r t i c l e i n f o Article history: Received 4 September 2018 Revised 22 December 2018 Accepted 22 February 2019 Available online 28 February 2019 Keywords: Classification Performance measures Imbalanced datasets Class Balance Metrics a b s t r a c t A major issue in the classification of class imbalanced datasets involves the determination of the most suitable performance metrics to be used. In previous work using several examples, it has been shown that imbalance can exert a major impact on the value and meaning of accuracy and on certain other wellknown performance metrics. In this paper, our approach goes beyond simply studying case studies and develops a systematic analysis of this impact by simulating the results obtained using binary classifiers. A set of functions and numerical indicators are attained which enables the comparison of the behaviour of several performance metrics based on the binary confusion matrix when they are faced with imbalanced datasets. Throughout the paper, a new way to measure the imbalance is defined which surpasses the Imbalance Ratio used in previous studies. From the simulation results, several clusters of performance metrics have been identified that involve the use of Geometric Mean or Bookmaker Informedness as the best null-biased metrics if their focus on classification successes (dismissing the errors) presents no limitation for the specific application where they are used. However, if classification errors must also be considered, then the Matthews Correlation Coefficient arises as the best choice. Finally, a set of nullbiased multi-perspective Class Balance Metrics is proposed which extends the concept of Class Balance Accuracy to other performance metrics. ©2019 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) 1. Introduction In recent years, the scientific community working on classification algorithms has shown an increasing interest in the challenges that arise when imbalanced datasets are considered. Several overviews on these issues have been addressed in [2,17,21,24] . In these analyses, the Synthetic Minority Over-sampling Technique (SMOTE) [9] and the AdaBoost [39] are highlighted as generalpurpose solutions, although algorithms of a more specific nature can also be found, either as general-purpose [3] , as problemoriented [15] or as classifier-oriented [33] . Algorithms that address the imbalance problem in multi-label classification [8] or that use advanced classifiers such as extreme learning machines [41] can also be found. An up-to-date comparison of these techniques addressing a specific problem can be found in [40] . ∗Corresponding author. E-mail addresses: amalialuq[email protected] (A. Luque), acarr[email protected] (A. Carrasco), [email protected] (A. Martín), [email protected] (A. de las Heras). A key aspect of these methods involves the determination of the classification performance, not only in order to assess the final result, but also to obtain a figure which has to be optimized by tuning the classifier parameters. However, there is no single way to select the best algorithm as any algorithm can obtain good results in one class but poor scores in other classes. For this reason, several metrics are usually considered, which permits the polyhedral characteristics of the classification performance to be viewed from different points of views. The impact of class imbalance on classification performance metrics has therefore become a major issue. Several authors have addressed this topic by showing a few examples of this impact on accuracy [10,22] and on several other metrics [5,13,20] . To the best of our knowledge, only one systematic (albeit limited) study has been published that is not simply based on examples, [25] . The quantitative research on classification performance metrics has traditionally been tackled either by using a collection of known and widely available datasets [1,4] , or by randomly simulating the classifier results [27,34] . A mixture of random simulated variations on known datasets is employed in Jeni et al. [25] . https://doi.org/10.1016/j.patcog.2019.02.023 0031-3203/© 2019 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ ) A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 217 In order to overcome the effect of imbalance on performance metrics, several solutions have been suggested, among which the most cited is the use of the Class Balanced Accuracy ( CBA ) [1,6,32] . Approaches such as relevance-based evaluation [5] , the Normalized Precision Rate [13] , the Index of Balanced Accuracy [18,28,29] , and the multiclass performance score (MPS) [23] have also been proposed. In this paper, an extensive and systematic study is undertaken of the impact of class imbalance on classification performance metrics. Several dozen performance metrics can be found in the scientific literature, some based on a threshold, others based on probabilities, while yet others are based on ranks [14] . However, the most widely employed metrics are those based on the confusion matrix [38] , where the multi-class case is usually reduced to a set of binary cases using the One-versus-All or the One-versus- One approach [31] . For these reasons, our research is focused on classification performance metrics based on the binary confusion matrix. The rest of the paper is organized as follows. Section 2 presents the methodology employed to measure the impact of imbalance in performance metrics, thereby formally defining the confusion matrix ( Section 2.1 ) and the metrics based thereon ( Section 2.2 ). Section 2.3 proposes a new figure for the quantification of the class imbalance, and several functions and indicators of the aforementioned impact of imbalance are defined in Section 2.4 . The application to a set of classification performance metrics based on the binary confusion matrix is presented in Section 3 . The discussion and conclusion of these results are addressed in Section 4 . 2. Methodology 2.1. Definition of the confusion matrix Consider a dataset D = { d 1 , d 2 , ···, d m } made up of m elements where d k represents the k -th element. Let be a set of C classes ={ θ1 , θ2 , ···, θC } where θi defines the i -th class. The classifier C operating on d k (the k -th element of the dataset D) assigns a label θj and estimates that this element belongs to the j -th class, that is, d k C → θj or C( d k ) = θj , while it really belongs to the i -th class θi , thereby causing a misclassification (a confusion) when i  = j . Let A = { α1 , α2 , ···, αm } be the set of actual classes corresponding to the dataset D, where αk is the actual class of the element d k . Furthermore, let E = { ε 1 , ε 2 , ···, ε m } be the set of classes estimated by the classifier Cfor each element in D, where ɛ k is the estimated class of the element d k . The performance of Ccan be assessed using a measuring function M , which assigns a metric μ∈R to the pair ( A , E ) , that is, ( A , E ) M → μ. In this paper, we will focus on metrics based on the confusion matrix, which represents one of the most common methods to present the results obtained by a classifier, and is defined as CM ≡⎡ ⎢ ⎢ ⎣ m 11 m 12 . . . m 1 C m 21 m 22 . . . m 2 C . . . . . . . . . . . . m C1 m C2 . . . m CC ⎤ ⎥ ⎥ ⎦ . (1) In this expression, m ij represents the number of elements actually belonging to the i -th class ( θi ) but that are classified as members of the j -th class ( θj ). In the context of our research, it is better to describe the term m ij in relation to the total number of elements m i belonging to the i -th class ( θi ). By denoting λij as the ratio m ij / m i , the confusion matrix can be rewritten as CM = ⎡ ⎢ ⎢ ⎣ λ11 m 1 λ12 m 1 . . . λ1 C m 1 λ21 m 2 λ22 m 2 . . . λ2 C m 2 . . . . . . . . . . . . λC1 m C λC2 m C . . . λCC m C ⎤ ⎥ ⎥ ⎦ , (2) which can be expressed as the Hadamard (element-wise) product of two matrices in the form CM = ⎡ ⎢ ⎢ ⎣ λ11 λ12 . . . λ1 C λ21 λ22 . . . λ2 C . . . . . . . . . . . . λC1 λC2 . . . λCC ⎤ ⎥ ⎥ ⎦ ◦⎡ ⎢ ⎢ ⎣ m 1 m 1 . . . m 1 m 2 m 2 . . . m 2 . . . . . . . . . . . . m C m C . . . m C ⎤ ⎥ ⎥ ⎦ . (3) In the binary case, that is, when the number of classes is C = 2 , then the confusion matrix can be written as CM ≡m 11 m 12 m 21 m 22 . (4) In general, one of the classes is called the “Positive” class and the other is named the “Negative” class. Therefore, the confusion matrix can be rewritten according to this new terminology as CM ≡m PP m PN m NP m NN . (5) The elements of this matrix are named with the following convention: m PP , “True Positive” ( TP ); m PN , “False Negative” ( FN ); m NP , “False Positive” ( FP ); and m NN , “True Negative” ( TN ). The total number of positive m P and negative m N elements in Dmeet that they sum m , the total number of elements, that is, m P + m N = m . Furthermore, it is also true that the number of elements correctly classified in class P ( m PP ), and the number of elements misclassified in that class P ( m PN ), adds up to the number of elements in the positive class ( m P ), that is, m PP + m PN = m P . Similarly, it can be stated that m NP + m NN = m N . The confusion matrix can therefore be written as CM = m PP m P −m PP m N −m NN m NN , (6) which can also be formulated in terms of the λij ratios as CM = λPP m P λPN m P λNP m N λNN m N = λPP m P ( 1 −λPP ) m P ( 1 −λNN ) m N λNN m N . (7) Additionally, the total number of elements in Destimated by C as positive (despite their actual class), e P , and those estimated as negative, e N , can be written as e P = m PP + m NP = λPP m P + ( 1 −λNN ) m N . e N = m NN + m PN = λNN m N + ( 1 −λPP ) m P . (8) They also add up to the total number of elements, e P + e N = m . The definitions regarding the confusion matrix are summarized in Fig. 1 . 2.2. Metrics based on the binary confusion matrix Based on the binary confusion matrix, numerous performance metrics have been proposed [19,27,30,34,37] . For our study, the focus is placed on 10 of these metrics, which are summarized in Table 1 . All these metrics, take values within the [0, 1] range, except the last three ( MCC, BM , and MK ), whose ranges lie within the [ −1 , 1 ] interval. For comparison purposes, these metrics are used herein in their normalized version ( MCCn, BMn , and MKn ). By naming a metric defined within the [ −1 , 1 ] interval as μ, it can be normalized within the [0, 1] range by the expression μn ≡μ+ 1 2 . (9) 218 A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 Fig. 1. Confusion matrix for binary classification. Table 1 Definition of classification performance metrics. Symbol Metric Defined as SNS Sensitivity TP TP+ FN SPC Specificity TN TN+ FP PRC Precision TP TP+ FP NPV Negative Predictive Value TN TN+ FN ACC Accuracy T P+ T N T P+ FN+ T N+ FP F 1 F 1 score 2 PRC·SNS PRC+ SNS GM Geometric Mean √ SNS ·SPC MCC Matthews Correlation Coefficient T P·T N−F P·F N √ ( TP+ FP )( TP+ FN )( TN+ FP )( TN+ FN ) BM Bookmaker Informedness SNS + SPC −1 MK Markedness P P V + NP V −1 It can easily be shown (see supplementary material in the electronic version of this paper) that all these metrics can be expressed as a function μ= μ( λPP , λNN , πP , πN ) , where πP is the ratio of positive elements in the dataset ( m P / m ) and, analogously, where πN ≡m N / m . Furthermore, for balanced classes, when πP = πN = 0 . 5 , them the metrics can be formulated as μ= μ( λPP , λNN ) . These functions are depicted in Fig. 2 as a heat map for each metric. The best classifier achieves a value of λPP = 1 (all the positive elements are correctly classified as positive) and, also a value of λNN = 1 (all the negative elements are correctly classified as negative), corresponding to the upper right-hand-side corner in the graphic. Instead, the worst classifier ( λPP = 0 , λNN = 0 ) corresponds to the lower left-hand-side corner of the graphic. Although only performance metrics based on the confusion matrix are considered, a marginal approach to Receiver Operating Characteristics (ROC) analysis [16] can also be carried out. In this analysis, the Area Under Curve ( AUC ) is commonly used as a performance metric. However, for classifiers offering only a label (and not a set of scores for each label), or when a single threshold is used on scores, the value of AUC and BMn are the same [36] . Therefore, in the forthcoming sections, whenever BMn is mentioned it could also be understood as AUC . 2.3. Defining class imbalance The concept of class imbalance is relatively clear: it arises when the dataset has a different number of elements in positive and negative classes. However, its formalization is far from being univocally accepted. For instance, in [18,29] , the imbalance is characterized by the dominance ( Dom ) or prevalence relationship between the positive class and the negative class, and is defined as T P R −T NR . This value is later employed to compensate performance metrics affected by the imbalance problem. However, the dominance is not exactly a measure of the imbalance in the dataset because it considers imbalance in the outcomes of the classifier. Other authors formalize this concept by using the entropy [12] or, more commonly, the proportion between positive and negative instances (formalized as 1: X ) [13] which is similar to the imbalance ratio ( IR ) defined as m P / m N [1] , also called skew [25] . This value lies within the [0, ∞ ] range, having a value IR = 1 in the balanced case. Other authors formalize this concept by using the proportion between positive and negative instances (formalized as 1: X ) [13] or, which is similar, the imbalance ratio ( IR ) defined as m P / m N [1] , which is also called skew [25] . This value lies within the [0, ∞ ] range, having a value IR = 1 in the balanced case. In this paper, it is preferred to feature the imbalance with a value within the [ −1 , 1 ] range, while reserving the 0 value for when the classes are perfectly balanced (lack of imbalance). For this purpose, we propose the imbalance coefficient δas δ= δP ≡2 πP −1 = 2 m P m −1 . (10) For the negative class, the coefficient δN = 2 πN −1 is also defined. The sum of these coefficients is δP + δN = 2 πP −1 + 2 πN −1 = 2 ( πP + πN ) −2 = 0 . (11) Hence δN = −δP = −δ. From (10) , the value of πP can be obtained as πP = 1 + δ 2 . (12) Moreover, the value of πN can be derived from (11) πN = 1 −δ 2 . (13) Therefore, the metrics μ=μ( λPP , λNN , πP , πN ) can be redefined as μ=μ( λPP , λNN , δ) . It is clear that the value of the metric μ depends not only on the classifier’s performance, but also on the imbalance δ. It can easily be derived that the relationship between the imbalance ratio ( IR ) and the imbalance coefficient ( δ) is IR = 1 + δ 1 −δ. (14) 2.4. Assessing the impact of imbalance In order to assess the impact of the imbalance in a certain metric, its value ( μb ) for the balanced case (when δ= 0 ) is first considered. μb ≡μ| δ=0 = μ( λPP , λNN , 0 ) = μb ( λPP , λNN ) . (15) This definition is later employed to propose a family of metrics where the effect of the imbalance is dismissed. In the scientific literature, a few examples of these metrics can be found, as in the case of Class Balance Accuracy ( CBA ) [35] . On generalizing this approach, the metrics μb are called Class Balance Metrics ( CBM ). Table 2 summarizes the equations for each metric, both in the class imbalance and balance cases. These results are derived in the supplementary material of the paper, available online. With these definitions, it is now possible to quantify the impact of imbalance, by using the bias of the metric which is defined as B μ≡μ−μb = μ( λPP , λNN , δ) −μb ( λPP , λNN ) . (16) Table 3 summarizes the definition of bias for each metric. These results are derived in the supplementary material of the paper, available online. As can be observed, bias depends on three variables: B μ= B μ( λPP , λNN , δ) . In order to study this function, a 4-dimensional space is required. Its representations are first tackled using heat volumes (3D), where each point in the 3-dimensional ( λPP , λNN , δ) space has a bias-dependent colour. A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 219 Fig. 2. Heat maps for metrics with balanced classes. Table 2 Classification performance metrics as a function of imbalance. Metrics Class imbalance metrics μ( λPP , λNN , δ) Class Balance metrics μb ( λPP , λNN ) SNS λPP λPP SPC λNN λNN PRC λPP ( 1+ δ) λPP ( 1+ δ)+( 1 −λNN )( 1 −δ) λPP λPP +( 1 −λNN ) NPV λNN ( 1 −δ) λNN ( 1 −δ)+( 1 −λPP )( 1+ δ) λNN λNN +( 1 −λPP ) ACC λPP 1+ δ 2 + λNN 1 −δ 2 λPP + λNN 2 F 1 2 λPP ( 1+ δ) ( 1+ λPP )( 1+ δ)+( 1 −λNN )( 1 −δ) 2 λPP 2+ λPP −λNN GM  λPP ·λNN  λPP ·λNN MCCn 1 2 ( λPP + λNN −1  [ λPP +( 1 −λNN ) 1 −δ 1+ δ][ λNN +( 1 −λPP ) 1+ δ 1 −δ] + 1 ) 1 2 ( λPP + λNN −1 √ [ λPP +( 1 −λNN ) ][ λNN +( 1 −λPP ) ] + 1 ) BMn λPP + λNN 2 λPP + λNN 2 MKn 1 2 ( 1+ δ ( 1+ δ)+ 1 −λNN λPP ( 1 −δ) + 1 −δ ( 1 −δ)+ 1 −λPP λNN ( 1+ δ) ) 1 2 ( 1 1+ 1 −λNN λPP + 1 1+ 1 −λPP λNN ) Table 3 Bias of performance metrics due to class imbalance. Metrics Bias B μ( λPP , λNN , δ) SNS 0 SPC 0 PRC 1+ δ ( 1+ δ)+ 1 −λNN λPP ( 1 −δ) −1 1+ 1 −λNN λPP NPV 1 −δ ( 1 −δ)+ 1 −λPP λNN ( 1+ δ) −1 1+ 1 −λPP λNN ACC δ 2 ( λPP −λNN ) F 1 2 λPP ( 1+ δ) ( 1+ λPP )( 1+ δ)+( 1 −λNN )( 1 −δ) −2 λPP 2+ λPP −λNN GM 0 MCCn λPP + λNN −1 2  [ λPP +( 1 −λNN ) 1 −δ 1+ δ][ λNN +( 1 −λPP ) 1+ δ 1 −δ] −λPP + λNN −1 2 √ [ λPP +( 1 −λNN ) ][ λNN +( 1 −λPP ) ] BMn 0 MKn 1 2 ( 1+ δ ( 1+ δ)+ 1 −λNN λPP ( 1 −δ) −1 1+ 1 −λNN λPP + 1 −δ ( 1 −δ)+ 1 −λPP λNN ( 1+ δ) −1 1+ 1 −λPP λNN ) Alternatively, B μis also represented as a set of heat maps (or contour graphs). Each heat map (2D) represents the metric bias for a certain fixed value of the imbalance (let us say δ0 ), thereby making bias dependent on two variables ( λPP , λNN ) and on one constant ( δ0 ). Hence, B μ= B μ( λPP , λNN , δ0 ) . Nevertheless, drawing conclusions regarding a 4-dimensional space is, in most cases, a challenging task. Several partial prospects are therefore proposed that reduce the bias function dimensionality. In this respect, the first approach involves considering the Table 4 Bias indicators for singular classifiers. Singular classifier Symbol σB μ( δ) Worst classifier wcB μ( δ) lim ε→ 0 B μ( ε , ε , δ) Best classifier bcB μ( δ) lim ε→ 0 B μ( 1 −ε, 1 −ε, δ) Worst-positive classifier wpcB μ( δ) lim ε→ 0 B μ( ε, 1 −ε, δ) Worst-negative classifier wncB μ( δ) lim ε→ 0 B μ( 1 −ε , ε , δ) Medium classifier mcB μ( δ) B μ(0.5, 0.5, δ) metric bias for classifiers whose performance is singularly located on the ( λPP , λNN ) plane, thereby obtaining a set of bias indicators which are generically denoted as σB μ( δ). The singular classifiers and their formulations are proposed in Table 4. An alternative way to reduce the dimensionality of B μis through the consideration that λPP and λNN are randomly and uniformly distributed within the [0, 1] range. Bias can therefore be seen for each value of the imbalance coefficient δas a random variable B μ( δ), which is characterized by its probability density function ( pdf [ B μ( δ)]). Additionally, several local statistical indicators can be defined, which are generically denoted as ψB μ( δ). The term local attributed to these indicators (summarized in Table 5 ) means that they are defined for each value of δ. Definitions in Tables 4 and 5 have introduced several prospects of bias, all of which depend on the imbalance δ. They are gener- 220 A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 Table 5 Local statistical indicators on bias. Local statistical indicator Symbol ψB μ( δ) Mean mB μ( δ) ∫ ∫ B μ( λPP , λNN , δ) d λPP d λNN Standard deviation sdB μ( δ)    [ B μ( λPP , λNN , δ) −m B μ(δ) ] 2 d λPP d λNN Root-Mean-Square Bias rmsB μ( δ)    B 2 μ( λPP , λNN , δ) d λPP d λNN Maximum absolute value maxaB μ( δ) max 0 ≤λPP ≤1 0 ≤λNN ≤1 | B μ( λPP , λNN , δ) | Skewness skB μ( δ) 1 sdB 3 μ(δ)    [ B μ( λPP , λNN , δ) −m B μ(δ) ] 3 d λPP d λNN Kurtosis kB μ( δ) 1 sdB 4 μ(δ)    [ B μ( λPP , λNN , δ) −m B μ(δ) ] 4 d λPP d λNN Table 6 Global statistical indicators on bias. Global statistical indicator Symbol B μ( δ) Mean MB μ1 2    B μ( λPP , λNN , δ) d λPP d λNN dδ Standard deviation SDB μ 1 2    [ B μ( λPP , λNN , δ) −M B μ] 2 d λPP d λNN dδ Root-Mean-Square Bias RMSB μ 1 2    B 2 μ( λPP , λNN , δ) d λPP d λNN dδ Maximum absolute value MAXAB μmax 0 ≤λPP ≤1 0 ≤λNN ≤1 0 ≤δ≤1 | B μ( λPP , λNN , δ) | Skewness SKB μ1 SDB 3 μ 1 2    [ B μ( λPP , λNN , δ) −M B μ] 3 d λPP d λNN dδ Kurtosis KB μ1 SDB 4 μ 1 2    [ B μ( λPP , λNN , δ) −M B μ(δ) ] 4 d λPP d λNN dδ ically denoted as x B μ(δ) = { σB μ(δ) , ψ B μ(δ) } . It should be observed that these are not numbers but functions. In order to obtain a single value derived from these functions, we can make the hypothesis that δis randomly and uniformly distributed within the [ −1 , 1 ] range. Mean values of each function can therefore be computed as xB μ≡1 2  1 −1 x B μ( δ) dδ. (17) Another way to reduce the generic function xB μ( δ) to a single value is by focusing on its value for extremely positive-imbalanced datasets where δ→ 1. A generic value can therefore be obtained through the expression xB εP μ≡lim ε→ 0 x B μ( 1 −ε ) . (18) The corresponding negative counterpart is defined as xB εN μ≡lim ε→ 0 x B μ( −1 + ε ) . (19) A global regard of bias can also be undertaken by considering that λPP and λNN are randomly and uniformly distributed within the [0, 1] range, and also that δlies within the [ −1 , 1 ] range. Bias can now be seen as a random variable B μthat is independent of the imbalance coefficient δ. Bias can therefore be characterized by its probability density function ( pdf [ B μ]). Based on this overall pdf , several global statistical indicators can be defined, which are generically denoted as B μand are summarized in Table 6 . Definitions in Eqs. (17) –(19) and in Table 6 have introduced several single-valued indicators regarding bias which will generically be denoted as X B μ(δ) = { xB μ, xB εP μ, xB εN μ, B μ(δ) } . Throughout this subsection, several function and single-valued indicators have been introduced to assess the impact of dataset imbalance on classification performance metrics. A summary of these indicators is depicted in Fig. 3 . 3. Results 3.1. Performance metric bias function The methods described in Section 2 will now be applied to the ten metrics defined in Table 1 . As explained above, bias depends on three variables, that is, B μ= B μ( λPP , λNN , δ) and its formulation for the selected metrics is also shown in Table 3 . The first approach for their representation is based on the heat volumes as depicted in Fig. 4 , where each point in the 3-dimensional ( λPP , λNN , δ) space has a bias-dependent colour. In certain performance metrics (for instance in MCCn ), bias has low values for many points in the ( λPP , λNN , δ) space. In these cases, the expressive power of the whole range of colours is not completely exploited. It is therefore better to select the colour of each point not directly based on bias, but on the relative value of bias within the range of values for their corresponding metric. The colour-map is then rescaled to show the relative bias with the value −1 corresponding to the minimum bias, and the value +1 to the maximum. The result is depicted in Fig. 5 , where the range of each metric is shown in its corresponding subplot title. As explained above, B μcan also be represented as a set of heat maps. Each heat map represents the metric bias for a certain fixed value of the imbalance B μ= B μ( λPP , λNN , δ0 ) . The result for precision ( PRC ) is portrayed in Fig. 6 . Similar graphics can be obtained for the remaining metrics. Now let us suppose that the value of δis known, for instance δ= δ0 = 0 . 95 . Therefore B μ= B μ( λPP , λNN , δ0 ) depends on only two variables ( λPP and λNN ) and can be represented as a heat map. The results for each metric are shown in Fig. 7 where the colours represent the absolute value of bias. A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 221 Fig. 3. Functions and single-valued indicators assessing bias in performance metrics due to class imbalance. Fig. 4. Heat volumes of bias for each performance metric. This information can also be presented in the form of contour graphs as in Fig. 8 where the colours represent the absolute value of bias. 3.2. Bias indicators depending on δ In the previous section, several bias indicators depending only on the imbalance coefficient δwere defined. As mentioned therein, the first approach is to consider classifiers whose performance is singularly located on the ( λPP , λNN ) plane. The results obtained for each bias indicator and performance metric σB μ( δ) are summarized in Table 7 , where unbiased performance metrics ( SNS, SPC, GM, BMn ) have been omitted. Table 7 Bias indicators for singular classifiers: σB μ( δ). PRC NPV ACC F 1 MCCn MKn wcB μ0 0 0 0 0 0 bcB μ0 0 0 0 0 0 wpcB μδ/2 −δ/ 2 −δ/ 2 0 0 0 wncB μδ/2 −δ/ 2 δ/2 2( 1+ δ) 3+ δ−2 3 0 0 mcB μδ/2 −δ/ 2 0 1+ δ 2+ δ−1 2 0 0 It can be observed that only four types of non-null indicators appear. Their dependence on δis plotted in Fig. 9 . Alternatively, it can be assumed that λPP and λNN are randomly and uniformly distributed across the [0, 1] range and the bias 222 A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 Fig. 5. Heat volumes of relative bias for each performance metric. Fig. 6. Set of heat maps of bias for precision. B μ( δ) can then be statistically characterized. First, the probability density function ( pdf [ B μ( δ)]) is derived for each performance metric. The results are shown in Fig. 10 , where, for each δand each bias B μ, the values of pdf are shown using different colours (dark blue corresponds to pdf = 0 ). Additionally, several local statistical indicators have been defined (see Table 5 ) and these are generically denoted as ψB μ( δ). The results obtained for each performance metric are depicted in Fig. 11 . For easier reading, unbiased performance metrics ( SNS, SPC, GM, BMn ) have been omitted. Moreover, NPV presents symmetric behaviour to PRC and has also been disregarded from the graphs for the sake of simplicity. 3.3. Single-valued bias indicators As has already been pointed out, there are various ways to obtain single-valued bias indicators. First, we consider bias functions σB μ( δ) as summarized in Table 7 . By assuming that δis randomly and uniformly distributed within the [ −1 , 1 ] range, the mean values of each measure can be computed. The results for each performance metric are shown in Table 8 . Now let us consider bias functions ψB μ( δ) depicted in Fig. 11 . By applying the same method, mean values of these measures can also be obtained. Results for each metric are shown in Table 9 . A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 223 Fig. 7. Heat maps of bias for each performance metric ( δ= 0 . 95 ). Fig. 8. Contour graphs of bias for each performance metric ( δ= 0 . 95 ). Table 8 Mean values of bias functions for singular classifiers σB μ( δ). Symbol PRC NPV ACC F 1 MCCn MKn 1 wcB μ0 0 0 0 0 0 2 bcB μ0 0 0 0 0 0 3 wpcB μ0 0 0 0 0 0 4 wncB μ0 0 0 −0 . 053 0 0 5 mcB μ0 0 0 −0 . 049 0 0 Table 9 Mean values of bias of local statistical indicators ψB μ( δ). Symbol PRC NPV ACC F 1 MCCn MKn 6 mB μ0 0 0 −0 . 041 0 0 7 sdB μ0.082 0.082 0.102 0.066 0.038 0.066 8 rmsB μ0.228 0.228 0.102 0.135 0.038 0.066 9 maxaB μ0.308 0.308 0.25 0.244 0.090 0.154 10 skB μ0 0 0 0.129 0 0 11 kB μ0.080 0.080 −0 . 6 −1 . 093 0.006 0.028 Instead of computing the mean of a function, single-valued bias indicators can also be obtained by focusing on their value for extremely imbalanced datasets. By first considering the bias metrics of singular classifiers ( σB μ( δ)), their results are shown in Table 10 . Regarding the extremely imbalanced case for local statistical indicators ( ψB μ( δ)), their results are shown in Table 11 . For a global view of bias, let us consider that λPP and λNN are randomly and uniformly distributed across the [0, 1] range while δlies within the [ −1 , 1 ] range, and then bias B μ is statistically characterized. First, the probability density function ( pdf ( B μ)) is derived for each performance metric. The results are shown in Fig. 12 , where NPV has been omitted from the graph because it shows symmetric behaviour to PRC and has the same pdf . Finally, several global statistical indicators based on pdf ( B μ) have been defined (see Table 6 ), which are generically denoted as B μ. The results obtained for each performance metric are shown in Table 12 . 224 A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 Table 10 Bias indicators on singular classifiers ( σB μ( δ)) for extremely imbalanced datasets. Symbol PRC NPV ACC F 1 MCCn MKn Extremely positiveimbalanced 12 wcB εP μ0.667 0 0 0 0.211 0.333 13 bcB εP μ0 −0 . 667 0 0 −0 . 211 −0 . 333 14 wpcB εP μ0.5 −0 . 5 −0 . 5 0 0 0 15 wncB εP μ0.5 −0 . 5 0.5 0.333 0 0 16 mcB εP μ0.5 −0 . 5 0 0.167 0 0 Extremely negativeimbalanced 17 wcB εN μ0 0.667 0 0 −0 . 211 0.333 18 bcB εN μ−0.667 0 0 −0 . 5 0.211 −0 . 333 19 wpcB εN μ−0 . 5 0.5 0.5 0 0 0 20 wncB εN μ−0 . 5 0.5 −0 . 5 −0 . 667 0 0 21 mcB εN μ−0 . 5 0.5 0 −0 . 5 0 0 Table 11 Bias measures on local statistical indicators ( ψB μ( δ)) for extremely imbalanced datasets. Symbol PRC NPV ACC F 1 MCCn MKn Extremely positiveimbalanced 22 mB εP μ0.5 −0.5 0 0.137 0 0 23 sdB εP μ0.238 0.238 0.204 0.088 0.213 0.226 24 rmsB εP μ0.554 0.554 0.204 0.163 0.213 0.226 25 maxaB εP μ 1 1 0.5 0.333 0.5 0.5 26 skB εP μ0 0 0 0.244 0 0 27 kB εP μ−0 . 651 −0 . 651 −0 . 6 −1 . 043 −0 . 790 −1 . 014 Extremely negativeimbalanced 28 mB εN μ−0 . 5 0.5 0 −0 . 477 0 0 29 sdB εN μ0.238 0.238 0.204 0.241 0.213 0.226 30 rmsB εN μ0.554 0.554 0.204 0.534 0.213 0.226 31 maxaB εN μ 1 1 0.5 1 0.5 0.5 32 skB εN μ0 0 0 0.168 0 0 33 kB εN μ−0 . 651 −0 . 651 −0 . 6 −0 . 933 −0 . 790 −1 . 014 Fig. 9. Four types of bias indicators for singular classifiers. Table 12 Global statistical indicators of bias B μfor each performance metric. Symbol PRC NPV ACC F 1 MCCn MKn 34 MB μ0 0 0 −0 . 041 0 0 35 SDB μ0.271 0.271 0.118 0.169 0.055 0.086 36 RMSB μ0.271 0.271 0.118 0.174 0.055 0.086 37 MAXAB μ1 1 0.5 1 0.5 0.5 38 SKB μ0 0 0 −1 . 269 0 0 39 KB μ−0 . 046 −0 . 046 1.32 2.043 6.9 3.061 In this section, 39 single-valued bias indicators have been considered, which can also be presented in a graphical form, as shown in Fig. 13 . Here, the bias indicators σB μ(δ) , detailed in Table 8 , have been omitted, since all these indicators are null (except for F 1 score) and therefore their comparison would be meaningless. 3.4. Symmetry of bias functions In order to categorize the bias functions for each performance metric, it is convenient to study their symmetry. For this purpose, let us begin by considering the Matthews Correlation Coefficient ( MCC ), since this coefficient is particularly clear. Its heat map for an imbalance coefficient δ= 0 . 95 is shown in the upper left-hand-side plot of Fig. 14 . A first anti-clockwise 90 °rotation on the ( λPP , λNN ) plane is performed, and the result is shown in the upper right-hand-side plot. The result of a second 90 ºrotation is shown in the lower right-hand-side graph. Finally, the sign of the bias values is changed, as shown in the lower left-hand-side plot. It can be observed that the result coincides with the original heat map. The 180 ºrotation enables the bias to be defined on a new set of axes ( PP , NN ), which are related to the original ( λPP , λNN ) through the expressions PP = 1 −λPP ; NN = 1 −λNN . This symmetry can therefore be formalized as B MCC ( λPP , λNN , δ) = −B MCC ( PP , NN , δ) = −B MCC ( 1 −λPP , 1 −λNN , δ) . (20) Hence, the MCC bias function shows an order-2 (180 º) rotational odd symmetry (or anti-symmetry) on the ( λPP , λNN ) plane. Furthermore, this bias function shows symmetry with respect to the principal diagonal on the ( λPP , λNN ) plane if the sign of δis inverted, that is, B MCC ( λNN , λPP , −δ) = B MCC ( λPP , λNN , δ) . (21) This dual behaviour is called Type I symmetry. Bias functions for ACC and MK also exhibit this type of symmetry. When the bias of precision ( PRC ) is considered, no symmetry on the ( λPP , λNN ) plane can be found. However, it exhibits a symmetry in the ( λPP , λNN , δ) space as can be observed in Fig. 15 where its heat volume for an imbalance coefficient δ= 0 . 95 is shown in the upper left-hand-side plot. First, a mirror symmetry, with respect to the δ= 0 plane, is performed and the result is A. Luque, A. Carrasco and A. Martín et al. / Pattern Recognition 91 (2019) 216–231 231 [33] H. Núñez , L. Gonzalez-Abril , C. Angulo , Improving SVM classification on imbalanced datasets by introducing a new bias, J. Classification 34 (3) (2017) 427–443 . [34] D.M. Powers , Evaluation: from precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation, School of Informatics and Engineering, Flinders University, Adelaide, Australia, 2011 Technical Report SIE-07-001 . [35] B. Raman , T.R. Ioerger , Enhancing Learning Using Feature and Example Selection, Texas A&M University, College Station, TX, USA, 2003 . [36] M. Sokolova , N. Japkowicz , S. Szpakowicz , Beyond accuracy, F-score and ROC: a family of discriminant measures for performance evaluation, in: Australasian Joint Conference on Artificial Intelligence, Berlin, Heidelberg, Springer, 2006, December, pp. 1015–1021 . [37] M. Sokolova , G. Lapalme , A systematic analysis of performance measures for classification tasks, Inf. Process. Manage. 45 (4) (2009) 427–437 . [38] S.V. Stehman , Selecting and interpreting measures of thematic classification accuracy, Remote Sens. Environ. 62 (1) (1997) 77–89 . [39] Y. Sun , M.S. Kamel , A.K. Wong , Y. Wang , Cost-sensitive boosting for classification of imbalanced data, Pattern Recognit. 40 (12) (2007) 3358–3378 . [40] X. Yuan , L. Xie , M. Abouelenien , A regularized ensemble framework of deep learning for cancer detection from multi-class, imbalanced training data, Pattern Recognit. 77 (2018) 160–172 . [41] W. Zong , G.B. Huang , Y. Chen , Weighted extreme learning machine for imbalance learning, Neurocomputing 101 (2013) 229–242 . Amalia Luque received her Industrial Engineering degree in 2007, Master in Automation in 2010 and her Doctoral Degree in 2014. She has been involved in teaching related to Project Engineering at the University of Seville since 2015. Her main areas of research are business intelligence, data mining and feature extraction. Alejandro Carrasco received his Computer Engineering degree in 1998 and his Ph.D. in 2003. He is a Lecturer and Researcher at the University of Seville since 1998, has founded a NTBF and directed several R&D projects. His main areas of research are data mining, industrial computing and network security. Alejandro M. Martín-Gómez, B. Industrial Electronics and Automation Engineer, MS Product and facilities design and development, MS Security and health and PhD Student (manufacturing engineering). He worked across a wide variety of projects of industrial sector. He is assistant professor at University of Seville (Field of Knowledge: Engineering Project). Ana de las Heras García de Vinuesa is a BSc. Eng Industrial Design, MSc. Ecodesign, and PhD. in Manufacturing and Environmental Engineering. She works as an assistant professor in the Department of Design Engineering, Engineering Project area, at University of Seville, Spain.