Exploring Symmetry of Binary Classification Performance Metrics

symmetry S S Article Exploring Symmetry of Binary Classification Performance Metrics Amalia Luque 1,* , Alejandro Carrasco 2, Alejandro Martín1and Juan Ramón Lama 1 1Ingeniería del Diseño; Escuela Politécnica Superior. Universidad de Sevilla, 41011 Sevilla, Spain; [email protected] (A.M.); [email protected] (J.R.L.) 2Tecnología Electrónica; Escuela Ingeniería Informática. Universidad de Sevilla, 41012 Sevilla, Spain; [email protected] *Correspondence: [email protected]; Tel.: +34-955-420-187 Received: 14 November 2018; Accepted: 24 December 2018; Published: 3 January 2019   Abstract: Selecting the proper performance metric constitutes a key issue for most classification problems in the field of machine learning. Although the specialized literature has addressed several topics regarding these metrics, their symmetries have yet to be systematically studied. This research focuses on ten metrics based on a binary confusion matrix and their symmetric behaviour is formally defined under all types of transformations. Through simulated experiments, which cover the full range of datasets and classification results, the symmetric behaviour of these metrics is explored by exposing them to hundreds of simple or combined symmetric transformations. Cross-symmetries among the metrics and statistical symmetries are also explored. The results obtained show that, in all cases, three and only three types of symmetries arise: labelling inversion (between positive and negative classes); scoring inversion (concerning good and bad classifiers); and the combination of these two inversions. Additionally, certain metrics have been shown to be independent of the imbalance in the dataset and two cross-symmetries have been identified. The results regarding their symmetries reveal a deeper insight into the behaviour of various performance metrics and offer an indicator to properly interpret their values and a guide for their selection for certain specific applications. Keywords: performance metrics; classification; computational symmetry; machine learning 1. Introduction Symmetry has played and continues playing, a highly significant role in the way of how humans perceive the world [ 1 ]. In the scientific fields, symmetry plays a key role as it can be discovered in nature [ 2 , 3 ], society [ 4 ] and mathematics [ 5 ]. Moreover, symmetry also provides an intuitive way to attain faster and deeper insights into scientific problems. In recent years, an increasing interest has arisen in detecting and taking advantage of symmetry in various aspects of theoretical and applied computing [ 6 ]. Several studies involving symmetry have been published in network technology [ 7 ], human interfaces [ 8 ], image processing [ 9 ], data hiding [ 10 ] and many other applications [11]. On the other hand, pattern recognition and machine learning procedures are becoming key aspects of modern science [ 12 ] and the hottest topics in the scientific literature on computing [ 13 ]. Furthermore, in this field, symmetry is playing an interesting role either as a subject of study, in the form of machine learning algorithms to discover symmetries [ 14 ] or as a means to improve the results obtained by automatic recognition systems [ 15 ]. Let us emphasize this point: not only can knowing the symmetry of a certain computer algorithm be intrinsically rewarding since it sheds light on the behaviour of the algorithm but it can also be very useful for its interpretation, its optimization or as a Symmetry 2019,11, 47; doi:10.3390/sym11010047 www.mdpi.com/journal/symmetry Symmetry 2019,11, 47 2 of 31 criterion for the selection among various competing algorithms. As an example, in recent research, we have employed a symmetric criterion to select the best feature-extraction procedures (Discrete Cosine Transform versus Discrete Fourier Transform) [ 16 ] in an application of the classification of sounds [ 17 , 18 ] effectively deployed in a Wireless Sensor Network as shown in Figure 1. Another examples of industrial applications using classification of sounds can be found in Refs. [19,20]. Symmetry 2018, 10, x FOR PEER REVIEW 2 of 31 in recent research, we have employed a symmetric criterion to select the best feature-extraction procedures (Discrete Cosine Transform versus Discrete Fourier Transform) [16] in an application of the classification of sounds [17,18] effectively deployed in a Wireless Sensor Network as shown in Figure 1. Another examples of industrial applications using classification of sounds can be found in Refs. [19,20]. Figure 1. Node of the Wireless Sensor Network where the symmetry of classification performance metrics has been primarily applied. In the broad field of machine learning, the study of how to measure the performance of various classifiers has attracted continued attention [21–23]. Classification performance metrics play a key role in the assessment of the overall classification process in the test phase, in the selection from among various competing classifiers in the validation phase and are even sometimes used as the loss function to be optimized in the process of model construction during the classification training phase. However, to the best of our knowledge, no systematic study into the symmetry of these metrics has yet been undertaken. By discovering their symmetries, we would reach a better understanding of their meaning, we could obtain useful insights into when their use would be more appropriate and we would also gain additional and meaningful indicators for the selection of the best performance metric. Although several dozen performance metrics can be found in the literature, we will focus on those which are probably the most commonly used: the metrics based on the confusion matrix [24]. Accuracy, precision and recall (sensitivity) are undoubtedly some of the most popular metrics. On the other hand, our research will be focused on the cases where there are only two classes (binary classifiers). Although this is certainly a limitation, it does provide a solid ground base for further research. Moreover, multiclass performance metrics are usually obtained by decomposing the multiclass problem into several binary classification sub-problems [25]. 2. Materials and Methods 2.1. Definitions Let us first consider an original (baseline) experiment 𝐸, defined by the duple 𝐸=󰇛𝐶,𝐷) composed of a set of 𝑛 classifiers, 𝐶=󰇝𝑐󰇞 and a set of their corresponding 𝑛 datasets, 𝐷= 󰇝𝐷󰇞,𝑖=1,⋯,𝑛. The elements in every dataset belong to either of two classes, 𝐺 and 𝐺, which are called Positive (𝑃) and Negative (𝑁) classes, respectively. The 𝑖-th classifier 𝑐 operates on the corresponding 𝐷 dataset, thereby obtaining a resulting classification which can be defined by its binary confusion matrix 𝑐𝑚 and hence 𝐷  󰇒 󰇏 𝑐𝑚. The set of confusion matrices are denominated Figure 1. Node of the Wireless Sensor Network where the symmetry of classification performance metrics has been primarily applied. In the broad field of machine learning, the study of how to measure the performance of various classifiers has attracted continued attention [ 21 – 23 ]. Classification performance metrics play a key role in the assessment of the overall classification process in the test phase, in the selection from among various competing classifiers in the validation phase and are even sometimes used as the loss function to be optimized in the process of model construction during the classification training phase. However, to the best of our knowledge, no systematic study into the symmetry of these metrics has yet been undertaken. By discovering their symmetries, we would reach a better understanding of their meaning, we could obtain useful insights into when their use would be more appropriate and we would also gain additional and meaningful indicators for the selection of the best performance metric. Although several dozen performance metrics can be found in the literature, we will focus on those which are probably the most commonly used: the metrics based on the confusion matrix [ 24 ]. Accuracy, precision and recall (sensitivity) are undoubtedly some of the most popular metrics. On the other hand, our research will be focused on the cases where there are only two classes (binary classifiers). Although this is certainly a limitation, it does provide a solid ground base for further research. Moreover, multiclass performance metrics are usually obtained by decomposing the multiclass problem into several binary classification sub-problems [25]. 2. Materials and Methods 2.1. Definitions Let us first consider an original (baseline) experiment EB , defined by the duple EB=CB,DB composed of a set of nB classifiers, CB=cB i and a set of their corresponding nB datasets, DB=DB i,i=1, ··· ,nB. The elements in every dataset belong to either of two classes, G1 and G2 , which are called Positive ( P ) and Negative ( N ) classes, respectively. The i -th classifier cB i operates on the corresponding DB i dataset, thereby obtaining a resulting classification which can be defined by its binary confusion matrix cmB i and hence DB i cB i →cmB i . The set of confusion matrices are denominated Symmetry 2019,11, 47 3 of 31 CMB=cmB i . The baseline experiment can therefore be defined as the set of classifiers operating on the set of datasets to obtain a set of confusion matrices, EB:DBCB →CMB. This paper will explore the behaviour of binary classification performance metrics when the original experiment is subject to nE different types of transformations. Let us define the k -th transformed experiment Ek=Ck,Dk composed of a set of nk classifiers, Ck=nck io and a set of their corresponding nk datasets, Dk=nDk io , whose result is a set of confusion matrices CMk=ncmk io . Hence, Ek:DkCk →CMk , where k=B, 1, 2, ··· ,nE , indicates the type of transformation. In the k -th experiment, when the i -th classifier ck i operates on its corresponding Dk i dataset, the result is summarized in the binary confusion matrix defined as cmk i="ak ifk i gk ibk i#, (1) where •ak iis the number of positive elements in Dk icorrectly classified as positive; •bk iis the number of negative elements in Dk icorrectly classified as negative; •fk iis the number of positive elements in Dk iincorrectly classified as negative; and •gk iis the number of negative elements in Dk iincorrectly classified as positive. Let us call Pk i , Nk iand Mk i the positive, negative and total number of elements in Dk i . Therefore Mk i=Pk i+Nk i,ak i+fk i=Pk iand gk i+bk i=Nk i. The confusion matrix can then be described as cmk i="ak iPk i−ak i Nk−bk ibk i#. (2) Let us now define αk i as the ratio of positive elements in Dk i correctly classified as positive; and βk i as the ratio of negative elements in Dk icorrectly classified as negative. That is, αk i≡ak i Pk i ,βk i≡bk i Nk i . (3) The confusion matrix can therefore be rewritten as cmk i="αk iPk iPk i−αk iPk i Nk i−βk iNk iβk iNk i#=  αk iPk i1−αk iPk i 1−βk iNk iβk iNk i . (4) On the other hand, a dataset Dk i is called imbalanced if it has a different number of positive and negative elements, that is, Pk i6=Nk i . Classification on the presence of imbalanced datasets is a challenging task requiring specific considerations [ 26 ]. To quantify the imbalance, several indicators have been proposed, such as the dominance [ 27 , 28 ], the proportion between positive and negative instances (formalized as 1 :X ) [ 29 ] and the imbalance ratio ( IR ) defined as Pk i/Nk i [ 30 ], which is also called skew [ 31 ]. This value lies within the [0, ∞] range and has a value IR = 1 in the balanced case. We prefer to use an indicator showing a value 0 in the balanced case, a value + 1 when all the elements in the dataset are positive and − 1 if all the elements are negative. We define the imbalance coefficient δk i, which is an indicator that has these characteristics, as δk i≡2Pk i Mk i−1. (5) Symmetry 2019,11, 47 4 of 31 The imbalance coefficient is graphically shown in Figure 2(solid blue cline) as a function of the proportion of positive elements in the dataset. For the sake of comparison, that figure also shows the IR imbalance ratio (dashed green line). Symmetry 2018, 10, x FOR PEER REVIEW 4 of 31 Figure 2. Imbalance coefficient (solid blue line) and imbalance ratio (dashed green line) vs. the proportion of positive elements in the dataset. Based on the imbalance coefficient, the number of positive and negative elements in the dataset can be rewritten as 𝑃=1+𝛿 2𝑀. (6) 𝑁=𝑀−𝑃=𝑀󰇧1−1+𝛿 2󰇨=1−𝛿 2𝑀. (7) By substituting these expressions into Equation (4), the confusion matrix becomes 𝑐𝑚=󰇯 𝛼  𝑀1−𝛼  𝑀 1−𝛽  𝑀𝛽  𝑀󰇰=𝜆𝑀, (8) where 𝜆 is the unitary confusion matrix defined as 𝜆≡⎣ ⎢ ⎢ ⎡ 𝛼1+𝛿 21−𝛼1+𝛿 2 1−𝛽1−𝛿 2𝛽1−𝛿 2⎦ ⎥ ⎥ ⎤ . (9) It can be seen that 𝜆 is a function of 3 variables: the ratio of positive 𝛼 and negative 𝛽 correctly classified elements and the imbalance coefficient 𝛿, that is, 𝜆=𝜆𝛼,𝛽,𝛿. In order to measure the performance of the classification process, 𝑚 metrics are used. In this paper we focus on metrics that are based on the unitary confusion matrix and, for the sake of much easier comparison, all these metrics are converted within the range [−1,1]. Let us define 𝛾  as the 𝑗-th of such metrics for the 𝑐 classifier operating on the 𝐷 dataset, where 𝑗=1,…,𝑚. Since it is based on the unitary confusion matrix, 𝛾 =𝛾  𝜆= 𝛾 𝛼,𝛽,𝛿. Let us now define 𝜇 as the set of the 𝑗-th metric values corresponding to the 𝑘-th experiment 𝐸=󰇛𝐶,𝐷), that is, 𝜇≡𝛾 ,𝑖=1,2,⋯,𝑛. Additionally, the sets 𝛼≡𝛼, 𝛽≡ 𝛽 𝑎𝑛𝑑 𝛿≡𝛿 are also defined. 2.2. Representation of Metrics. With these definitions, it is clear that the metric 𝜇=𝜇󰇛𝛼,𝛽,𝛿) and hence it is a 4-dimensional function since 𝜇 (one dimension) depends on 𝛼,𝛽 𝑎𝑛𝑑 𝛿 (three independent dimensions). To depict their values, a first approach could involve a 3D representation space where each 𝛼,𝛽,𝛿 point is color-coded according to the value 𝛾 𝛼,𝛽,𝛿. To show the different types of representations, let us define an arbitrary metric function Figure 2. Imbalance coefficient (solid blue line) and imbalance ratio (dashed green line) vs. the proportion of positive elements in the dataset. Based on the imbalance coefficient, the number of positive and negative elements in the dataset can be rewritten as Pk i=1+δk i 2Mk i. (6) Nk i=Mk i−Pk i=Mk i 1−1+δk i 2!=1−δk i 2Mk i. (7) By substituting these expressions into Equation (4), the confusion matrix becomes cmk i=  αk i 1+δk i 2Mk i1−αk i1+δk i 2Mk i 1−βk i1−δk i 2Mk iβk i 1−δk i 2Mk i =λk iMk i, (8) where λk iis the unitary confusion matrix defined as λk i≡  αk i 1+δk i 21−αk i1+δk i 2 1−βk i1−δk i 2βk i 1−δk i 2 . (9) It can be seen that λk i is a function of 3 variables: the ratio of positive αk i and negative βk i correctly classified elements and the imbalance coefficient δk i, that is, λk i=λk iαk i,βk i,δk i. In order to measure the performance of the classification process, m metrics are used. In this paper we focus on metrics that are based on the unitary confusion matrix and, for the sake of much easier comparison, all these metrics are converted within the range [−1, 1] . Let us define jγk i as the j -th of such metrics for the ck i classifier operating on the Dk i dataset, where j= 1, . . . , m . Since it is based on the unitary confusion matrix, jγk i=jγk iλk i=jγk iαk i,βk i,δk i. Let us now define µk j as the set of the j -th metric values corresponding to the k -th experiment Ek=Ck,Dk , that is, µk j≡njγk io , i= 1, 2, ··· , nk . Additionally, the sets αk≡nαk io , βk≡nβk ioand δk≡nδk ioare also defined. Symmetry 2019,11, 47 5 of 31 2.2. Representation of Metrics With these definitions, it is clear that the metric µk j=µk jαk,βk,δk and hence it is a 4-dimensional function since µk j (one dimension) depends on αk , βkand δk (three independent dimensions). To depict their values, a first approach could involve a 3D representation space where each αk i,βk i,δk i point is color-coded according to the value jγk iαk i,βk i,δk i. To show the different types of representations, let us define an arbitrary metric function µk jαk,βk,δk=sin2παkδk+sin2πβk 2. (10) This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results. Figure 3depicts the 3D representation for said example function. The nE= 1000 pairs of classifiers and datasets used in the experiment Ek=Ck,Dk are selected in such a way that the space αk,βk,δk is covered with equally spaced points. The above figure may cause confusion, mainly when the number of points (nE) increases. An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient. Figure 4a depicts such a slice in the 3D graphic for an arbitrary value δ=0.75 and Figure 4b shows the slice on a 2D plane. Symmetry 2018, 10, x FOR PEER REVIEW 5 of 31 𝜇󰇛𝛼,𝛽,𝛿)=sin󰇛2𝜋𝛼𝛿)+sin󰇛2𝜋𝛽) 2. (10) This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results. Figure 3 depicts the 3D representation for said example function. The 𝑛 =1000 pairs of classifiers and datasets used in the experiment 𝐸=󰇛𝐶,𝐷) are selected in such a way that the space 󰇛𝛼,𝛽,𝛿) is covered with equally spaced points. The above figure may cause confusion, mainly when the number of points (𝑛) increases. An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient. Figure 4a depicts such a slice in the 3D graphic for an arbitrary value 𝛿=0.75 and Figure 4b shows the slice on a 2D plane. Figure 3. 3D representation of a 4-dimension metric value 𝜇  󰇛𝛼  ,𝛽  ,𝛿  ). The value of the metric 𝜇  is colour-coded for every point in the 󰇛𝛼  ,𝛽  ,𝛿  ) 3D space. (a) (b) Figure 4. Representation of a metric value 𝜇  󰇛𝛼  ,𝛽  ) for 𝛿=0.75. (a) Slice of the 3D graphic by a plane corresponding to 𝛿=0.75; (b) 2D representation of the slice. In the previous figure, the slice contains 100 values of the metric. However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended. For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets 󰇛𝐶,𝐷) in such a way that the plane 󰇛𝛼,𝛽) is fully covered. The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient (𝛿=0.75 in the example). Figure 3. 3D representation of a 4-dimension metric value µk jαk,βk,δk . The value of the metric µk j is colour-coded for every point in the αk,βk,δk3D space. Symmetry 2018, 10, x FOR PEER REVIEW 5 of 31 𝜇󰇛𝛼,𝛽,𝛿)=sin󰇛2𝜋𝛼𝛿)+sin󰇛2𝜋𝛽) 2. (10) This function is only used as an example, corresponds to no specific classification metric and has been selected for its aesthetic results. Figure 3 depicts the 3D representation for said example function. The 𝑛 =1000 pairs of classifiers and datasets used in the experiment 𝐸=󰇛𝐶,𝐷) are selected in such a way that the space 󰇛𝛼,𝛽,𝛿) is covered with equally spaced points. The above figure may cause confusion, mainly when the number of points (𝑛) increases. An alternative is to slice the 3D graphic by a plane corresponding to a certain value of the imbalance coefficient. Figure 4a depicts such a slice in the 3D graphic for an arbitrary value 𝛿=0.75 and Figure 4b shows the slice on a 2D plane. Figure 3. 3D representation of a 4-dimension metric value 𝜇  󰇛𝛼  ,𝛽  ,𝛿  ). The value of the metric 𝜇  is colour-coded for every point in the 󰇛𝛼  ,𝛽  ,𝛿  ) 3D space. (a) (b) Figure 4. Representation of a metric value 𝜇  󰇛𝛼  ,𝛽  ) for 𝛿=0.75. (a) Slice of the 3D graphic by a plane corresponding to 𝛿=0.75; (b) 2D representation of the slice. In the previous figure, the slice contains 100 values of the metric. However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended. For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets 󰇛𝐶,𝐷) in such a way that the plane 󰇛𝛼,𝛽) is fully covered. The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient (𝛿=0.75 in the example). Figure 4. Representation of a metric value µk jαk,βk for δ= 0.75. ( a ) Slice of the 3D graphic by a plane corresponding to δ=0.75; (b) 2D representation of the slice. In the previous figure, the slice contains 100 values of the metric. However, to obtain a clearer understanding of the metric behaviour, a much larger number of points is recommended. For this purpose, the experiment is designed by selecting a set of virtual pairs of classifiers and datasets Symmetry 2019,11, 47 6 of 31 Ck,Dk in such a way that the plane αk,βk is fully covered. The result, as shown in Figure 5, appears as a heat map for a certain value of the imbalance coefficient (δ=0.75 in the example). Symmetry 2018, 10, x FOR PEER REVIEW 6 of 31 Figure 5. Heat map of a metric value 𝜇󰇛𝛼,𝛽) for 𝛿=0.75. In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6. Figure 6. Panel of heat maps representing the metric 𝜇󰇛𝛼,𝛽,𝛿). 2.3. Transformations The original baseline experiment 𝐸=󰇛𝐶,𝐷) is subject to various types of transformations. As a result of the 𝑘-th transformation, the metrics related to the baseline experiment 𝜇󰇛𝛼,𝛽,𝛿) are transformed into 𝜇󰇛𝛼,𝛽,𝛿), which can be written either as 𝜇󰇛𝛼,𝛽,𝛿)= 𝑇𝜇󰇛𝛼,𝛽,𝛿) or as 𝜇󰇛𝛼,𝛽,𝛿) 󰇒 󰇏 𝜇󰇛𝛼,𝛽,𝛿). (11) It is said that the metric 𝜇 is symmetric under the transformation 𝑇 if 𝜇=𝜇. Conversely, 𝜇 is called antisymmetric under 𝑇 (or symmetric under the complementary transformation 𝑇) if 𝜇=−𝜇. Analogously, it is said that the metrics 𝜇 and 𝜇 are cross-symmetric under the transformation 𝑇 if 𝜇 =𝜇. Conversely, 𝜇 and 𝜇 are called anti-cross-symmetric under 𝑇 (or cross-symmetric under the complementary transformation 𝑇) if 𝜇 =−𝜇. α β 00.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 δ : -1 β δ : -0.95 δ : -0.9 δ : -0.75 δ : -0.5 δ : -0.25 β δ : -0.1 δ : 0 δ : 0.1 δ : 0.25 δ : 0.5 α β δ : 0.75 α δ : 0.9 α δ : 0.95 α δ : 1 α -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Figure 5. Heat map of a metric value µk jαk,βkfor δ=0.75. In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6. Symmetry 2018, 10, x FOR PEER REVIEW 6 of 31 Figure 5. Heat map of a metric value 𝜇󰇛𝛼,𝛽) for 𝛿=0.75. In order to analyse the behaviour of the metric for different values of the imbalance coefficient, a panel of heat maps can be used, as depicted in Figure 6. Figure 6. Panel of heat maps representing the metric 𝜇󰇛𝛼,𝛽,𝛿). 2.3. Transformations The original baseline experiment 𝐸=󰇛𝐶,𝐷) is subject to various types of transformations. As a result of the 𝑘-th transformation, the metrics related to the baseline experiment 𝜇󰇛𝛼,𝛽,𝛿) are transformed into 𝜇󰇛𝛼,𝛽,𝛿), which can be written either as 𝜇󰇛𝛼,𝛽,𝛿)= 𝑇𝜇󰇛𝛼,𝛽,𝛿) or as 𝜇󰇛𝛼,𝛽,𝛿) 󰇒 󰇏 𝜇󰇛𝛼,𝛽,𝛿). (11) It is said that the metric 𝜇 is symmetric under the transformation 𝑇 if 𝜇=𝜇. Conversely, 𝜇 is called antisymmetric under 𝑇 (or symmetric under the complementary transformation 𝑇) if 𝜇=−𝜇. Analogously, it is said that the metrics 𝜇 and 𝜇 are cross-symmetric under the transformation 𝑇 if 𝜇 =𝜇. Conversely, 𝜇 and 𝜇 are called anti-cross-symmetric under 𝑇 (or cross-symmetric under the complementary transformation 𝑇) if 𝜇 =−𝜇. α β 00.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 δ : -1 β δ : -0.95 δ : -0.9 δ : -0.75 δ : -0.5 δ : -0.25 β δ : -0.1 δ : 0 δ : 0.1 δ : 0.25 δ : 0.5 α β δ : 0.75 α δ : 0.9 α δ : 0.95 α δ : 1 α -1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1 Figure 6. Panel of heat maps representing the metric µk jαk,βk,δk. 2.3. Transformations The original baseline experiment EB=CB,DB is subject to various types of transformations. As a result of the k -th transformation, the metrics related to the baseline experiment µB jαB,βB,δB are transformed into µk jαk,βk,δk , which can be written either as µk jαk,βk,δk=Tk[µB jαB,βB,δB] or as µB jαB,βB,δBTk →µk jαk,βk,δk. (11) It is said that the metric µj is symmetric under the transformation Tk if µk j=µB j . Conversely, µj is called antisymmetric under Tk (or symmetric under the complementary transformation Tk ) if µk j=−µB j. Analogously, it is said that the metrics µu and µv are cross-symmetric under the Symmetry 2019,11, 47 7 of 31 transformation Tk if µk u=µB v . Conversely, µu and µv are called anti-cross-symmetric under Tk (or cross-symmetric under the complementary transformation Tk) if µk u=−µB v. 2.3.1. One-Dimensional Transformations One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric. Type α transformation implies that the i -th transformed classifier cα i shows a ratio of correctly classified positive elements αα i which has the symmetric value of the ratio αB i obtained by the baseline classifier cB i . Since the values of such ratios lie within the range [0, 1] , the symmetry exists with respect to the hyperplane α=0.5 and can be stated as αα i=1−αB i. An example of this transformation is depicted in Figure 7. Symmetry 2018, 10, x FOR PEER REVIEW 7 of 31 2.3.1. One-Dimensional Transformations. One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric. Type 𝛼 transformation implies that the 𝑖-th transformed classifier 󰇛𝑐) shows a ratio of correctly classified positive elements 󰇛𝛼) which has the symmetric value of the ratio 󰇛𝛼) obtained by the baseline classifier 󰇛𝑐). Since the values of such ratios lie within the range [0,1], the symmetry exists with respect to the hyperplane 𝛼=0.5 and can be stated as 𝛼=1−𝛼. An example of this transformation is depicted in Figure 7. Figure 7. Transformation type 𝛼 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝛼=0.5. Analogously, type 𝛽 transformation implies that the 𝑖-th transformed classifier 𝑐 shows a ratio of correctly classified negative elements 𝛽, which has the symmetric value of the ratio 󰇛𝛽) obtained by the baseline classifier 󰇛𝑐). Since the value of such ratios also lie within the range [0,1], the symmetry exists with respect to the hyperplane 𝛽=0.5 and can be stated as 𝛽=1−𝛽. An example of this transformation is depicted in Figure 8. Figure 8. Transformation type 𝛽 of a metric. (a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane 𝛽=0.5. Conversely, type 𝛿 transformation, which, instead of operating on classifiers, operates on datasets, implies that the 𝑖-th transformed dataset 𝐷 has an imbalance ratio 𝛿, which has the symmetric value of the imbalanced ratio 󰇛𝛿) in the baseline corresponding to dataset 󰇛𝛿). Since the value of such imbalance ratios lie within the range [−1,1], the symmetry exists with respect to the hyperplane 𝛿=0 and can be stated as 𝛿=−𝛿. An example of this transformation is depicted in Figure 9. Figure 7. Transformation type α of a metric. ( a ) Baseline metric. ( b ) Reflection symmetry with respect to the hyperplane α=0.5. Analogously, type β transformation implies that the i -th transformed classifier (cβ i) shows a ratio of correctly classified negative elements (ββ i) , which has the symmetric value of the ratio βB i obtained by the baseline classifier cB i . Since the value of such ratios also lie within the range [0, 1] , the symmetry exists with respect to the hyperplane β= 0.5 and can be stated as ββ i= 1 −βB i . An example of this transformation is depicted in Figure 8. Symmetry 2018, 10, x FOR PEER REVIEW 7 of 31 2.3.1. One-Dimensional Transformations. One-dimensional transformations is the name given to those mirror reflections with respect to a single (one and only one) dimension of the 4-dimensional performance metric. Type 𝛼 transformation implies that the 𝑖-th transformed classifier 󰇛𝑐) shows a ratio of correctly classified positive elements 󰇛𝛼) which has the symmetric value of the ratio 󰇛𝛼) obtained by the baseline classifier 󰇛𝑐). Since the values of such ratios lie within the range [0,1], the symmetry exists with respect to the hyperplane 𝛼=0.5 and can be stated as 𝛼=1−𝛼. An example of this transformation is depicted in Figure 7. Figure 7. Transformation type 𝛼 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝛼=0.5. Analogously, type 𝛽 transformation implies that the 𝑖-th transformed classifier 𝑐 shows a ratio of correctly classified negative elements 𝛽, which has the symmetric value of the ratio 󰇛𝛽) obtained by the baseline classifier 󰇛𝑐). Since the value of such ratios also lie within the range [0,1], the symmetry exists with respect to the hyperplane 𝛽=0.5 and can be stated as 𝛽=1−𝛽. An example of this transformation is depicted in Figure 8. Figure 8. Transformation type 𝛽 of a metric. (a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane 𝛽=0.5. Conversely, type 𝛿 transformation, which, instead of operating on classifiers, operates on datasets, implies that the 𝑖-th transformed dataset 𝐷 has an imbalance ratio 𝛿, which has the symmetric value of the imbalanced ratio 󰇛𝛿) in the baseline corresponding to dataset 󰇛𝛿). Since the value of such imbalance ratios lie within the range [−1,1], the symmetry exists with respect to the hyperplane 𝛿=0 and can be stated as 𝛿=−𝛿. An example of this transformation is depicted in Figure 9. Figure 8. Transformation type β of a metric. ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the hyperplane β=0.5. Conversely, type δ transformation, which, instead of operating on classifiers, operates on datasets, implies that the i -th transformed dataset Dδ i has an imbalance ratio δδ i , which has the symmetric value of the imbalanced ratio δB i in the baseline corresponding to dataset δB i . Since the value of such imbalance ratios lie within the range [−1, 1] , the symmetry exists with respect to the hyperplane δ=0 and can be stated as δδ i=−δB i. An example of this transformation is depicted in Figure 9. Symmetry 2019,11, 47 8 of 31 Symmetry 2018, 10, x FOR PEER REVIEW 8 of 31 Figure 9. Transformation type 𝛿 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝛿=0. Finally, type 𝜇 transformation jointly operates on classifiers and datasets in such a way that the 𝑗-th of performance metrics 𝛾 for the 𝑐 classifier operating on the 𝐷 dataset has the symmetric value of the performance metric in the baseline experiment 𝛾  . Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane 𝜇=0 and can be stated as 𝛾 =−𝛾 . An example of this transformation is depicted in Figure 10 where it should be noted that the 𝜇 dimension is shown by the colour code of each point. Therefore, an inversion in 𝜇 is shown as a colour inversion. Figure 10. Transformation type 𝜇 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝜇=0. 2.3.2. Multidimensional Transformations. Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric. Firstly, let us define type 𝜎 transformation as that which exchanges 𝛼 and 𝛽 dimensions. This implies that the 𝑖-th transformed classifier/dataset pair 󰇛𝑐,𝐷) shows a ratio of correctly classified positive elements 󰇛𝛼) which has the same value as the ratio of correctly classified negative elements 󰇛𝛽) obtained by the baseline classifier/dataset pair 󰇛𝑐,𝐷). This exchange can be seen as the symmetry with respect to the hyperplane 𝛼=𝛽 (main diagonal of the 𝛼,𝛽 plane) and can be stated as 𝛼=𝛽; 𝛽=𝛼. An example of this transformation is depicted in Figure 11. Figure 11. Transformation type 𝜎 of a metric. (a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane 𝛼=𝛽. Figure 9. Transformation type δ of a metric. ( a ) Baseline metric. ( b ) Reflection symmetry with respect to the hyperplane δ=0. Finally, type µ transformation jointly operates on classifiers and datasets in such a way that the j -th of performance metrics jγµ i for the cµ i classifier operating on the Dµ i dataset has the symmetric value of the performance metric in the baseline experiment (jγB i) . Since the value of such metrics lie within the range [−1, 1] , the symmetry exists with respect to the hyperplane µ= 0 and can be stated as jγµ i=−jγB i . An example of this transformation is depicted in Figure 10 where it should be noted that the µ dimension is shown by the colour code of each point. Therefore, an inversion in µ is shown as a colour inversion. Symmetry 2018, 10, x FOR PEER REVIEW 8 of 31 Figure 9. Transformation type 𝛿 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝛿=0. Finally, type 𝜇 transformation jointly operates on classifiers and datasets in such a way that the 𝑗-th of performance metrics 𝛾 for the 𝑐 classifier operating on the 𝐷 dataset has the symmetric value of the performance metric in the baseline experiment 𝛾  . Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane 𝜇=0 and can be stated as 𝛾 =−𝛾 . An example of this transformation is depicted in Figure 10 where it should be noted that the 𝜇 dimension is shown by the colour code of each point. Therefore, an inversion in 𝜇 is shown as a colour inversion. Figure 10. Transformation type 𝜇 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝜇=0. 2.3.2. Multidimensional Transformations. Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric. Firstly, let us define type 𝜎 transformation as that which exchanges 𝛼 and 𝛽 dimensions. This implies that the 𝑖-th transformed classifier/dataset pair 󰇛𝑐,𝐷) shows a ratio of correctly classified positive elements 󰇛𝛼) which has the same value as the ratio of correctly classified negative elements 󰇛𝛽) obtained by the baseline classifier/dataset pair 󰇛𝑐,𝐷). This exchange can be seen as the symmetry with respect to the hyperplane 𝛼=𝛽 (main diagonal of the 𝛼,𝛽 plane) and can be stated as 𝛼=𝛽; 𝛽=𝛼. An example of this transformation is depicted in Figure 11. Figure 11. Transformation type 𝜎 of a metric. (a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane 𝛼=𝛽. Figure 10. Transformation type µ of a metric. ( a ) Baseline metric. ( b ) Reflection symmetry with respect to the hyperplane µ=0. 2.3.2. Multidimensional Transformations Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric. Firstly, let us define type σ transformation as that which exchanges α and β dimensions. This implies that the i -th transformed classifier/dataset pair cσ i,Dσ i shows a ratio of correctly classified positive elements ασ i which has the same value as the ratio of correctly classified negative elements βB i obtained by the baseline classifier/dataset pair cB i,DB i . This exchange can be seen as the symmetry with respect to the hyperplane α=β (main diagonal of the α , β plane) and can be stated as ασ i=βB i;βσ i=αB i. An example of this transformation is depicted in Figure 11. Symmetry 2018, 10, x FOR PEER REVIEW 8 of 31 Figure 9. Transformation type 𝛿 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝛿=0. Finally, type 𝜇 transformation jointly operates on classifiers and datasets in such a way that the 𝑗-th of performance metrics 𝛾 for the 𝑐 classifier operating on the 𝐷 dataset has the symmetric value of the performance metric in the baseline experiment 𝛾  . Since the value of such metrics lie within the range [−1,1], the symmetry exists with respect to the hyperplane 𝜇=0 and can be stated as 𝛾 =−𝛾 . An example of this transformation is depicted in Figure 10 where it should be noted that the 𝜇 dimension is shown by the colour code of each point. Therefore, an inversion in 𝜇 is shown as a colour inversion. Figure 10. Transformation type 𝜇 of a metric. (a) Baseline metric. (b) Reflection symmetry with respect to the hyperplane 𝜇=0. 2.3.2. Multidimensional Transformations. Let us now consider transformations that exchange two or more dimensions of the 4-dimensional performance metric. Firstly, let us define type 𝜎 transformation as that which exchanges 𝛼 and 𝛽 dimensions. This implies that the 𝑖-th transformed classifier/dataset pair 󰇛𝑐,𝐷) shows a ratio of correctly classified positive elements 󰇛𝛼) which has the same value as the ratio of correctly classified negative elements 󰇛𝛽) obtained by the baseline classifier/dataset pair 󰇛𝑐,𝐷). This exchange can be seen as the symmetry with respect to the hyperplane 𝛼=𝛽 (main diagonal of the 𝛼,𝛽 plane) and can be stated as 𝛼=𝛽; 𝛽=𝛼. An example of this transformation is depicted in Figure 11. Figure 11. Transformation type 𝜎 of a metric. (a) Baseline metric; (b) Reflection symmetry with respect to the hyperplane 𝛼=𝛽. Figure 11. Transformation type σ of a metric. ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the hyperplane α=β. Although the four axes in these plots remain dimensionless, not all of them have the same meaning. So, α and β are both ratios of correctly classified elements. It would be nonsensical, for instance, to rescale α without also rescaling β . However, δ has a completely different meaning Symmetry 2019,11, 47 9 of 31 and its scale can and in fact does, differ from α and β . The same reasons can be applied to the axes µ . Therefore, all the exchanges of multidimensional axes are meaningless, except the interchange of α and β. All the other remaining exchanges are dismissed in our study. The one- and two-dimensional transformations described above are called basic transformations and are summarized in Table 1. Table 1. Summary of basic transformations. Transformation αkβkδkµk α1−αBβBδBµB β αB1−βBδBµB δ αBβB−δBµB µ αBβBδB−µB σ βBαBδBµB 2.3.3. Combined Transformations. More complex transformations can be obtained by concatenating basic transformations. For instance, applying basic transformation α ( Tα ) and then basic transformation β ( Tβ ) produces a new combined transformation Tαβ =Tα·Tβ featured by ααβ = 1 −αB ; βαβ = 1 −βB . As each of the one-dimensional transformations operates on an independent axis, they have the commutative and associative properties, that is, given 3 one-dimensional transformations, TU , TVand TW , it is true that TU·TV=TV·TUand that that TU·TV·TW=TU·TV·TW. However, bi-dimensional type σ transformation Tσ operates on the same axis as Tα and Tβ . In this case, the order of transformation matters, as they do not have the commutative property. For instance, Tασ[µB j] = Tσ{Tα[µB j(αB,βB,δB)]}=Tσ{µB j(1−αB,βB,δB)}=µB j(βB, 1 −αB,δB) . On the other hand, Tσα[µB j] = Tα{Tσ[µB j(αB,βB,δB)]}=Tα{µB j(βB,αB,δB)}=µB j(1βB,αB,δB) . Therefore, it is clear that Tασ 6=Tσα. Having 5 basic transformations and not initially considering their order, any combined transformation can be binary coded in terms of the presence/absence of each basic component. Therefore 2 5= 32 combinations are possible; only 31 if the identity transformation (coded 00000) is dismissed. In order to code a combined transformation, the order µ , σ , δ , β , α is used where transformation µ indicates the Most Significant Bit (MSB) and the transformation α specifies the Least Significant Bit (LSB). An example of this code is shown in Table 2. With this selection, codes greater than 15 contain a transformation type µ , that is, they are useful in exploring antisymmetric behaviour. In the cases where the order of transformations matters, σ= 1 and ( α= 1 or β= 1), then their corresponding codes refer to various different combined transformations. Table 2. Example of the coding of combined transformations. Transformation Code µ σ δ β α 28 11 1 0 0 A first example of combined transformations is that of the inverse labelling of classes. As stated above, the elements in every dataset belong to either of two classes, G1 and G2 , which are called Positive ( P ) and Negative ( N ) classes, respectively. The inverse labelling transformation ( TL ) explores the classification metric behaviour when the labelling of the classes is inverted, that is, when G2 is called the Positive class and G1 the Negative class. Let us consider the i -th classifier cL i operating on its corresponding DL i dataset. In the baseline experiment, the ratio of correctly classified positive elements ( αB i ) refers to class G1and conversely ( βB i ) refers to class G2 . In the TL transformed experiment, the ratio of correctly classified positive elements ( αL i ) refers to class G2and conversely ( βL i ) refers to Symmetry 2019,11, 47 16 of 31 These two metrics present symmetric behaviour for only the combined transformations code 31 (11111) which, in any of its ordering, is equivalent to the full inversion TF=TL·TS=Tαβδσµ and can be formulated for precision as µPRCn(α,β,δ)=−µPRCn(1−β, 1 −α,−δ). (27) In other words, precision is symmetric with respect to the concatenation of inverse labelling and the inverse scoring transformations. The results are depicted in Figure 19. Symmetry 2018, 10, x FOR PEER REVIEW 16 of 31 𝜎𝛼𝛽 𝜎𝛽𝛼 These two metrics present symmetric behaviour for only the combined transformations code 31 (11111) which, in any of its ordering, is equivalent to the full inversion 𝑇=𝑇·𝑇=𝑇 and can be formulated for precision as 𝜇󰇛𝛼,𝛽,𝛿)=−𝜇󰇛1−𝛽,1−𝛼,−𝛿). (27) In other words, precision is symmetric with respect to the concatenation of inverse labelling and the inverse scoring transformations. The results are depicted in Figure 19. Figure 19. Symmetry of precision with respect to the full inversion (𝑇). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇); (c) Reflection symmetry with respect to the plane 𝛽=0 (𝑇); (d) Reflection symmetry with respect to the main diagonal (𝑇); (e) Reflection symmetry with respect to the plane 𝛿=0 (𝑇); (f) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇). Let us now analyse the geometric mean (𝐺𝑀𝑛), which presents symmetric behaviour for the combined transformations shown in Table 6. Figure 19. Symmetry of precision with respect to the full inversion ( TF ). ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the plane α= 0 ( Tα ); ( c ) Reflection symmetry with respect to the plane β= 0 ( Tβ) ; ( d ) Reflection symmetry with respect to the main diagonal ( Tσ ); ( e ) Reflection symmetry with respect to the plane δ= 0 ( Tδ) ; ( f ) Reflection symmetry with respect to the plane µ= 0 (colour inversion, Tµ). Let us now analyse the geometric mean ( GMn ), which presents symmetric behaviour for the combined transformations shown in Table 6. Table 6. Symmetric transformations of GMn. Code µ σ δ β α Specific Order Any Order 4 0 0 1 0 0 δ 8 0 1 0 0 0 σ 11 0 1 0 1 1 ασβ (=σ) βσα (=σ) 12 0 1 1 0 0 δσ 15 0 1 1 1 1 ασβ (=σ) βσα (=σ)δ In first place, code 4 corresponds to Tδ . In fact, this metric is not only symmetric with respect to δ but also independent of δ , as it can be seen in Table 3. Secondly, combined transformations coded as 8 and 11 are equivalent to the Tσ transformation, that is, GMn is symmetric with respect to the diagonal in the α,βplane. This can be formulated as µGMn(α,β)=µGMn(β,α). (28) Finally, codes 12 and 15 imply concatenating Tδ to Tσ but as the metric is independent of δ , it is again equivalent to Tσ, that is, Tσδ =Tσ·Tδ=Tσ. These results are depicted in Figure 20. Symmetry 2018, 10, x FOR PEER REVIEW 17 of 31 Table 6. Symmetric transformations of 𝐺𝑀𝑛. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 4 0 0 1 0 0 𝛿 8 0 1 0 0 0 𝜎 11 0 1 0 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 12 0 1 1 0 0 𝛿𝜎 15 0 1 1 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 𝛿 In first place, code 4 corresponds to 𝑇. In fact, this metric is not only symmetric with respect to 𝛿 but also independent of 𝛿, as it can be seen in Table 3. Secondly, combined transformations coded as 8 and 11 are equivalent to the 𝑇 transformation, that is, 𝐺𝑀𝑛 is symmetric with respect to the diagonal in the 𝛼,𝛽 plane. This can be formulated as 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼). (28) Finally, codes 12 and 15 imply concatenating 𝑇 to 𝑇 but as the metric is independent of 𝛿, it is again equivalent to 𝑇, that is, 𝑇=𝑇·𝑇=𝑇. These results are depicted in Figure 20. Figure 20. Symmetry of geometric mean with respect to 𝑇  . (a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (𝑇  ). In the case of bookmaker informedness (𝐵𝑀), the symmetric behaviour is obtained for the combined transformations shown in Table 7. Table 7. Symmetric transformations of 𝐵𝑀. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 4 0 0 1 0 0 𝛿 8 0 1 0 0 0 𝜎 11 0 1 0 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 12 0 1 1 0 0 𝛿𝜎 15 0 1 1 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 𝛿 19 1 0 0 1 1 𝛼𝛽𝜇 23 1 0 1 1 1 𝛼𝛽𝛿𝜇 27 1 1 0 1 1 𝛼𝛽𝜎 𝛽𝛼𝜎 𝜎𝛼𝛽 𝜎𝛽𝛼 𝜇 31 1 1 1 1 1 𝛼𝛽𝜎 𝛽𝛼𝜎 𝜎𝛼𝛽 𝜎𝛽𝛼 𝛿𝜇 Figure 20. Symmetry of geometric mean with respect to Tσ . ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the main diagonal (Tσ). Symmetry 2019,11, 47 17 of 31 In the case of bookmaker informedness ( BM ), the symmetric behaviour is obtained for the combined transformations shown in Table 7. Table 7. Symmetric transformations of BM. Code µ σ δ β α Specific Order Any Order 4 0 0 1 0 0 δ 8 0 1 0 0 0 σ 11 0 1 0 1 1 ασβ (=σ) βσα (=σ) 12 0 1 1 0 0 δσ 15 0 1 1 1 1 ασβ (=σ) βσα (=σ)δ 19 1 0 0 1 1 αβµ 23 1 0 1 1 1 αβδµ 27 1 1 0 1 1 αβσ βασ σαβ σβα µ 31 1 1 1 1 1 αβσ βασ σαβ σβα δµ Again code 4 corresponds to Tδ as a consequence that this metric is independent of δ (see Table 3). Secondly, combined transformations coded as 8 and 11 are equivalent to the Tσ transformation, that is, BM is symmetric with respect to the diagonal in the α,βplane. This can be formulated as µBM(α,β)=µBM(β,α). (29) Additionally, codes 12 and 15 imply concatenating Tδ to Tσ but since the metric is independent of δ, it is again equivalent to Tσ, that is, Tσδ =Tσ·Tδ=Tσ. These results are depicted in Figure 21. Symmetry 2018, 10, x FOR PEER REVIEW 18 of 31 Again code 4 corresponds to 𝑇 as a consequence that this metric is independent of 𝛿 (see Table 3). Secondly, combined transformations coded as 8 and 11 are equivalent to the 𝑇 transformation, that is, 𝐵𝑀 is symmetric with respect to the diagonal in the 𝛼,𝛽 plane. This can be formulated as 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼). (29) Additionally, codes 12 and 15 imply concatenating 𝑇 to 𝑇 but since the metric is independent of 𝛿, it is again equivalent to 𝑇, that is, 𝑇=𝑇·𝑇=𝑇. These results are depicted in Figure 21. Figure 21. Symmetry of bookmaker informedness with respect to 𝑇. (a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (𝑇). Code 19 and also code 23 since the metric does not depend on 𝛿, correspond to the transformation 𝑇 or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated for bookmaker informedness as 𝜇󰇛𝛼,𝛽)=−𝜇󰇛1−𝛼,1−𝛽)=−𝜇󰇛1−𝛽,1−𝛼). (30) The results are depicted in Figure 22. Figure 22. Symmetry of bookmaker informedness with respect to the inverse scoring (𝑇). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇); (c) Reflection symmetry with respect to the plane 𝛽=0 (𝑇); (d) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇). In other words, the bookmaker informedness is symmetric with respect to the inverse labelling and to the inverse scoring transformations. This implies that it is also symmetric with respect to the concatenations of these two transforms, which occurs in codes 27 and 31 (recall that the latter is independent of 𝛿) corresponding to the full inversion 𝑇=𝑇+𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽)=−𝜇󰇛1−𝛽,1−𝛼). (31) The results are depicted in Figure 23. Figure 21. Symmetry of bookmaker informedness with respect to Tσ . ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the main diagonal (Tσ). Code 19 and also code 23 since the metric does not depend on δ , correspond to the transformation Tαβµ or, in other words, to the inverse scoring transformation TS=Tαβµ , which can be formulated for bookmaker informedness as µBM(α,β)=−µBM(1−α, 1 −β)=−µBM(1−β, 1 −α). (30) The results are depicted in Figure 22. Symmetry 2019,11, 47 18 of 31 Symmetry 2018, 10, x FOR PEER REVIEW 18 of 31 Again code 4 corresponds to 𝑇 as a consequence that this metric is independent of 𝛿 (see Table 3). Secondly, combined transformations coded as 8 and 11 are equivalent to the 𝑇 transformation, that is, 𝐵𝑀 is symmetric with respect to the diagonal in the 𝛼,𝛽 plane. This can be formulated as 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼). (29) Additionally, codes 12 and 15 imply concatenating 𝑇 to 𝑇 but since the metric is independent of 𝛿, it is again equivalent to 𝑇, that is, 𝑇=𝑇·𝑇=𝑇. These results are depicted in Figure 21. Figure 21. Symmetry of bookmaker informedness with respect to 𝑇. (a) Baseline metric; (b) Reflection symmetry with respect to the main diagonal (𝑇). Code 19 and also code 23 since the metric does not depend on 𝛿, correspond to the transformation 𝑇 or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated for bookmaker informedness as 𝜇󰇛𝛼,𝛽)=−𝜇󰇛1−𝛼,1−𝛽)=−𝜇󰇛1−𝛽,1−𝛼). (30) The results are depicted in Figure 22. Figure 22. Symmetry of bookmaker informedness with respect to the inverse scoring (𝑇). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇); (c) Reflection symmetry with respect to the plane 𝛽=0 (𝑇); (d) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇). In other words, the bookmaker informedness is symmetric with respect to the inverse labelling and to the inverse scoring transformations. This implies that it is also symmetric with respect to the concatenations of these two transforms, which occurs in codes 27 and 31 (recall that the latter is independent of 𝛿) corresponding to the full inversion 𝑇=𝑇+𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽)=−𝜇󰇛1−𝛽,1−𝛼). (31) The results are depicted in Figure 23. Figure 22. Symmetry of bookmaker informedness with respect to the inverse scoring ( TS ). ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the plane α= 0 ( Tα ); ( c ) Reflection symmetry with respect to the plane β= 0 ( Tβ) ; ( d ) Reflection symmetry with respect to the plane µ= 0 (colour inversion, Tµ). In other words, the bookmaker informedness is symmetric with respect to the inverse labelling and to the inverse scoring transformations. This implies that it is also symmetric with respect to the concatenations of these two transforms, which occurs in codes 27 and 31 (recall that the latter is independent of δ ) corresponding to the full inversion TF=TL+TS=Tαβδσµ , which can be formulated as µBM(α,β)=−µBM(1−β, 1 −α). (31) The results are depicted in Figure 23. Symmetry 2018, 10, x FOR PEER REVIEW 19 of 31 Figure 23. Symmetry of bookmaker informedness with respect to the full inversion (𝑇  ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ); (c) Reflection symmetry with respect to the plane 𝛽=0 (𝑇  ); (c) Reflection symmetry with respect to the main diagonal (𝑇  ); (d) Reflection symmetry with respect to the plane 𝛿=0 (𝑇  ); (e) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). In the case of sensitivity (𝑆𝑁𝑆𝑛), the symmetric behaviour is found for the combined transformations shown in Table 8. Table 8. Symmetric transformations of sensitivity. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 2 0 0 0 1 0 𝛽 4 0 0 1 0 0 𝛿 6 0 0 1 1 0 𝛽𝛿 17 1 0 0 0 1 𝛼𝜇 19 1 0 0 1 1 𝛼𝛽𝜇 21 1 0 1 0 1 𝛼𝛿𝜇 23 1 0 1 1 1 𝛼𝛽𝛿𝜇 Codes 2 and 4 correspond to 𝑇 and 𝑇 as a consequence of this metric being independent of 𝛽 and 𝛿 (see Table 3). Code 19 (and also codes 17, 21 and 23 since the metric does not depend on 𝛽 nor 𝛿) corresponds to the transformation 𝑇, or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼)=−𝜇󰇛1−𝛼). (32) This result is depicted in Figure 24. Figure 24. Symmetry of sensitivity with respect to the combined transformation (𝑇  ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ); (c) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). On considering the specificity (𝑆𝑃𝐶𝑛), its symmetric behaviour is shown in Table 9. Figure 23. Symmetry of bookmaker informedness with respect to the full inversion ( TF ). ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the plane α= 0 ( Tα ); ( c ) Reflection symmetry with respect to the plane β= 0 ( Tβ) ; ( c ) Reflection symmetry with respect to the main diagonal ( Tσ ); ( d ) Reflection symmetry with respect to the plane δ= 0 ( Tδ) ; ( e ) Reflection symmetry with respect to the plane µ=0 (colour inversion, Tµ). In the case of sensitivity ( SNSn ), the symmetric behaviour is found for the combined transformations shown in Table 8. Table 8. Symmetric transformations of sensitivity. Code µ σ δ β α Specific Order Any Order 2 0 0 0 1 0 β 4 0 0 1 0 0 δ 6 0 0 1 1 0 βδ 17 1 0 0 0 1 αµ 19 1 0 0 1 1 αβµ 21 1 0 1 0 1 αδµ 23 1 0 1 1 1 αβδµ Codes 2 and 4 correspond to Tβ and Tδ as a consequence of this metric being independent of β and δ (see Table 3). Code 19 (and also codes 17, 21 and 23 since the metric does not depend on β nor Symmetry 2019,11, 47 19 of 31 δ ) corresponds to the transformation Tαβµ , or, in other words, to the inverse scoring transformation TS=Tαβµ, which can be formulated as µSNSn(α)=−µSNSn(1−α). (32) This result is depicted in Figure 24. Symmetry 2018, 10, x FOR PEER REVIEW 19 of 31 Figure 23. Symmetry of bookmaker informedness with respect to the full inversion (𝑇  ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ); (c) Reflection symmetry with respect to the plane 𝛽=0 (𝑇  ); (c) Reflection symmetry with respect to the main diagonal (𝑇  ); (d) Reflection symmetry with respect to the plane 𝛿=0 (𝑇  ); (e) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). In the case of sensitivity (𝑆𝑁𝑆𝑛), the symmetric behaviour is found for the combined transformations shown in Table 8. Table 8. Symmetric transformations of sensitivity. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 2 0 0 0 1 0 𝛽 4 0 0 1 0 0 𝛿 6 0 0 1 1 0 𝛽𝛿 17 1 0 0 0 1 𝛼𝜇 19 1 0 0 1 1 𝛼𝛽𝜇 21 1 0 1 0 1 𝛼𝛿𝜇 23 1 0 1 1 1 𝛼𝛽𝛿𝜇 Codes 2 and 4 correspond to 𝑇 and 𝑇 as a consequence of this metric being independent of 𝛽 and 𝛿 (see Table 3). Code 19 (and also codes 17, 21 and 23 since the metric does not depend on 𝛽 nor 𝛿) corresponds to the transformation 𝑇, or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼)=−𝜇󰇛1−𝛼). (32) This result is depicted in Figure 24. Figure 24. Symmetry of sensitivity with respect to the combined transformation (𝑇  ). (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ); (c) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). On considering the specificity (𝑆𝑃𝐶𝑛), its symmetric behaviour is shown in Table 9. Figure 24. Symmetry of sensitivity with respect to the combined transformation ( Tαµ ). ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the plane α= 0 ( Tα ); ( c ) Reflection symmetry with respect to the plane µ=0 (colour inversion, Tµ). On considering the specificity (SPCn), its symmetric behaviour is shown in Table 9. Table 9. Symmetric transformations of specificity. Code µ σ δ β α Specific Order Any Order 1 0 0 0 0 1 α 4 0 0 1 0 0 δ 5 0 0 1 0 1 αδ 18 1 0 0 1 0 βµ 19 1 0 0 1 1 αβµ 22 1 0 1 1 0 βδµ 23 1 0 1 1 1 αβδµ Codes 1 and 4 corresponds to Tα and Tδ as a consequence of this metric being independent of α and δ (see Table 3). Code 19 (and also codes 18, 22 and 23 as the metric depends neither on α nor on δ ) corresponds to the transformation Tαβµ , that is, to the inverse scoring transformation TS=Tαβµ , which can be formulated as µSPCn(β)=−µSPCn(1−β). (33) This result is depicted in Figure 25. Symmetry 2018, 10, x FOR PEER REVIEW 20 of 31 Table 9. Symmetric transformations of specificity. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 1 0 0 0 0 1 𝛼 4 0 0 1 0 0 𝛿 5 0 0 1 0 1 𝛼𝛿 18 1 0 0 1 0 𝛽𝜇 19 1 0 0 1 1 𝛼𝛽𝜇 22 1 0 1 1 0 𝛽𝛿𝜇 23 1 0 1 1 1 𝛼𝛽𝛿𝜇 Codes 1 and 4 corresponds to 𝑇 and 𝑇 as a consequence of this metric being independent of 𝛼 and 𝛿 (see Table 3). Code 19 (and also codes 18, 22 and 23 as the metric depends neither on 𝛼 nor on 𝛿) corresponds to the transformation 𝑇, that is, to the inverse scoring transformation 𝑇= 𝑇, which can be formulated as 𝜇󰇛𝛽)=−𝜇󰇛1−𝛽). (33) This result is depicted in Figure 25. Figure 25. Symmetry of specificity with respect to the combined transformation (𝑇  ) (a) Baseline metric; (b) Reflection symmetry with respect to the plane 𝛽=0 (𝑇  ); (c) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). Finally, it can be observed that the 𝐹𝑛 score metric is not symmetric under any transformation. The results for each metric are summarized in Table 10. Table 10. Summary of symmetries. Metric Independent of Symmetry (under Inversion of) 𝜶 𝜷 𝜹 Labelling Scoring Full 𝑆𝑁𝑆𝑛    𝑆𝑃𝐶𝑛    𝑃𝑅𝐶𝑛  𝑁𝑃𝑉𝑛  𝐴 𝐶𝐶𝑛    𝐹𝑛 𝐺𝑀𝑛   𝑀𝐶𝐶    𝐵𝑀     𝑀𝐾    3.2. Identifying Cross-Symmetries In order to explore whether any cross-symmetry can be identified among the 10 metrics, we have computed the distance (using Equation (20)) of the baseline of each metric (and its 31 possible Figure 25. Symmetry of specificity with respect to the combined transformation ( Tβµ ) ( a ) Baseline metric; ( b ) Reflection symmetry with respect to the plane β= 0 ( Tβ ); ( c ) Reflection symmetry with respect to the plane µ=0 (colour inversion, Tµ). Symmetry 2019,11, 47 20 of 31 Finally, it can be observed that the F1n score metric is not symmetric under any transformation. The results for each metric are summarized in Table 10. Table 10. Summary of symmetries. Metric Independent of Symmetry (under Inversion of) α β δ Labelling Scoring Full SNSn 3 3 3 SPCn 3 3 3 PRCn 3 NPVn 3 ACCn 3 3 3 F1n GMn 3 3 MCC 3 3 3 BM 3 3 3 3 MK 3 3 3 3.2. Identifying Cross-Symmetries In order to explore whether any cross-symmetry can be identified among the 10 metrics, we have computed the distance (using Equation (20)) of the baseline of each metric (and its 31 possible transformations), to the remaining baseline metrics. The results are depicted in Figure 26. Each row corresponds to the baseline of a metric and each column to the baseline and its 31 transformations of the other metric. Any given metric-metric pair (small squares in the graphic) is shown in yellow if it has zero-distance for any possible transformation. Symmetry 2018, 10, x FOR PEER REVIEW 21 of 31 transformations), to the remaining baseline metrics. The results are depicted in Figure 26. Each row corresponds to the baseline of a metric and each column to the baseline and its 31 transformations of the other metric. Any given metric-metric pair (small squares in the graphic) is shown in yellow if it has zero-distance for any possible transformation. The diagonal presents a summary of the results explored in the previous section, that is, every metric, except for the 𝐹𝑛 score, presents some kind of symmetry under some transformation. The cases of cross-symmetries appear in the elements off diagonal. Two cross-symmetries arise: the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 and the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛. Figure 26. Cross-symmetric behaviour of performance metrics for any combined transformation. In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric. The results are depicted in Figure 27. Each row shows the cross-symmetries of a pair of metrics. In the columns are the 32 different transformations (including the baseline) of the second metric in the pair. Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline. As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the 𝜇 axis has been inverted, that is, where the transformation type 𝜇 is present. This is therefore the area for antisymmetric behaviour. Figure 27. Cross-symmetric behaviour for any combined transformation. Let us first analyse each pair of metrics in terms of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 or 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair, which present cross-symmetric behaviour for the combined transformations shown in Table 11. Table 11. Cross-symmetric transformations of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 12 0 1 1 0 0 𝛿𝜎 SNSn SPCn PRCn NPVn ACCn F1n GMn MCC BM MK SNSn SPCn PRCn NPVn ACCn F1n GMn MCC BM MK Combined transformation (code) Symmetric Antisymmetric 0246810 12 14 16 18 20 22 24 26 28 30 SNSn-SPCn SPCn-SNSn PRCn-NPVn NPVn-PRCn Figure 26. Cross-symmetric behaviour of performance metrics for any combined transformation. The diagonal presents a summary of the results explored in the previous section, that is, every metric, except for the F1n score, presents some kind of symmetry under some transformation. The cases of cross-symmetries appear in the elements off diagonal. Two cross-symmetries arise: the SNSn −SPCn and the PRCn −NPVn. In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric. The results are depicted in Figure 27. Each row shows the Symmetry 2019,11, 47 21 of 31 cross-symmetries of a pair of metrics. In the columns are the 32 different transformations (including the baseline) of the second metric in the pair. Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline. As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the µ axis has been inverted, that is, where the transformation type µ is present. This is therefore the area for antisymmetric behaviour. Symmetry 2018, 10, x FOR PEER REVIEW 21 of 31 transformations), to the remaining baseline metrics. The results are depicted in Figure 26. Each row corresponds to the baseline of a metric and each column to the baseline and its 31 transformations of the other metric. Any given metric-metric pair (small squares in the graphic) is shown in yellow if it has zero-distance for any possible transformation. The diagonal presents a summary of the results explored in the previous section, that is, every metric, except for the 𝐹𝑛 score, presents some kind of symmetry under some transformation. The cases of cross-symmetries appear in the elements off diagonal. Two cross-symmetries arise: the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 and the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛. Figure 26. Cross-symmetric behaviour of performance metrics for any combined transformation. In order to attain a deeper insight into these cross-symmetries, let us consider, for each of the two pairs, the distances between the baseline of the first metric in the pair and the full set of transformations (including the baseline) of the second metric. The results are depicted in Figure 27. Each row shows the cross-symmetries of a pair of metrics. In the columns are the 32 different transformations (including the baseline) of the second metric in the pair. Any given (second-metric transformation) pair (small squares in the graphic) is shown in yellow if it has zero-distance with the first metric baseline. As in Figure 15, the right-hand-side of the plot (with code greater than or equal to 16) corresponds to combined transformation where the 𝜇 axis has been inverted, that is, where the transformation type 𝜇 is present. This is therefore the area for antisymmetric behaviour. Figure 27. Cross-symmetric behaviour for any combined transformation. Let us first analyse each pair of metrics in terms of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 or 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair, which present cross-symmetric behaviour for the combined transformations shown in Table 11. Table 11. Cross-symmetric transformations of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 12 0 1 1 0 0 𝛿𝜎 SNSn SPCn PRCn NPVn ACCn F1n GMn MCC BM MK SNSn SPCn PRCn NPVn ACCn F1n GMn MCC BM MK Combined transformation (code) Symmetric Antisymmetric 0246810 12 14 16 18 20 22 24 26 28 30 SNSn-SPCn SPCn-SNSn PRCn-NPVn NPVn-PRCn Figure 27. Cross-symmetric behaviour for any combined transformation. Let us first analyse each pair of metrics in terms of the PRCn −NPVn or PRCn −NPVn pair, which present cross-symmetric behaviour for the combined transformations shown in Table 11. Table 11. Cross-symmetric transformations of the PRCn −NPVn pair. Code µ σ δ β α Specific Order Any Order 12 0 1 1 0 0 δσ 15 0 1 1 1 1 ασβ (=σ) βσα (=σ)δ 19 1 0 0 1 1 αβµ Codes 12 and 15 correspond to the transformation Tσδ or, in other words, to the inverse labelling transformation TL=Tσδ, which can be formulated as µPRCn(α,β,δ)=µNPVn(β,α,−δ). (34) The results are depicted in Figure 28. Symmetry 2018, 10, x FOR PEER REVIEW 22 of 31 15 0 1 1 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 𝛿 19 1 0 0 1 1 𝛼𝛽𝜇 Codes 12 and 15 correspond to the transformation 𝑇 or, in other words, to the inverse labelling transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽,𝛿)=𝜇󰇛𝛽,𝛼,−𝛿). (34) The results are depicted in Figure 28. Figure 28. Cross-symmetry of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair with respect to the inverse labelling (𝑇  ). (a) Baseline 𝑃𝑅𝐶𝑛 metric; (b) Baseline 𝑁𝑃𝑉𝑛 metric; (c) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the main diagonal (𝑇  ); (d) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛿=0 (𝑇  ). Code 19 corresponds to the transformation 𝑇 or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽,𝛿)=−𝜇󰇛1−𝛼,1−𝛽,𝛿). (35) The results are depicted in Figure 29. Figure 29. Cross-symmetry of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair with respect to the inverse scoring (𝑇  ). (a) Baseline 𝑃𝑅𝐶𝑛 metric; (b) Baseline 𝑁𝑃𝑉𝑛 metric. (c) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛼=0 (𝑇  ); (d) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛽=0 (𝑇  ); (e) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). Although the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair is cross-symmetric with respect to the inverse labelling and to the inverse scoring transformations, this does not imply that it is also cross-symmetric with respect to the concatenations of these two transforms (see equation 22). This is the reason why code 31 (corresponding to the full inversion 𝑇=𝑇+𝑇=𝑇 is not present in Table 11. The results for the pair 𝑁𝑃𝑉𝑛−𝑃𝑅𝐶𝑛 are exactly the same. Therefore, 𝜇󰇛𝛼,𝛽,𝛿)=𝜇󰇛𝛽,𝛼,−𝛿)=−𝜇󰇛1−𝛼,1−𝛽,𝛿). (36) Let us now consider the pair of metrics 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 and its cross-symmetric behaviour, which is found for the combined transformations shown in Table 12. Figure 28. Cross-symmetry of the PRCn −NPVn pair with respect to the inverse labelling ( TL ). ( a ) Baseline PRCn metric; ( b ) Baseline NPVn metric; ( c ) Reflection symmetry of NPVn with respect to the main diagonal (Tσ); (d) Reflection symmetry of NPVn with respect to the plane δ=0 (Tδ). Code 19 corresponds to the transformation Tαβµ or, in other words, to the inverse scoring transformation TS=Tαβµ, which can be formulated as µPRCn(α,β,δ)=−µNPVn(1−α, 1 −β,δ). (35) The results are depicted in Figure 29. Symmetry 2019,11, 47 22 of 31 Symmetry 2018, 10, x FOR PEER REVIEW 22 of 31 15 0 1 1 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝛽𝜎𝛼 󰇛=𝜎) 𝛿 19 1 0 0 1 1 𝛼𝛽𝜇 Codes 12 and 15 correspond to the transformation 𝑇 or, in other words, to the inverse labelling transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽,𝛿)=𝜇󰇛𝛽,𝛼,−𝛿). (34) The results are depicted in Figure 28. Figure 28. Cross-symmetry of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair with respect to the inverse labelling (𝑇  ). (a) Baseline 𝑃𝑅𝐶𝑛 metric; (b) Baseline 𝑁𝑃𝑉𝑛 metric; (c) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the main diagonal (𝑇  ); (d) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛿=0 (𝑇  ). Code 19 corresponds to the transformation 𝑇 or, in other words, to the inverse scoring transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽,𝛿)=−𝜇󰇛1−𝛼,1−𝛽,𝛿). (35) The results are depicted in Figure 29. Figure 29. Cross-symmetry of the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair with respect to the inverse scoring (𝑇  ). (a) Baseline 𝑃𝑅𝐶𝑛 metric; (b) Baseline 𝑁𝑃𝑉𝑛 metric. (c) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛼=0 (𝑇  ); (d) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝛽=0 (𝑇  ); (e) Reflection symmetry of 𝑁𝑃𝑉𝑛 with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). Although the 𝑃𝑅𝐶𝑛−𝑁𝑃𝑉𝑛 pair is cross-symmetric with respect to the inverse labelling and to the inverse scoring transformations, this does not imply that it is also cross-symmetric with respect to the concatenations of these two transforms (see equation 22). This is the reason why code 31 (corresponding to the full inversion 𝑇=𝑇+𝑇=𝑇 is not present in Table 11. The results for the pair 𝑁𝑃𝑉𝑛−𝑃𝑅𝐶𝑛 are exactly the same. Therefore, 𝜇󰇛𝛼,𝛽,𝛿)=𝜇󰇛𝛽,𝛼,−𝛿)=−𝜇󰇛1−𝛼,1−𝛽,𝛿). (36) Let us now consider the pair of metrics 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 and its cross-symmetric behaviour, which is found for the combined transformations shown in Table 12. Figure 29. Cross-symmetry of the PRCn −NPVn pair with respect to the inverse scoring ( TS ). ( a ) Baseline PRCn metric; ( b ) Baseline NPVn metric. ( c ) Reflection symmetry of NPVn with respect to the plane α= 0 ( Tα ); ( d ) Reflection symmetry of NPVn with respect to the plane β= 0 ( Tβ) ; (e) Reflection symmetry of NPVn with respect to the plane µ=0 (colour inversion, Tµ). Although the PRCn −NPVn pair is cross-symmetric with respect to the inverse labelling and to the inverse scoring transformations, this does not imply that it is also cross-symmetric with respect to the concatenations of these two transforms (see Equation (22)). This is the reason why code 31 (corresponding to the full inversion TF=TL+TS=Tαβδσµ is not present in Table 11. The results for the pair NPVn −PRCn are exactly the same. Therefore, µNPVn(α,β,δ)=µPRCn(β,α,−δ)=−µPRCn(1−α, 1 −β,δ). (36) Let us now consider the pair of metrics SNSn −SPCn and its cross-symmetric behaviour, which is found for the combined transformations shown in Table 12. Table 12. Cross-symmetric transformations of the SNSn −SPCn pair. Code µ σ δ β α Specific Order Any Order 8 0 1 0 0 0 σ 9 0 1 0 0 1 ασ 10 0 1 0 1 0 σβ 11 0 1 0 1 1 ασβ (=σ) σβα (=σ) 12 0 1 1 0 0 σδ 13 0 1 1 0 1 ασ δ 14 0 1 1 1 0 σβ δ 15 0 1 1 1 1 ασβ (=σ) σβα (=σ)δ 25 1 1 0 0 1 σα µ 26 1 1 0 1 0 βσ µ 27 1 1 0 1 1 αβσ βασ σαβ σβα µ 29 1 1 1 0 1 σα δµ 30 1 1 1 1 0 βσ δµ 31 1 1 1 1 1 αβσ βασ σαβ σβα δµ Symmetry 2019,11, 47 23 of 31 Since specificity remains independent from δ (see Table 3), codes 25, 11, 12 and 15 correspond to Tσδ, that is, to the inverse labelling which can be formulated as µSNSn(α,β)=µSPCn(β,α). (37) Additionally, since specificity is also independent of α , then codes 9 ( Tασ ) and 13 ( Tασδ ) are equivalent to Tσδ . Moreover, after a Tσ transformation, the resulting metric has no dependence on β (due to the axis inversion) and hence codes 10 ( Tσβ ) and 14 ( Tσβδ ) are also equivalent to Tσδ . These results are depicted in Figure 30. Symmetry 2018, 10, x FOR PEER REVIEW 23 of 31 Table 12. Cross-symmetric transformations of the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 pair. Code 𝝁 𝝈 𝜹 𝜷 𝜶 Specific Order Any Order 8 0 1 0 0 0 𝜎 9 0 1 0 0 1 𝛼𝜎 10 0 1 0 1 0 𝜎𝛽 11 0 1 0 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝜎𝛽𝛼 󰇛=𝜎) 12 0 1 1 0 0 𝜎𝛿 13 0 1 1 0 1 𝛼𝜎 𝛿 14 0 1 1 1 0 𝜎𝛽 𝛿 15 0 1 1 1 1 𝛼𝜎𝛽 󰇛=𝜎) 𝜎𝛽𝛼 󰇛=𝜎) 𝛿 25 1 1 0 0 1 𝜎𝛼 𝜇 26 1 1 0 1 0 𝛽𝜎 𝜇 27 1 1 0 1 1 𝛼𝛽𝜎 𝛽𝛼𝜎 𝜎𝛼𝛽 𝜎𝛽𝛼 𝜇 29 1 1 1 0 1 𝜎𝛼 𝛿𝜇 30 1 1 1 1 0 𝛽𝜎 𝛿𝜇 31 1 1 1 1 1 𝛼𝛽𝜎 𝛽𝛼𝜎 𝜎𝛼𝛽 𝜎𝛽𝛼 𝛿𝜇 Since specificity remains independent from 𝛿 (see Table 3), codes 25, 11, 12 and 15 correspond to 𝑇, that is, to the inverse labelling which can be formulated as 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼). (37) Additionally, since specificity is also independent of 𝛼, then codes 9 (𝑇) and 13 (𝑇) are equivalent to 𝑇. Moreover, after a 𝑇 transformation, the resulting metric has no dependence on 𝛽 (due to the axis inversion) and hence codes 10 (𝑇) and 14 (𝑇) are also equivalent to 𝑇. These results are depicted in Figure 30. Figure 30. Cross-symmetry of the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 pair with respect to the inverse labelling (𝑇  ). (a) Baseline 𝑆𝑁𝑆𝑛 metric; (b) Baseline 𝑆𝑃𝐶𝑛 metric; (c) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the main diagonal (𝑇  ); (d) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the plane 𝛿=0 (𝑇  ). On the other hand, code 31 corresponds to full inversion transformation 𝑇=𝑇, which can be formulated as 𝜇󰇛𝛼,𝛽)=−𝜇󰇛1−𝛽,1−𝛼). (38) It can be shown that the remaining codes (25, 26, 27, 29 and 30) are also equivalent to 𝑇. Moreover, after a 𝑇 transformation, the resulting metric does not depend on 𝛽 (due to the axis Figure 30. Cross-symmetry of the SNSn −SPCn pair with respect to the inverse labelling ( TL ). ( a ) Baseline SNSn metric; ( b ) Baseline SPCn metric; ( c ) Reflection symmetry of SPCn with respect to the main diagonal (Tσ); (d) Reflection symmetry of SPCn with respect to the plane δ=0 (Tδ). On the other hand, code 31 corresponds to full inversion transformation TF=Tσδαβµ , which can be formulated as µSNSn(α,β)=−µSPCn(1−β, 1 −α). (38) It can be shown that the remaining codes (25, 26, 27, 29 and 30) are also equivalent to TF . Moreover, after a Tσ transformation, the resulting metric does not depend on β (due to the axis inversion) and hence codes 10 (Tσβ) and 14 (Tσβδ) are also equivalent to Tσδ. These results are depicted in Figure 31. Symmetry 2018, 10, x FOR PEER REVIEW 24 of 31 inversion) and hence codes 10 (𝑇) and 14 (𝑇) are also equivalent to 𝑇. These results are depicted in Figure 31. Figure 31. Cross-symmetry of the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 pair with respect to the full inversion (𝑇  ). (a) Baseline 𝑆𝑁𝑆𝑛 metric. (b) Baseline 𝑆𝑃𝐶𝑛 metric. (c) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the main diagonal (𝑇  ). (d) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the plane 𝛿=0 (𝑇  ). (e) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ). (f) Reflection symmetry with respect to the plane 𝛽= 0 (𝑇  ). (g) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). The results for the pair 𝑆𝑃𝐶𝑛−𝑆𝑁𝑆𝑛 are exactly the same, so 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼)=−𝜇󰇛1−𝛽,1−𝛼). (39) The results for every pair of cross-symmetric metrics are summarized in Table 13. Table 13. Summary of cross-symmetries. Metric Cross-Symmetry (under Inversion of) Labelling Scoring Full 𝑆𝑁𝑆𝑛 𝑆𝑃𝐶𝑛 󰇛𝑆𝑁𝑆𝑛) 𝑆𝑃𝐶𝑛 𝑆𝑃𝐶𝑛 𝑆𝑁𝑆𝑛 󰇛𝑆𝑃𝐶𝑛) 𝑆𝑁𝑆𝑛 𝑃𝑅𝐶𝑛 𝑁𝑃𝑉𝑛 𝑁𝑃𝑉𝑛 󰇛𝑃𝑅𝐶𝑛) 𝑁𝑃𝑉𝑛 𝑃𝑅𝐶𝑛 𝑃𝑅𝐶𝑛 󰇛𝑁𝑃𝑉𝑛) 3.3. Skewness of the Statistical Descriptions of the Metrics In order to explore the symmetric behaviour of the statistical descriptions of the metrics, let us recall that, for the baseline experiment, 𝜇=𝜇󰇛𝛼,𝛽,𝛿) can be considered a statistical variable. First of all, let us select a subset of the 𝜇 corresponding to a certain value 𝛿 of the imbalance coefficient, that is, 𝜇󰇛𝛼,𝛽,𝛿) and obtain its probability density function (pdf) which will be called local pdf (since it is obtained solely for a value of 𝛿). The results 𝑝𝑑𝑓𝜇,𝛿 for every metric with 𝛿=0.5 are shown in Figure 32. Figure 32. Local probability density function of every metric and 𝛿=0. Figure 31. Cross-symmetry of the SNSn −SPCn pair with respect to the full inversion ( TF ). ( a ) Baseline SNSn metric. ( b ) Baseline SPCn metric. ( c ) Reflection symmetry of SPCn with respect to the main diagonal ( Tσ ). ( d ) Reflection symmetry of SPCn with respect to the plane δ= 0 ( Tδ) . ( e ) Reflection symmetry with respect to the plane α= 0 ( Tα ). ( f ) Reflection symmetry with respect to the plane β= 0 (Tβ). (g) Reflection symmetry with respect to the plane µ=0 (colour inversion, Tµ). The results for the pair SPCn −SNSn are exactly the same, so µSPCn(α,β)=µSNSn(β,α)=−µSNSn(1−β, 1 −α). (39) The results for every pair of cross-symmetric metrics are summarized in Table 13. Symmetry 2019,11, 47 24 of 31 Table 13. Summary of cross-symmetries. Metric Cross-Symmetry (under Inversion of) Labelling Scoring Full SNSn SPCn (SNSn)SPCn SPCn SNSn (SPCn)SNSn PRCn NPVn NPVn (PRCn) NPVn PRCn PRCn (NPVn) 3.3. Skewness of the Statistical Descriptions of the Metrics In order to explore the symmetric behaviour of the statistical descriptions of the metrics, let us recall that, for the baseline experiment, µB j=µB jαB,βB,δB can be considered a statistical variable. First of all, let us select a subset of the µB j corresponding to a certain value δ0 of the imbalance coefficient, that is, µB jαB,βB,δ0 and obtain its probability density function (pdf) which will be called local pdf (since it is obtained solely for a value of δB ). The results pd f (µk j,δ0) for every metric with δB= 0.5 are shown in Figure 32. Symmetry 2018, 10, x FOR PEER REVIEW 24 of 31 inversion) and hence codes 10 (𝑇) and 14 (𝑇) are also equivalent to 𝑇. These results are depicted in Figure 31. Figure 31. Cross-symmetry of the 𝑆𝑁𝑆𝑛−𝑆𝑃𝐶𝑛 pair with respect to the full inversion (𝑇  ). (a) Baseline 𝑆𝑁𝑆𝑛 metric. (b) Baseline 𝑆𝑃𝐶𝑛 metric. (c) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the main diagonal (𝑇  ). (d) Reflection symmetry of 𝑆𝑃𝐶𝑛 with respect to the plane 𝛿=0 (𝑇  ). (e) Reflection symmetry with respect to the plane 𝛼=0 (𝑇  ). (f) Reflection symmetry with respect to the plane 𝛽= 0 (𝑇  ). (g) Reflection symmetry with respect to the plane 𝜇=0 (colour inversion, 𝑇  ). The results for the pair 𝑆𝑃𝐶𝑛−𝑆𝑁𝑆𝑛 are exactly the same, so 𝜇󰇛𝛼,𝛽)=𝜇󰇛𝛽,𝛼)=−𝜇󰇛1−𝛽,1−𝛼). (39) The results for every pair of cross-symmetric metrics are summarized in Table 13. Table 13. Summary of cross-symmetries. Metric Cross-Symmetry (under Inversion of) Labelling Scoring Full 𝑆𝑁𝑆𝑛 𝑆𝑃𝐶𝑛 󰇛𝑆𝑁𝑆𝑛) 𝑆𝑃𝐶𝑛 𝑆𝑃𝐶𝑛 𝑆𝑁𝑆𝑛 󰇛𝑆𝑃𝐶𝑛) 𝑆𝑁𝑆𝑛 𝑃𝑅𝐶𝑛 𝑁𝑃𝑉𝑛 𝑁𝑃𝑉𝑛 󰇛𝑃𝑅𝐶𝑛) 𝑁𝑃𝑉𝑛 𝑃𝑅𝐶𝑛 𝑃𝑅𝐶𝑛 󰇛𝑁𝑃𝑉𝑛) 3.3. Skewness of the Statistical Descriptions of the Metrics In order to explore the symmetric behaviour of the statistical descriptions of the metrics, let us recall that, for the baseline experiment, 𝜇=𝜇󰇛𝛼,𝛽,𝛿) can be considered a statistical variable. First of all, let us select a subset of the 𝜇 corresponding to a certain value 𝛿 of the imbalance coefficient, that is, 𝜇󰇛𝛼,𝛽,𝛿) and obtain its probability density function (pdf) which will be called local pdf (since it is obtained solely for a value of 𝛿). The results 𝑝𝑑𝑓𝜇,𝛿 for every metric with 𝛿=0.5 are shown in Figure 32. Figure 32. Local probability density function of every metric and 𝛿=0. Figure 32. Local probability density function of every metric and δ=0. This result can be generalized for various values of the imbalance coefficient δB by obtaining the pd f (µk j,δB) depicted in Figure 33 as a set of heatmap plots. In every plot, the horizontal axis represents the imbalance coefficient while the value of the metric is drawn in the vertical axis. The value of the pd f is colour-coded. Symmetry 2018, 10, x FOR PEER REVIEW 25 of 31 This result can be generalized for various values of the imbalance coefficient 𝛿 by obtaining the 𝑝𝑑𝑓𝜇,𝛿 depicted in Figure 33 as a set of heatmap plots. In every plot, the horizontal axis represents the imbalance coefficient while the value of the metric is drawn in the vertical axis. The value of the 𝑝𝑑𝑓 is colour-coded. Figure 33. Local probability density function of every metric as a function of 𝛿. The value of pdf is colour coded. In Figure 32 and Figure 33, the symmetry of the statistical descriptions of the metrics can easily be observed. However, in order to achieve a more precise insight, the local skewness 𝜉 of every 𝑝𝑑𝑓 is obtained in accordance with equation 23 and its value 𝜉󰇛𝛿) is shown in Figure 34 for every metric. It can be observed that 6 metrics (𝑆𝑁𝑆𝑛, 𝑆𝑃𝐶𝑛, 𝐴𝐶𝐶𝑛, 𝑀𝐶𝐶, 𝐵𝑀 𝑎𝑛𝑑 𝑀𝐾) have a symmetric 𝑝𝑑𝑓; one metric (𝐺𝑀𝑛) has a 𝑝𝑑𝑓 slightly asymmetric but its asymmetry does not depend on 𝛿; 2 metrics (𝑃𝑅𝐶𝑛 and 𝑁𝑃𝑉𝑛) have a clearly asymmetric 𝑝𝑑𝑓 but their skewness is symmetric with respect to the origin; and finally, the 𝐹𝑛 metric has a 𝑝𝑑𝑓 and a skewness that are both asymmetric. Figure 34. Skewness of the statistical description for every metric as a function of 𝛿. Let us now examine the 𝜇 for all the values of the imbalance coefficient 𝛿, that is, 𝜇󰇛𝛼,𝛽,𝛿) and obtain its probability density function (pdf) which will be called global pdf (as it is obtained for every 𝛿). The resulting 𝑝𝑑𝑓𝜇 is shown in Figure 35 for every metric. Figure 33. Local probability density function of every metric as a function of δ . The value of pdf is colour coded. Symmetry 2019,11, 47 25 of 31 In Figures 32 and 33, the symmetry of the statistical descriptions of the metrics can easily be observed. However, in order to achieve a more precise insight, the local skewness ξB j of every pd f is obtained in accordance with Equation (23) and its value ξB jδB is shown in Figure 34 for every metric. It can be observed that 6 metrics ( SNSn , SPCn , ACCn , MCC , BM and MK ) have a symmetric pd f ; one metric ( GMn ) has a pd f slightly asymmetric but its asymmetry does not depend on δB ; 2 metrics ( PRCn and NPVn ) have a clearly asymmetric pd f but their skewness is symmetric with respect to the origin; and finally, the F1nmetric has a pd f and a skewness that are both asymmetric. Symmetry 2018, 10, x FOR PEER REVIEW 25 of 31 This result can be generalized for various values of the imbalance coefficient 𝛿 by obtaining the 𝑝𝑑𝑓𝜇,𝛿 depicted in Figure 33 as a set of heatmap plots. In every plot, the horizontal axis represents the imbalance coefficient while the value of the metric is drawn in the vertical axis. The value of the 𝑝𝑑𝑓 is colour-coded. Figure 33. Local probability density function of every metric as a function of 𝛿. The value of pdf is colour coded. In Figure 32 and Figure 33, the symmetry of the statistical descriptions of the metrics can easily be observed. However, in order to achieve a more precise insight, the local skewness 𝜉 of every 𝑝𝑑𝑓 is obtained in accordance with equation 23 and its value 𝜉󰇛𝛿) is shown in Figure 34 for every metric. It can be observed that 6 metrics (𝑆𝑁𝑆𝑛, 𝑆𝑃𝐶𝑛, 𝐴𝐶𝐶𝑛, 𝑀𝐶𝐶, 𝐵𝑀 𝑎𝑛𝑑 𝑀𝐾) have a symmetric 𝑝𝑑𝑓; one metric (𝐺𝑀𝑛) has a 𝑝𝑑𝑓 slightly asymmetric but its asymmetry does not depend on 𝛿; 2 metrics (𝑃𝑅𝐶𝑛 and 𝑁𝑃𝑉𝑛) have a clearly asymmetric 𝑝𝑑𝑓 but their skewness is symmetric with respect to the origin; and finally, the 𝐹𝑛 metric has a 𝑝𝑑𝑓 and a skewness that are both asymmetric. Figure 34. Skewness of the statistical description for every metric as a function of 𝛿. Let us now examine the 𝜇 for all the values of the imbalance coefficient 𝛿, that is, 𝜇󰇛𝛼,𝛽,𝛿) and obtain its probability density function (pdf) which will be called global pdf (as it is obtained for every 𝛿). The resulting 𝑝𝑑𝑓𝜇 is shown in Figure 35 for every metric. Figure 34. Skewness of the statistical description for every metric as a function of δ. Let us now examine the µB j for all the values of the imbalance coefficient δB , that is, µB jαB,βB,δB and obtain its probability density function (pdf) which will be called global pdf (as it is obtained for every δB). The resulting pd f (µk j)is shown in Figure 35 for every metric. Symmetry 2018, 10, x FOR PEER REVIEW 26 of 31 Figure 35. Global probability density function of every metric and 𝛿=0. It can be observed that all the metrics show a symmetric 𝑝𝑑𝑓 except for 𝐺𝑀𝑛 and 𝐹𝑛. The global pdf for 𝐺𝑀𝑛 maintains the slight asymmetry of local pdf (global skewness of 0.18) since 𝐺𝑀𝑛 does not depend on 𝛿. In the cases of 𝑃𝑅𝐶𝑛 and 𝑁𝑃𝑉𝑛, the symmetry of the local skewness compensates for their values and hence they show a symmetric global pdf. Finally, the positive values of 𝐹𝑛 local skewness partially compensate for its negative values (see Figure 34), which results in an almost uniform global pdf except for their extreme values (global skewness of 0.14). These results are summarized in Table 14. Table 14. Summary of statistical symmetry. Metric Statistical Symmetry Local Global (Skewness) 𝑆𝑁𝑆𝑛   𝑆𝑃𝐶𝑛   𝑃𝑅𝐶𝑛  𝑁𝑃𝑉𝑛  𝐴 𝐶𝐶𝑛   𝐹𝑛 (0.14) 𝐺𝑀𝑛 (0.18) 𝑀𝐶𝐶   𝐵𝑀   𝑀𝐾   4. Discussion From the previous results, summarized in Table 10, Table 13 and Table 14, it can be seen that although several thousands of combined transformations have been tested, the performance metrics only present three types of symmetries: under labelling inversion; under scoring inversion; and under full inversion (the sequence of labelling and scoring inversion). For a certain performance metric to be symmetric under labelling inversion means that it pays attention to or focuses on, positive and negative classes with the same intensity and therefore classes can be exchanged without affecting the value of the metric. These metrics should be used in applications where the cost of misclassification is the same for each class. This is the case for 5 out of the 10 metrics tested: 𝐴𝐶𝐶𝑛, 𝑀𝐶𝐶, 𝐵𝑀, 𝑀𝐾 and 𝐺𝑀𝑛. Other metrics, however, are more focused on the classification results obtained for the positive class. This is the case of 3 metrics: 𝑆𝑁𝑆𝑛, which only depends on 𝛼; 𝑃𝑅𝐶𝑛, which measures the ratio of success on the elements classified as positive; and the 𝐹 score, which is a combination of 𝑆𝑁𝑆𝑛 Figure 35. Global probability density function of every metric and δ=0. It can be observed that all the metrics show a symmetric pd f except for GMn and F1n . The global pdf for GMn maintains the slight asymmetry of local pdf (global skewness of 0.18) since GMn does not depend on δ . In the cases of PRCn and NPVn , the symmetry of the local skewness compensates for their values and hence they show a symmetric global pdf. Finally, the positive values of F1n local skewness partially compensate for its negative values (see Figure 34), which results in an almost uniform global pdf except for their extreme values (global skewness of 0.14). These results are summarized in Table 14.