Visual interpretability of deep learning algorithms in medical applications
Abstract
Artificial intelligence is having a very big boost in recent times, and after the success of deep learning algorithms in many applications, they are also providing successful results for medical imaging, especially because of the good performance of convolutional neural networks. However, the black box behaviour of these networks makes it very difficult to assign them tasks that an expert human normally does. This project aims to interpret in human terms what a convolutional neural network trained to classify fetal different ultrasound planes is based on. We use transfer learning to build a network with good performance in the classification task and apply interpretability techniques on it. These methods include Activation Maximization, Saliency Maps, Occlusion Sensitivity Maps, Class Activation Mapping and LIME. The trained network is able to classify fetal ultrasound images with an accuracy of 91.7%, and we provide a robust interpretation of its performance that allows us to understand the most important characteristics of each class for the model.
Full text
VISUAL INTERPRETABILITY OF DEEP LEARNING ALGORITHMS IN MEDICAL APPLICATIONS A Master's Thesis submitted to the faculty of Escola Tècnica d'Enginyeria de Telecomunicació de Barcelona Universitat Politècnica de Catalunya by Christian Jorba Soler In partial fulfilment of the requirements for the degree of MASTER IN ENGINEERING PHYSICS Advisors: Raúl Benítez & Daniel Malagarriga Barcelona, September 2020
1 Abstract Artificial intelligence is having a very big boost in recent times, and after the success of deep learning algorithms in many applications, they are also providing successful results for medical imaging, especially because of the good performance of convolutional neural networks. However, the black box behaviour of these networks makes it very difficult to assign them tasks that an expert human normally does. This project aims to interpret in human terms what a convolutional neural network trained to classify different fetal ultrasound planes is based on. We use transfer learning to build a network with good performance in the classification task and apply interpretability techniques on it. These methods include Activation Maximization, Saliency Maps, Occlusion Sensitivity Maps, Class Activation Mapping and LIME. The trained network is able to classify fetal ultrasound images with an accuracy of 91.7%, and we provide a robust interpretation of its performance that allows us to understand which are the most important characteristics of each class for the model.
2 DEDICATION Als meus pares i a la meva germana, per haver-me ajudat a arribar fins aquí. També als meus avis, per tot el que han fet per mi. Gràcies.
3 Acknowledgements I would like to acknowledge to my tutors Raúl Benítez and Daniel Malagarriga for their patience, support and the desire that they have put in this project and transmitted to me. I also want to thank Professor Blas Echebarria for helping me find this project and for always offering to help me with bureaucratic matters. In addition, I thank to my managers at work, Mireia and Jordi, for having continued count on me while I was studying this master.
4 Contents 1 Introduction ..................................................................................................................................................... 9 1.1 Background .................................................................................................................................................................. 9 1.2 Problem statement ................................................................................................................................................ 10 1.3 Project Overview ..................................................................................................................................................... 11 1.3.1 Requirements and specifications ............................................................................................................. 11 1.3.2 The data: fetal ultrasounds ......................................................................................................................... 12 1.3.3 Programming environment ........................................................................................................................ 15 1.4 Theoretical Background ...................................................................................................................................... 15 1.4.1 Machine Learning ............................................................................................................................................ 15 1.4.2 Artificial Neural Networks .......................................................................................................................... 17 1.4.3 Training an artificial neural network ..................................................................................................... 18 1.4.4 Activation Functions ...................................................................................................................................... 21 1.4.5 Deep learning and Convolutional Neural Networks ....................................................................... 22 2 State of the Art ............................................................................................................................................... 26 2.1 Deep Learning in Healthcare ............................................................................................................................ 26 2.2 Deep Learning Interpretability Methods ..................................................................................................... 27 3 Methodology .................................................................................................................................................. 29 3.1 Image Preprocessing............................................................................................................................................. 29 3.2 Data Augmentation ............................................................................................................................................... 34 3.3 Building a CNN with transfer learning ......................................................................................................... 36 3.3.1 Transfer Learning ............................................................................................................................................ 36 3.3.2 Transfer Learning with Inception V3 ..................................................................................................... 38 3.3.3 Validation methods ......................................................................................................................................... 39 3.4 Description of the interpretability methods ............................................................................................... 41 3.4.1 Activation Maximization .............................................................................................................................. 41 3.4.2 Saliency Maps .................................................................................................................................................... 42 3.4.3 Occlusion Sensitivity Maps .......................................................................................................................... 43 3.4.4 Class Activation Mapping ............................................................................................................................. 44 3.4.5 LIME ....................................................................................................................................................................... 46 4 Results .............................................................................................................................................................. 52 4.1 Image Preprocessing and Data Augmentation ......................................................................................... 52
5 4.2 Building the CNN .................................................................................................................................................... 54 4.3 Applying Interpretability Methods ................................................................................................................. 56 4.3.1 Activation Maximization .............................................................................................................................. 56 4.3.2 Saliency Maps .................................................................................................................................................... 57 4.3.3 Occlusion Sensitivity Maps .......................................................................................................................... 58 4.3.4 Class Activation Mapping ............................................................................................................................. 60 4.3.5 LIME ....................................................................................................................................................................... 63 4.4 Interpretation of Misclassifications................................................................................................................ 65 5 Discussion ....................................................................................................................................................... 68 6 Conclusions and future work .................................................................................................................. 71 7 References ...................................................................................................................................................... 72 Annex .......................................................................................................................................................................... 77
6 List of Figures FIGURE 1.1 COMPARISON BETWEEN TRADITIONAL MACHINE LEARNING PROCESSES AND CNNS ................................... 10 FIGURE 1.2 FETAL ULTRASOUND IMAGE OF THE BPD CLASS .................................................................................................. 13 FIGURE 1.3 FETAL ULTRASOUND IMAGE OF THE CRL CLASS................................................................................................... 14 FIGURE 1.4 FETAL ULTRASOUND IMAGE OF THE NT CLASS ..................................................................................................... 15 FIGURE 1.5 SUPERVISED LEARNING SCHEME: DOG-CAT CLASSIFIER ...................................................................................... 16 FIGURE 1.6 UNSUPERVISED LEARNING SCHEME: CLUSTERING ................................................................................................ 17 FIGURE 1.7 A FOUR-LAYER FULLY-CONNECTED ANN STRUCTURE........................................................................................ 17 FIGURE 1.8 TRAINING PROCESS OF AN ANN .............................................................................................................................. 19 FIGURE 1.9 REPRESENTATION OF THE GRADIENT DESCENT ALGORITHM............................................................................ 20 FIGURE 1.10 REPRESENTATION OF OVERFITTING, UNDERFITTING AND A CORRECT GENERALIZATION ......................... 20 FIGURE 1.11 GENERAL STRUCTURE OF A CNN .......................................................................................................................... 23 FIGURE 1.12 CONVOLUTION BETWEEN THE INPUT I AND THE KERNEL K ............................................................................ 23 FIGURE 1.13 MAXPOOLING AND AVGPOOLING LAYER EXAMPLES WITH STRIDE 1............................................................. 24 FIGURE 1.14 EXAMPLE OF DROPOUT IN A FULLY-CONNECTED NN ....................................................................................... 25 FIGURE 2.1 GLOBAL AI INVESTMENTS IN HEALTHCARE (SOURCE: CBINSIGHTS) ............................................................. 26 FIGURE 2.2 RELATION BETWEEN MACHINE LEARNING TECHNIQUES AND INTERPRETABILITY. (SOURCE: DARPA) .. 28 FIGURE 3.1 ON THE LEFT, IMAGE TAKEN WITH A SIEMENS MACHINE. ON THE RIGHT, IMAGE TAKEN WITH A GEVOLUSON MACHINE. .............................................................................................................................................................. 29 FIGURE 3.2 IMAGE PREPROCESSING FLOW .................................................................................................................................. 30 FIGURE 3.3 FINDING THE YELLOW THRESHOLDS IN MATLAB COLOR THRESHOLDER ....................................................... 31 FIGURE 3.4 EXAMPLE OF THE AVERAGING FILTER APPLICATION ............................................................................................ 31 FIGURE 3.5 IMAGE BEFORE (LEFT) AND AFTER (RIGHT) REMOVING YELLOW, BLUES AND GREEN PIXELS ..................... 32 FIGURE 3.6 IMAGE AND HISTOGRAM BEFORE (LEFT) AND AFTER (RIGHT) APPLYING THE INTENSITY ADJUSTMENT METHOD ................................................................................................................................................................................... 32 FIGURE 3.7 EXAMPLE OF THE MEDIAN FILTER APPLICATION .................................................................................................. 33 FIGURE 3.8 COMPARISON OF THE OUTPUT AFTER APPLYING A MEDIAN FILTER WITH KERNEL 5X5 AND 15X15 ........ 33 FIGURE 3.9 RESIZING AND CONVERSION FROM GRAYSCALE TO RGB..................................................................................... 34 FIGURE 3.10 EXAMPLE OF TECHNIQUES USED IN THE DATA AUGMENTATION PROCESS .................................................... 34 FIGURE 3.11 EXAMPLE OF HORIZONTAL FLIP APPLICATION.................................................................................................... 35 FIGURE 3.12 EXAMPLE OF TRANSFER LEARNING....................................................................................................................... 37 FIGURE 3.13 INCEPTION V3 ARCHITECTURE. (SOURCE: [59]) .............................................................................................. 38 FIGURE 3.14 OCCLUSION SENSITIVITY PROCESS ........................................................................................................................ 43 FIGURE 3.15 CLASS ACTIVATION MAPPING PROCESS ............................................................................................................... 44 FIGURE 3.16 GAP LAYER EXAMPLE .............................................................................................................................................. 45 FIGURE 3.17: LIME METHOD SCHEME (SOURCE: [36]) ........................................................................................................... 47 FIGURE 3.18 SUPERPIXELS GENERATED USING THREE DIFFERENT ALGORITHMS ............................................................... 47 FIGURE 3.19 EXAMPLE ABOUT THE LINEAR MODEL CONSTRUCTION..................................................................................... 48 FIGURE 3.20 COMPARISON OF THE RESULTS USING LASSO PATH AND FORWARD SELECTION .......................................... 48 FIGURE 3.21 SCORE OF THE LINEAR MODEL AS A FUNCTION OF THE NUMBER OF SUPERPIXELS...................................... 49
7 FIGURE 3.22 PERTURBATIONS GENERATED AVERAGING THE SUPERPIXEL .......................................................................... 50 FIGURE 3.23 EXAMPLE OF THE LINEAR MODEL WEIGHTS ........................................................................................................ 50 FIGURE 3.24 REPRESENTATION OF THE 5 MOST IMPORTANT FEATURES ............................................................................. 51 FIGURE 4.1 PRE-PROCESSED IMAGES OF EACH CLASS ............................................................................................................... 53 FIGURE 4.2 SYNTHETIC IMAGES OF EACH CLASS ........................................................................................................................ 53 FIGURE 4.3 EVOLUTION OF ACCURACY AND LOSS DURING THE TRAINING PROCESS WITHOUT DATA AUGMENTATION 54 FIGURE 4.4 EVOLUTION OF ACCURACY AND LOSS DURING THE TRAINING PROCESS WITH DATA AUGMENTATION ....... 55 FIGURE 4.5 INPUT THAT MAXIMIZES THE OUTPUT AND EXAMPLE OF EACH CLASS.............................................................. 57 FIGURE 4.6 SALIENCY MAPS OF EACH CLASS .............................................................................................................................. 58 FIGURE 4.7 RESULTS AFTER APPLYING OCCLUSION ON BPD IMAGES .................................................................................... 59 FIGURE 4.8 RESULTS AFTER APPLYING OCCLUSION ON CRL IMAGES .................................................................................... 59 FIGURE 4.9 RESULTS AFTER APPLYING OCCLUSION ON NT IMAGES ...................................................................................... 60 FIGURE 4.10 RESULTS AFTER APPLYING THE CAM METHOD ON BPD IMAGES .................................................................. 61 FIGURE 4.11 RESULTS AFTER APPLYING THE CAM METHOD ON CRL IMAGES ................................................................... 62 FIGURE 4.12 RESULTS AFTER APPLYING THE CAM METHOD ON NT IMAGES ..................................................................... 62 FIGURE 4.13 RESULTS AFTER APPLYING LIME ON BPD IMAGES .......................................................................................... 63 FIGURE 4.14 RESULTS AFTER APPLYING LIME ON CRL IMAGES ........................................................................................... 64 FIGURE 4.15 RESULTS AFTER APPLYING LIME ON NT IMAGES ............................................................................................. 65 FIGURE 4.16 MISCLASSIFICATIONS REPRESENTED USING THE CONFUSION MATRIX .......................................................... 66 FIGURE 5.1 COMPARISON OF THE EXPLANATION OF MISCLASSIFIED IMAGES....................................................................... 70
14 flexed, amniotic fluid should be visible between the fetal chin and chest. Once CRL is obtained, there are many tables or formulas to know the approximate age of the fetus, however, the Societat Catalana d’Obstetricia i Ginecologia recommends using the table published by Robinson and Fleming in 1975 (Robinson HP 1975) [14]. Figure 1.3 Fetal ultrasound image of the CRL class Nuchal Translucency thickness (NT) The nuchal translucency is an accumulation of fluid, coming from the paracervical lymphatic system, which can be observed in the fetal nape. The increase of the liquid in this region can be associated with various fetal malformations, especially cardiac and chromosomal abnormalities like trisomy 21 and 18 [13]. This is the most important measurement in the first trimester pregnancy and should only be done by trained and certified operators. NT should be measured in this time window (between 11 weeks and 13 weeks and 6 days) because it generally corresponds to a CRL measurement of between 45 and 84 mm. This gestational age window is chosen because NT as a screening test performs optimally and fetal size allows diagnosis of major fetal abnormalities. For the measurement, the fetus should be in a neutral position, and a sagittal section should be obtained and the image should be magnified in order to include only the fetal head and upper thorax. Furthermore, the amniotic membrane should be identified separately from the fetus. The median view of the fetal face is defined by the presence of tip of the nose and rectangular shape of the palate anteriorly, the translucent diencephalon in the center and the nuchal membrane posteriorly. If the section is not exactly median, the tip of the nose will not be visualized. Once the plane is correct, NT is measured as the maximum distance between the nuchal membrane and the edge of the soft tissue overlying the cervical spine.
15 Figure 1.4 Fetal ultrasound image of the NT class 1.3.3 Programming environment As mentioned above, this project has been developed with libraries based on Python [9], an open source programming language widely used for machine learning projects, computer vision and image processing, since most of the algorithms are implemented. Specifically, Keras library [10] has been used to train the CNNs, which run on top of TensorFlow, allowing easy and intuitive programming [11]. For Image preprocessing, the Matlab tool Color Thresholder [55] has been used to find the colour thresholds in the image to remove annotations. 1.4 Theoretical Background 1.4.1 Machine Learning Machine learning (ML) is a subset of artificial intelligence that provides computers with the ability to learn without being programmed with explicit rules [40]. ML enables the creation of algorithms that can learn and make predictions. In contrast to rules-based algorithms, a machine learning algorithm is applied to a set of data then it can learn from the training data and apply what it has learned to make a prediction [41]. ML is having a substantial effect on many areas of technology and science, such as speech recognition, self-driving cars, product recommendations, medical diagnosis, automatic language translation and financial data analysis [42-43].
16 ML algorithms can be classified, basically, in two categories based on different principles in the learning process. These two categories are called supervised learning and unsupervised learning [44]. Supervised Learning Supervised learning starts with the goal of predicting a known output or target. For that, each sample contains two parts: one is the input observations or features and the other is output observations or labels. For instance, if we want to train a model to classify images about cats and dogs, we need the images and the label of each one, that is, the classification of the images in the category of cat or dog (Figure 1.5). The expected outputs are usually labelled by human experts and serve as ground truth for the algorithm [40]. The purpose of supervised learning is to deduce a functional relationship from training data that generalizes well to testing data, and it focuses on classification and prediction tasks [45]. Figure 1.5 Supervised learning scheme: dog-cat classifier Unsupervised Learning In unsupervised learning, we only have one set of observations and there is no label information for each sample (Figure 1.6). The main purpose of unsupervised learning is to discover patterns and hidden structure in the data to separate it into clusters or groups, and it is done without the algorithm system being provided with information regarding what the groups are [45]. The groups learned by unsupervised learning are labelled later and evaluated by supervised learning tasks in order to know if these new patterns are useful in some way. Hence, clustering is the most common use of that kind of algorithms [41][43].
17 Figure 1.6 Unsupervised learning scheme: clustering 1.4.2 Artificial Neural Networks Artificial neural networks (ANNs) or neural networks (NN) are machine learning techniques inspired by the brain in the way it learns and processes information through a large number of highly interconnected processing elements, which are called neurons, nodes or cells [46]. During the forward pass [46], each neuron receives inputs from multiple sources, performs a weighted sum and computes the activation function to the summed data, which applies a non-linear transformation [47] and the result is propagated to the next layer, where the same process is repeated in each neuron. The structure of an artificial neural network (Figure 1.7) is composed by an input layer of neurons, one or more “hidden layers”, and one output layer. In hidden and output layers, each neuron is fully connected to all neurons in the previous layer. The strength of each connection is quantified with its own weight. For the network to yield the correct outputs, the weights must be set to suitable values, which are estimated through a training process. Figure 1.7 A four-layer fully-connected ANN structure
18 1.4.3 Training an artificial neural network As mentioned in the previous section, the connection between neurons of different layers in an ANN is defined by the weights and bias that each neuron applies in the dot product with the outputs of the previous layer. Therefore, the final output of the ANN depends on the values of that weights and biases, and the main goal of the training process is to find a combination of these parameters that produces an accurate mapping between the inputs and the corresponding labels or targets. Training an ANN is the same as using the data to minimize a loss function L, also called cost function, which serves to score the output of the ANN. The lost function must be defined convex since optimizing a convex function is computationally more tractable [53]. There are many types of loss functions, depending on the problem (Table 1.1). Table 1.1 Three most used cost functions according to the type of problem Loss function Type of problem Mean Squared Error Regression Binary Cross-Entropy Binary Classification Multi-Class Cross-Entropy Multi-class classification problems The loss function depends on the outputs 𝑦𝑖, which in turn depend on weights and biases, and targets 𝑦𝑖. Hence, the strategy to minimize the loss function (Figure 1.8) is as follows: 1. Split the data into two sets: the training data and the test data. The test set is normally 15-20% of the total data. Sometimes a third group is included, the validation set. 2. Use the training set to perform a forward pass in the ANN in order to compute the derivative of the loss function with respect to all weights and biases of the ANN. This is done by the Backpropagation algorithm [54].
19 Figure 1.8 Training process of an ANN 3. Use a Gradient Descent method to update weights and biases. As you can see in Equation 1.1 and 1.2, we are modifying the value of weights and biases by moving in the opposite direction to the gradient of the loss function, since the gradient points in the direction of greatest growth of the function (Figure 1.9). The magnitude of the change in the parameters is regulated by the learning rate, 𝜂 [46]. The other parameter present in the Equations is M, which establishes the size of the mini batch which is the number of samples we use before updating the weights and biases. If they update with each sample, the algorithm is called Stochastic Gradient Descent (SGD), whereas if they update after calculating the mean gradient of M samples it is called Mini-Batch Gradient Descent, and finally, if the parameters update with the mean gradient of all samples it is called Batch Gradient Descent. 𝑤𝑛+1 =𝑤𝑛−𝜂 𝑀∑∇𝑤𝐿𝑘 𝑀 𝑘=1 (1.1) 𝑏𝑛+1 =𝑏𝑛−𝜂 𝑀∑∇𝑏𝐿𝑘 𝑀 𝑘=1 (1.2) 4. Once all training samples have been used to compute the gradients, a new epoch can start again with the updated weights and biases. An epoch is defined as a complete run through the training data [53]. 5. When the last epoch has finished, use the test dataset to check the training result. If it went correctly, we have to get good and similar results both for the training set and for the test set. If not, overfitting or underfitting may have occurred, and we should use another set of hyperparameters (learning rate, mini batch size, epochs, hidden layers, etc), change the model or increase the dataset, depending on the situation.
20 Figure 1.9 Representation of the Gradient Descent algorithm Overfitting Overfitting is the effect of overtraining a model (Figure 1.10), which has a good performance on the training data, but not on new data. That means that the algorithm is not able to generalize, which is probably the most fundamental concept in machine learning [53], because it has learned so many useless details from the training data. It can be produced by many factors, such as a high number of epochs, but also by having a small training dataset or a model that is too complex for our data, so it is learning too much from the data. Therefore, the solution could be to change the hyperparameters of the model, the model itself, or increase the training dataset. Underfitting Unlike overfitting, underfitting occurs when the model generalizes too much because of being too simple to fit the data well and represent all relevant class characteristics (Figure 1.10). Therefore, an under-fitted model performs poorly with both training and test data, and the solution could be to find a more complex model. Figure 1.10 Representation of overfitting, underfitting and a correct generalization
21 1.4.4 Activation Functions The power of NN is that they can learn complex patterns in the data and this is achieved by the activation functions, which add non-linearity into a NN allowing to create complex mappings between inputs and outputs [47]. In an artificial neuron, the activation function is applied on the dot product between inputs and weights, so it is also essential in deep neural networks in order to restrict the range of the outputs along the NN (between 0 and 1, for instance). But the most important thing is that the activation function establishes whether the neuron is activated or not, propagating the signal to the next layer or acting as a firewall. There are many non-linear activation functions [48], but in this section we are going to focus on the two most common functions and the ones that we have used in this project. Rectified Linear Unit (ReLU) function ReLU is the most widely used activation for deep learning applications since it offers better performance and generalization [48]. The ReLU activation function performs a threshold operation to each input element where values less than zero are set to zero, thus the ReLU function is given by Equation 1.3. 𝑅𝑒𝐿𝑈(𝑥)=max(0,𝑥)={𝑥, 𝑖𝑓 𝑥≥0 0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 (1.3) There are some variants of ReLU which are Leaky ReLU, Parametric ReLu or Exponential ReLU, among others [48]. Softmax Function The Softmax function is another type of activation function widely used in neural networks. It is commonly used in the output layers in multi-class classification problems where it returns the probability that an input belongs to a certain class, with the target class having the highest probability [48]. The Softmax function produces an output which is between 0 and 1, with the sum of probabilities being equal to 1, and it is computed using the Equation 1.4, where 𝑥𝑖 is the output of the weighted sum in the neuron i. The particularity of this function is that the probability of belonging to a certain class depends on the score of all the output neurons, not just the neuron that represents a certain class.
22 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑥𝑖)=exp(𝑥𝑖) ∑𝑒𝑥𝑝(𝑥𝑗) 𝑗 (1.4) 1.4.5 Deep learning and Convolutional Neural Networks Deep learning is a subset of artificial neural network algorithms that contain many hidden layers (typically more than 20) [41]. Deep learning models can be categorized as normal networks that take one-dimensional vectors as inputs, or convolutional neural networks (CNNs) that take two-dimensional or three-dimensional shaped inputs [40]. We will focus on CNNs because there are most commonly applied to images. CNNs are a specialised kind of deep neural networks that use convolution in place of general matrix multiplication in at least one of their layers, which is an integral that expresses the amount of overlap of one function g as it is shifted over another function f (Equation 1.4). An important benefit of CNNs, as compared with other machine learning methods, is that there is no need to compute features as a first step. CNNs effectively find the important features as a part of its search process [41]. In addition, CNNs are designed to be spatial invariant, that is, they are not sensible to the position of an object but to the patterns of that object. Therefore, if in one input there is a ball in the upper right and in the next sample the ball is at the bottom left, the CNN should be able to detect it in both cases. 𝑓∗𝑔≡∫ 𝑓 ∞ −∞ (𝜏) 𝑔(𝑡−𝜏) 𝑑𝜏 (1.4) The standard model of CNN has a structure (Figure 1.11) consisting of the input layer, alternating convolutional, pooling, and activation layers. The latest consists of a small number of fully-connected layers with the Softmax activation function in the output layer [49]. Dropout is also often used as a regularization layer. Thus, the basic layers of a CNN are listed below.
23 Figure 1.11 General structure of a CNN Input layer The input is usually a multidimensional array of data such as an image. Convolutional layers It is the main building block of CNN and it is made of small kernels (Figure 1.12) that allow extracting high-level features from the input, and each kernel is used to calculate a feature map. The first convolutional layer extracts low-level meaningful features such as edges, corners, textures and lines. Next convolutional layer extracts higher-level features, but the highestlevel features are extracted in the last convolution layer [50]. Kernel size refers to the size of the filter, which convolves around the feature map while the amount by which the filter slides (sliding process) is the stride. It controls how the filter convolves around the feature map. Another essential feature of CNNs is the padding that gives the option to make input data wider [14] by adding zeros to the outside of the image. Figure 1.12 Convolution between the input I and the kernel K
30 Once we have got the cropped images, the flow performed to preprocess the images is reflected in Figure 3.2. Figure 3.2 Image preprocessing flow Remove annotations and marks First step is to remove marks and annotations that we cannot remove easily because they are on anatomic regions, as the yellow line indicating the longitude of some feature. This information could be learned by the classification algorithms and could have a relevant weight in the classification result. Therefore, in order to avoid it, all this irrelevant information has to be removed and keep only what we want the algorithm learns. This task has to be done very carefully, trying to do the minimum impact on the anatomical structures. For this task, we have used the Matlab tool Color Thresholder [55] because it is a very simple way to get the maximum and minimum thresholds of a given colour in an RGB image. So, using this tool, we can find the range in the RGB channels where a certain colour lives, as we can see clearly in Figure 3.3, where the yellow region has been found. The same procedure is followed to obtain the thresholds for blue and green pixels using several images, and the results are in Table 3.1. Table 3.1 Threshold for blue, green and yellow colours in each RGB channel Red Channel Green Channel Blue Channel Pixels Minimum Threshold Maximum Threshold Minimum Threshold Maximum Threshold Minimum Threshold Maximum Threshold Blues 50.0 150.0 0.0 255.0 160.0 255.0 Greens 0.0 70.0 100.0 245.0 0.0 90.0 Yellows 60.0 255.0 180.0 255.0 0.0 190.0
31 Figure 3.3 Finding the yellow thresholds in Matlab Color Thresholder Once we have the range of the pixel values that we want to remove in the original image, we can create a copy of the image, convert it to grayscale and apply an averaging filter on it. It changes the value of each pixel for the average of the specified number of neighbours (Figure 3.4). In our case, we have chosen a kernel size of 25 since the original images have a high resolution. Figure 3.4 Example of the averaging filter application Now, we can change those pixels of the original grayscale image which match with the values of Table 2 with those of the same position in the filtered grayscale image. By doing that, we manage to remove annotations and marks of the original image and replace those pixels for an averaged value of the neighbours, making the effect on the image minimal (Figure 3.5).
32 Figure 3.5 Image before (left) and after (right) removing yellow, blues and green pixels Intensity adjustment As we will work with grayscale images, we need to highlight the main features of them. This is important to facilitate the extraction of that features by the convolutional neural networks that have to classify the images. Therefore, for this propose, we have used the Python’s collection for image processing Scikit-Image, specifically a function that calculates a minimum threshold, which is the pixel value from which the 1% of the total pixels are lower, and the same for a maximum threshold, but in this case the 1% of the pixels are higher. Once the thresholds are computed, the whole image is balanced in order to set the minimum and maximum threshold to 0 and 255, respectively. We can see the obtained results in Figure 3.6. Figure 3.6 Image and histogram before (left) and after (right) applying the intensity adjustment method
33 By doing that, we have saturated the bottom 1% and the top 1% of all pixel values achieving an increment of the contrast in the output image without increasing the noise, which is what happens using other techniques such as the histogram equalization [7]. Artifact removal This step is performed in order to finish smoothing those pixels changed in the first step corresponding to some marks and annotations. For that, we use the median filter, which is the most common filter used to reduce the noise because it preserves edges after doing it. This filter changes the value of each pixel by the median of the pixels around it (Figure 3.7), which is set by the kernel size. The median is calculated by first sorting all the pixel values from the surrounding neighbourhood into numerical order and then replacing the pixel being considered with the middle pixel value. Figure 3.7 Example of the median filter application After doing some tests, we have chosen a kernel size of 5x5 because we have considered that this size successfully removes any marks left on the image without over-smoothing it (Figure 3.8). Figure 3.8 Comparison of the output after applying a median filter with kernel 5x5 and 15x15
34 Resize The last step in the preprocessing process is to resize the images in the correct format depending on the CNN that we will use to classify them. In this project, we have used the network Inception V3, which default input has the shape (299, 299, 3). As we can see, it has three channels because this network is designed to work with colour images. In our case, as we have modified the images to grayscale (one channel), the output will be an image with three channels with exactly the same information in each of them (Figure 3.9). So, basically in this step we have to create three copies of the same grayscale image and resize them with the correct size before saving each class in a different directory. Figure 3.9 Resizing and conversion from grayscale to RGB 3.2 Data Augmentation Image data augmentation is a technique that can be used to artificially expand the size of a training dataset by creating modified versions of images in the dataset [56]. This can improve the performance and ability of the model to generalize and also avoid overfitting. Transforms include a range of operations from the field of image manipulation, such as shifts, zooms, rotations, flips, etc (Figure 3.10). After preprocessing the images, they have been divided randomly into a training (80%) and a test dataset (20%). So, we have an array of 1068 images corresponding to the training dataset, and another with 459 for testing. Data augmentation technique is applied on the training images since we want the method to increase the generalization during learning, while the validation tests must be done with original images. Figure 3.10 Example of techniques used in the data augmentation process
35 Thus, we have applied the function class ImageDataGerator(), which belongs to the Keras library to generate 196, 189 and 194 synthetic images of the class BPD, CRL and NT, respectively. To this function, we have specified the maximum value of the variations that it has to perform to the images, and the algorithm applies a combination of all in each iteration. For example, if the maximum angle that the image can rotate is ±20º, this function will generate images with rotations between -20º and 20º plus the corresponding shift, flip or zoom. The same behaviour applies to the other variables. In our case, the rotation angle has been chosen as 15º, the horizontal and vertical shifts as a 10% of the total width/height, the shear as 20% and the horizontal flip is enabled, so it is applied randomly (Figure 3.11). Zoom has been discarded because the original images already have the correct zoom to be able to take the ultrasound image correctly, and it is different for each class. So, it is important to keep it fixed. #Definition of the synthetic images generator datagen = ImageDataGenerator( rotation_range=15, width_shift_range=0.1, height_shift_range=0.1, shear_range=0.2, horizontal_flip=True) Figure 3.11 Example of horizontal flip application Finally, the synthetic images have been joined to the array of the corresponding class, which contains the original images for training. Then the synthetic and original images have been shuffled and saved to the corresponding folder. Therefore, our dataset is composed by a training dataset consisting of 1800 (85%) original and synthetic images, containing 600 images per class, on the one hand, and by a test dataset with 102 original images per class (15%), on the other.
36 Table 3.2 Composition of the dataset BPD CRL NT Original Synthetic Original Synthetic Original Synthetic Training 404 196 411 189 406 194 Test 102 - 102 - 102 - 3.3 Building a CNN with transfer learning The objective of this section is to build a CNN with a good performance classifying the ultrasound images in order to apply the interpretability methods on it. The training has been carried out first without data augmentation, and later it has been repeated with the synthetic data. Thus, the technique used, the architecture of the network and the validation methods will be explained. 3.3.1 Transfer Learning Usually, a lot of data is needed to train a neural network from scratch, but as shown in Table 3.2, we have a small training dataset of 1800 images, which can lead to a poor generalization. Additionally, the process to select the most suitable model (model selection) tends to take a long time since the process of training a CNN from scratch is very expensive, and in our case, we just want to find a model with a good performance to apply the interpretability methods, and not spend much time looking for the best one. This is where transfer learning plays an important role since the main advantages of it are saving training time, better performance of neural networks (in most cases), and not needing a lot of data. The basic idea of transfer learning is to take advantage of the knowledge of a pre-trained network to classify images of another specific problem, which in our case are the fetal ultrasound images. The advantage of this is that these pre-trained neural networks have been trained with huge datasets with a lot of classes, for example the ImageNet dataset [57], which contains 1.2 millions of labelled images of 1000 different classes (1200 images per class). Then, a CNN that is able to differentiate between images of 1000 different classes with high accuracy, it is because the model selection and training have been carried out adequately so that the CNN is able to extract a large number of patterns from the samples and classify them in the correct class. The fact that these pre-trained CNNs have not been trained with medical images is not much relevant because they have been trained to extract patterns that are also present in our images, such as shapes, edges, etc. Therefore, the parts of the pre-trained CNN that we have to take advantage of are the convolutional layers, which are the feature extractors, and we
37 just have to change and train the classifier (or part of it) to adapt it to our problem or, at most, also the last convolutional layers, which is called fine-tunning. In our case, only the last layer of the network has been changed so that it has only three neurons, which represent the three categories of our problem (Figure 3.12). Figure 3.12 Example of transfer learning So, all we have to do is remove the layers from the pre-trained model that we are not interested in and perform a forward pass with our dataset. The result of this process will be the new inputs that we will use to train the new fully connected layers that act as a classifier of the features extracted by the convolutional layers. Once we have the classifier trained with our dataset, it is time to join it to the rest of the pre-trained model (without the removed layers), and we will have completed the transfer learning process. The Keras library has many pre-trained networks available in a very simple way. For example, if we want to use the VGG19 pre-trained CNN with the ImageNet dataset, we just have to do from keras.applications.vgg19 import VGG19 model = VGG19(weights='imagenet') For this project, the CNNs Inception V3, pre-trained with the ImageNet dataset, which is widely used for classification tasks, has been chosen to carry out transfer learning and use its “knowledge” to classify our ultrasound images properly. Although finally the selected network has been Inception V3, the transfer learning process was also carried out with the VGG19 network [61], and the code and results can be found in A.3.
38 3.3.2 Transfer Learning with Inception V3 Inception V3 [59] is a deep convolutional architecture widely used for classification tasks. It has multiple symmetric and asymmetric building blocks, where each block has several branches of convolutions, average pooling, max-pooling, concatenated, dropouts, and fullyconnected layers. This network has 313 total layers and possesses 29.3 million of trainable parameters. The input of this network has to be the shape (299, 299, 3) and values between -1 and 1. The output of the feature extractor section is a vector of 2048 elements, since the last layer of this part is a Global Average Pooling layer (GAP), which takes a feature map of the shape (8, 8) as input, and returns its average as an output. So, if the number of feature maps with shape (8, 8) generated in the last convolutional layer is 2048, the output of the GAP layer will be a vector of 2048 elements, which are the inputs of the single FC layer of 1000 neurons and Softmax activation function that acts as a classifier. Figure 3.13 Inception V3 architecture. (Source: [59]) In this case, the part of the CNN that we want to take to carry out the transfer learning is from the input layer to the GAP layer. So, we have to remove the last FC layer and add a new classifier trained with the fetal ultrasound images. The process used to remove the FC layer is very simple with Keras: from keras.models import Model feature_extractor = Model(inputs=model.input, outputs=model.get_layer('avg_pool').output) Now, we have to perform a forward pass with our dataset, since the result will be the inputs that we will use to train a new classifier. x_train = feature_extractor.predict(x_train) y_test = feature_extractor.predict(y_test)
39 Once we have the data to train a new classifier, we have to define it, that is, we have to specify the hyperparameters such as the number of layers, the learning rate, the activation function, the optimizer, the number of epochs, the batch size, etc. To make this selection, we use the GridSearchCV function, which evaluates the performance of the model using different combinations and returns de best one, which is the one we see below def define_classifier(): sgd = optimizers.SGD(lr=0.001, momentum=0.8, nesterov=False) classifier = Sequential() classifier.add(Dropout(0.3)) classifier.add(Dense(3, name='classificador', activation='softmax')) classifier.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy']) return classifier As you can see, the best classifier found has only one layer of neurons, while it uses a dropout layer to reduce overfitting. Also, the optimal values for the learning rate, number of epochs and batch size are 0.001, 50 and 20, respectively. If we already know the optimal architecture of our classifier, we have to train it, join it to the rest of CNN, and finally save it for later use with the interpretability methods. classifier = define_classifier() history=classifier.fit(x_train, np_utils.to_categorical(y_train), epochs=50, batch_size=20, \ validation_data=(x_test, np_utils.to_categorical(y_test)), verbose=2) x = feature_extractor.output x = classifier(x) model = Model(inputs=feature_extractor.input, outputs=x) model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy']) model.save('./MODELS/InceptionV3/full_model.h5') It is important to note that if we are able to successfully carry out this transfer learning process, the number of trainable parameters will be 6147, which is a number far less than hundreds of thousands or millions of trainable parameters that any CNN has. Therefore, we will have managed to reduce the time invested in training and also in the model selection. 3.3.3 Validation methods To validate that the training process has been carried out satisfactorily, and that the model has been able to learn to classify the images of fetal ultrasound, we are going to use three metrics which are the accuracy of the model, the sensitivity and the specificity, all of them computed from the performance of the model with the test dataset. The last two are
46 will have the same size that the feature maps 𝐴𝑘, and shows the regions of the last feature maps that have contributed the most to the final score. Finally, as no spatial information is lost, the heat map can be resized to the size of the input image in order to overlaid them and see what regions of the input the CNN are considering as important to classify it as class c. 𝐶𝐴𝑀𝑐=∑𝑤𝑘 𝑐 𝑘𝐴𝑘 (3.8) This method strictly depends on the architecture mentioned above, but there is a generalization for any type of CNN called Grad-CAM [35], which takes the gradients of the output respect to the feature maps to compute the weights. Fortunately, our model has the required architecture, so we will focus just on the CAM method. Therefore, to apply this method, we have to load our model and take only those parts that we are interested in. On the one hand, we want the weights of the last layer, and on the other hand, we need to obtain the feature maps generated just before the global average pooling layer, that is, those generated by the “mixed10” layer. For that, we have to remove the layers after the “mixed10” layer, and save the resulting model in the variable “extractor”. model = load_model('./MODELS/InceptionV3/full_model.h5') weights = model.get_layer('classificador').get_weights()[0] extractor = Model(inputs=model.inputs, outputs=model.get_layer('mixed10').output) Once we have the weights and the feature maps, we can apply the Equation 3.8 to obtain the Class Activation Map of a desired image. image = cv2.imread('../IMATGES/InceptionV3/TEST/CRL/CRL_tst_0.jpg') class_to_explain = 1 fm = extractor.predict(preprocess_input(image[np.newaxis,...])) #computing the feature maps cam = np.dot(fm[0], weights[:,class_to_explain]) cam = cv2.resize(cam, (299, 299)) 3.4.5 LIME Local Interpretable Model-agnostic Explanations (LIME) [36] is a revolutionary method proposed in 2016 to explain the predictions of any classifier in a faithful way, by approximating it locally with an interpretable model. The method is called model-agnostic because it is able to explain any model, treating it as a black box and using only its inputs and outputs.
47 The basic idea of this method is to take a prediction in the predictions space and use the local neighbour points to build a classifier on top of them. This classifier is usually a linear regression, which is easily interpretable. Figure 3.17 shows the prediction space, where the pink colour indicates that the input belongs to the class, whereas blue is the opposite. The red cross is the prediction that we want to interpret, while the rest of crosses and circles are positive and negative predictions, respectively. Then, we would now use the predictions around which we want to explain to build a linear classifier that allows us to interpret the decision of the model. But, how can we know which samples are neighbours of the one we are interested in? The authors propose to use the superpixels. Figure 3.17: Lime method scheme (Source: [36]) A superpixel can be defined as a group of pixels that share common characteristics (like pixel intensity), and are found by image segmentation. In Figure 3.18 you can see an example of the superpixels found in an CRL image using different algorithms. So, the strategy is to randomly turn superpixels on and off, and make predictions with the resulting images, which should be close the prediction of the original image in the predictions space. Figure 3.18 Superpixels generated using three different algorithms To build the linear model, we encode the images generated by perturbing superpixels in feature vectors. In this case, the features are the superpixels, and the vector shows those that are activated and those that are not. In Figure 3.19, you can see an example of that,
48 where the black squares are disabled superpixels, which are equal to 0 in the feature vector, while the white ones are the active ones, which are represented by a 1 in the vector. After that, we can train a linear model weighting the samples, that is, giving more importance to those samples that are closer to the original one (note that the original vector is completely composed of 1). Once the linear model is trained, we obtain its weights which are interpreted as the importance of each feature or superpixel in the CNN decision. Figure 3.19 Example about the linear model construction There is the possibility of choosing with how many features we want to build the linear model, and there are two algorithms to select those features, which are lasso path and forward selection. In our case, different tests have been carried out and we have seen that the results of using one or another are generally the same (Figure 3.20). Thus, we have chosen lasso path for our analysis, which is computationally more efficient. However, we have not actually made use of either of the two algorithms since we have always worked with all the features to build the linear model, since that is how we have obtained the best score (Figure 3.21). This analysis should be done for all samples, but the computational cost is very high, so we are going to use all the superpixels in each case. Figure 3.20 Comparison of the results using lasso path and forward selection
49 Figure 3.21 Score of the linear model as a function of the number of superpixels The entire process described so far is integrated into a Python’s library called Lime, which is easily usable. The first thing we must do is generate an object of the class “lime_image”, which contains all the functions and algorithms that we need. In this case, it is only necessary to indicate the feature selection algorithm to build the linear model, but as we have already commented, it is not important in our case, since we are going to use all the superpixels. from Lime.lime import lime_image explainer = lime_image.LimeImageExplainer(verbose=True, feature_selection = 'lasso_path') Once we have the object, we can start the interpretability process. But before that, we have to decide how many samples we want to generate by randomly activating and deactivating superpixels. Obviously, the more samples the more realistic the result will be, but this has a very high computational cost, so you have to choose an intermediate number. In our case, we have seen that 500 is a suitable number of samples for the amount of superpixels that we generate in the samples, which are about 30. To generate them, we have different segmentation algorithms (Figure 3.18), but we have chosen the Quickshift [60] since it is the one that generates superpixels more or less of the same size and the one that we believe best recognizes the different regions of the image. To this algorithm, we indicate the corresponding parameters so that the maximum number of superpixels is about 30. Finally, another important parameter is the way superpixels are “disabled”, which consist of introducing a perturbation by changing the value of the pixels that make up the superpixel. In our case, we have chosen to change the values by the average of all the pixels in the superpixel (Figure 3.22), which is indicated by “hide_color = None” in the code, and it is the choice recommended by authors for gray-scale images. Def segmentation_algorithm(img): superpixels = quickshift(image, ratio=0.2, kernel_size=6, max_dist=200) return superpixels seg_alg = segmentation_algorithm
50 explanation = explainer.explain_instance(image, model.predict, top_labels=1, hide_color=None, num_samples=500, num_features=num_superpixels, segmentation_fn=seg_alg) Figure 3.22 Perturbations generated averaging the superpixel Once the explanation process ends, we can see the weights of the linear model, which indicate the importance of each superpixel in the decision of the model (Figure 3.23), and the score, which is the probability that the linear model gives to the input image (all superpixels turned on) to belong to the class we are trying to explain. If the linear model explains the prediction well locally, this probability must be similar to that given by the CNN. Note that the weights can be negative, indicating that this superpixel does not contribute to the image being classified in that class. Figure 3.23 Example of the linear model weights Finally, the most relevant superpixels, are plotted on the original image in order to see which areas contribute the most to the final decision (Figure 3.24). In our case, we have decided to do the analysis using only the 5 most important superpixels. num_features=5 fig, axs = plt.subplots(1, num_features+1, figsize=(20, 10)) axs[0].imshow(image) axs[0].set_title("Original", fontsize=20)
51 for i in range(1, num_features+1): temp, mask = explanation.get_image_and_mask(explanation.top_labels[0], positive_only=True, num_features=i, hide_rest=True) axs[i].imshow(image) axs[i].imshow(mask, cmap=cmap, alpha=0.6) axs[i].set_title(str(i) + ' feat', fontsize=20) plt.imshow() Figure 3.24 Representation of the 5 most important features
52 4 Results After seeing the methods chosen to carry out this project, as well as the parameters used in each one of them, we are ready to present and comment the results obtained. For that, we have divided this section on topics related to the dataset, the building of the model, and the application of the different interpretability methods. 4.1 Image Preprocessing and Data Augmentation Figure 4.1 shows some examples of the pre-processed images ready to train the neural network. As you can see, the marks have been removed successfully, and the most relevant anatomical regions have been enhanced. In the BPD class, the hemispheres are clearly visible, as well as the interhemispheric fissure and ossification of the skull. In some cases, the choroid plexuses are also visible. In the CRL class, the baby’s body is clearly differentiated due to the contrast with the amniotic fluid. In most cases, the chin, chest and legs are visible and also bones of the face or skull. In this class, the walls of the uterus have also been considerably enhanced. Finally, in the NT class, the results are also satisfactory, since the baby’s face is easily distinguished, as well as the nose, chin and other anatomical structures like the palate, diencephalon and the nuchal region, which is the most important feature of this class. Therefore, after seeing the results and observing that it has been possible to eliminate the marks and highlight the most relevant features of each class, we can say that the technique used to pre-process the images has been satisfactory and that the images are ready to train a neural network, or as in our case, use them first to generate synthetic images to increase the dataset. In Figure 4.2 we can see some of the synthetic images generated to train the model also divided by classes. With these results we can say that we have been successful in choosing small rotation angles and displacements since space that is generated after these movements is filled with an average of close values that can distort the image a bit. Despite this, these distortions do not affect any anatomical structure, so they can serve to increase the generalization of the model without having a negative influence.
53 Figure 4.1 Pre-processed images of each class Figure 4.2 Synthetic images of each class
54 4.2 Building the CNN In this part, we show the results of the model after training de classifier without using the synthetic data, and the results using them, since we think it is an interesting exercise to see the importance of the data in machine learning. Without data augmentation and using the optimal parameters found with the grid search technique, an accuracy of 89.9% is achieved after 50 epochs. Looking at Figure 4.3 the model is slightly overfitted, since it can not generalize to the testing data what has been learning with the training data. In addition, although the global accuracy is not bad, there is an imbalance between classes, especially between the BPD and NT, reaching a difference of 16%, as we can see in Table 4.1 and Table 4.2. With this exercise, we have seen the importance of using other metrics to evaluate the model, apart from the global accuracy of the model, since it can hide imbalances between classes, as in this case. Figure 4.3 Evolution of accuracy and loss during the training process without data augmentation Table 4.1 Confusion Matrix without using data augmentation BPD CRL NT BPD 0.82 0.09 0.09 CRL 0.04 0.89 0.07 NT 0 0.02 0.98 Table 4.2 TPR and TNR without using data augmentation Sensitivity (TPR) Specificity (TNR) BPD 0.82 0.98 CRL 0.89 0.95 NT 0.98 0.92
55 As we have seen in the theoretical section, to reduce the overfitting we can simplify the model or increase the data. On the one hand, we can not simplify the model since we have used transfer learning taking advantage of all convolutional layers of the pre-trained CNN Inception V3, which has been carefully designed. Therefore, in case of eliminating some convolutional layers, we should add other layers such as pooling layers to reduce the size of the feature maps, and that may be counterproductive if you do not have a deep knowledge of the network. On the other hand, the number of samples that we have is fixed, so we can generate synthetic images from them. By adding the synthetic data to the training dataset and looking again for the optimal parameters, the training and test curves have been equalized (Figure 4.4), reducing the overfitting, and even though the global accuracy of 91.7% is not much better than before, the accuracy of each class has been matched, eliminating the previously existing imbalance, as shown in Table 4.3 and Table 4.4. Figure 4.4 Evolution of accuracy and loss during the training process with data augmentation Table 4.3 Confusion Matrix with data augmentation BPD CRL NT BPD 0.90 0.05 0.05 CRL 0.03 0.93 0.04 NT 0.02 0.06 0.92 Table 4.4 TPR and TNR with data augmentation Sensitivity (TPR) Specificity (TNR) BPD 0.90 0.98 CRL 0.93 0.95 NT 0.92 0.96
62 Figure 4.11 Results after applying the CAM method on CRL images 4.3.4.3 Results for NT images For the NT images, we have again a bit of randomness in the areas that the method tells us are important to the model, but in this case, internal areas of the head and the nuchal area take more importance than in occlusion method. In addition, the face and neck are also highlighted in many samples. In this case, it does not detect the amniotic fluid or the walls of the uterus as important, that is, the model focuses only on the head and the areas near it to classify these images as NT, which is a good result. Figure 4.12 Results after applying the CAM method on NT images
63 4.3.5 LIME In this section we are going to show the results obtained after using the LIME method, which is the last of the interpretability algorithms used in this project. As mentioned in the Methodology section, the results will be presented using the 5 most important features or superpixels of each sample, since we believe that it is an appropriate number to see the relevant parts without contaminating the image. Finally, as you can see in the images below, a mask has been used to indicate the important superpixels, which are painted yellow, and as in the previous cases, you can find more results in A.8. 4.3.5.1 Results for BPD images The results for the BPD class are shown in Figure 4.13, and we can see that in all the samples there is more than one relevant superpixel in the area of the baby’s head. It is important to note that the first highlighted features are generally those that contain part of the skull, which agrees with the results obtained with the Occlusion method and CAM. Figure 4.13 Results after applying LIME on BPD images
64 4.3.5.2 Results for CRL images If we look now at the CRL class (Figure 4.14), the results are also very clear, since the important areas are concentrated on the baby’s body, be it the chest or the lower part that includes the rump. Sometimes the part of the wall of the uterus that is in contact with the baby’s back also highlighted. In general, these results are also consistent with those obtained with the Occlusion and CAM methods. Figure 4.14 Results after applying LIME on CRL images 4.3.5.3 Results for NT images Finally, if we analyse the results for the NT class (Figure 4.15), we see that the superpixels that are in contact with the baby’s face are always highlighted, that is those that contain the nose or chin. However, as in the other methods, there is a bit more variety of important areas in this class, since in many cases the chest and neck also appear as relevant.
65 Figure 4.15 Results after applying LIME on NT images 4.4 Interpretation of Misclassifications Apart from seeing what the model is focusing on to correctly classify the images, it is also important to study what is happening with those samples that are being classified badly. This is what we are going to try to do now. For that, we have decided to use the CAM technique because is the one that has given us the best and clearest results in this area. The way chosen to show these results has been through the representation of the confusion matrix, whose rows represent the correct class of the samples, whereas the columns are the prediction that the model makes of each of those samples. Note that, the cells that represent the diagonal of the matrix have already been explained, since they are those that have correctly classified. Therefore, now we are going to fucus on the rest of the cells, represented as a red cross in Figure 4.16. In the first row, we can see BPD images classified as CRL and NT have in common that the head circumference is not as well defined by the skull as in well classified samples, and the texture is very similar to the walls of the uterus, being null the visibility of possible internal structures or soft tissues. In addition, in most images that are classified as CRL, the head is far away, that is, the zoom is small so the model may be confusing with the body or the baby’s head of the CRL plane. On the other hand, those images classified as NT do have a high zoom of the head and the skull is visible in some samples, however, its contrast with the rest of the tissues is small, which could mean that the texture of the tissues is gaining importance instead of the curved shapes with high contrast that characterize the BPD class.
66 Figure 4.16 Misclassifications represented using the confusion matrix If we now analyse the misclassified CRL images (second row), we see that in the cases that are classified as BPD the model is giving great importance to the rounded shapes that form the interior areas of the walls of the uterus, perhaps because the baby does not present many internal structures due to the high pixel intensity, or it is not well defined. So, this can increase the importance of the rounded curves for the model. Looking at the CRL images classified as NT, we see that the model does not focus so much on the baby’s body and that the baby’s legs rarely appear, which is the important point in the well classified images, but rather focuses more on the face and head, which are important features of NT class. This
67 could be interpreted as that the body is not too distinguishable from the head, or that there is too much zoom, showing more details of the head than necessary for this class. Finally, we have those NT images that have been classified incorrectly (third row). First, in those that are classified as BPD, the previous pattern is repeated, and that is that the model focuses on the rounded shapes of the baby’s face and the walls of the uterus. This may be due to the fact that there is a very high contrast between these structures and the background, as happens in most BPD images, or that the head does not present internal structures or many details, giving more importance to the rounded areas of the face and the walls of the uterus with grater intensity. Secondly, in NT images classified as CRL, the baby’s body plays an important role, which is the main feature detected in CRL class. So, these images perhaps there is too great presence of the baby’s body to be classified as NT, and that could be the reason why the model is confused.
68 5 Discussion After seeing the results obtained with each method, it is time to analyse them in more depth and try to find links between them. First, we have seen that the activation maximization method has given us results that are a bit difficult to interpret, especially in the CRL and NT classes, since the patterns or shapes that maximize them were not easily recognizable. Therefore, this is a method that is not useful for our objective of identifying structures or key points that the model uses to differentiate the different classes. The same happens with Saliency Maps, which have shown us that the relevant parts for the model are those related with the area where the baby is, but the presence of noise does not allow us to identify with greater accuracy which are the features or anatomical regions of each class that the model uses most frequently to differentiate them. However, a small link can be seen between the results of these two methods, since the saliency maps of the BPD class show the area corresponding to the baby’s head as important, which is characterized by having a round shape. On the other hand, the saliency maps of the CRL class focus the sensible areas on the central region of the image, which is where the AM method presents more variations in texture and shapes for this class. Finally, the saliency map of the NT class is the one that shows most noise and dispersion, which agrees with the interpretation that we have made of the AM output for that class, since importance is given to a larger region of the image. If we now talk about the remaining methods, Occlusion, CAM and LIME have provided very satisfactory results, since with all three we have been able to identify with a very high coincidence, but different precision, specific regions that the model uses to differentiate the classes, and surprisingly in many they coincide with important anatomical areas of the baby. Despite this coincidence between methods, the Occlusion technique is the one that we believe has given us the best results in explaining the predictions of well classified samples, since it is the one that has the highest precision identifying specific areas. On the other hand, and as already commented, CAM provides very similar results but it tends to identify more general regions, losing a bit of detail that we obtained with the Occlusion technique. Nevertheless, this is the most suitable technique to interpret misclassified samples, as it offers clear results without much noise. Finally, with LIME we have corroborated the results of the other two previous methods, since the coincidence has also been surprising. This is a significantly important because this is a completely different method than the others, which builds a simple model on top of a complex one to perform the interpretation. For that it uses segmentation to divide the images into features, which is a very delicate process in ultrasound images and that in many cases it has not been produced correctly, overlapping
69 quite different areas of the image and not identifying specific anatomical structures. In addition, this method is considered as a black box in itself, but understanding its operation and making a correct analysis for the choice of parameters, we have achieved results that fully coincide with CAM and Occlusion since the superpixels that are classified as important are those that contain structures or areas that were relevant for the other two methods, which is very positive for us because it offers robustness to our analysis. Analyzing the results in detail for well classified images, we have seen that the most important characteristics to classify images as BPD are the pronounced curves with high contrast respect to nearby areas, which in many cases coincides with the baby’s skull circumference. This result is very relevant because it is telling us that to get a good image of the BPD plane the head area has to be well defined by the ossification of the skull, and for that it has to present dark areas inside (soft tissues) and outside (amniotic fluid) to increase the contrast of the skull. Note that if these requirements are fulfilled, the measurement of the biparietal diameter would be very simple because the head limits would be easily visible. The results for CRL images are also surprising since the algorithms have detected the baby’s trunk as the most important feature of this class. This decision may be based on the texture of this area since it is usually the area with the least movement, so it has a very similar texture in all CRL samples. In addition, in many cases the baby’s legs are also highlighted, which means that the algorithms focus on those parts of the baby that appear specifically in this plane. For NT images, we have more mixed results, but in general terms we can distinguish the two most important characteristics, which are the face and the nuchal area, although in some cases the entire head is detected. The fact that it detects the nuchal fold in many samples is very important because it is the main characteristic of this class. Now that we have seen the main features of each class, we can make the comparison with the first two methods to see that the results are not so different qualitatively, since for the BPD class the rounded forms are still important, for the CRL class the central part of the image is the relevant, and for the NT class there is a mixture between different parts of the sample, which is very similar to the conclusions obtained in the first methods. To interpret the misclassified samples, we have used the CAM method since as mentioned, it is the one that offer the best results for this propose. However, in Figure 5.1 it can be seen that the results generally coincide with the other methods. Thus, the algorithm has allowed us to interpret why the model is getting confused when classifying some images, and although this is a more complex task than the previous one, we have managed to satisfactorily explain some errors of the model. Nevertheless, those samples that do not have a clear or convincing explanation are samples that in many cases are not well taken and would not be valid to carry out any type of measurement. Therefore, in this case we are not looking for the ability to correctly interpret all misclassified images, but to analyze those that could be used to perform measurements but that do not fulfill all the requirements, and analyze what characteristics of each class that contribute to this lack of quality are important to the model in order to improve the taking of images. In this sense, we believe that the conclusions drawn are satisfactory.
70 Figure 5.1 Comparison of the explanation of misclassified images
71 6 Conclusions and future work The aim of this project was to interpret in human terms the behaviour of a convolutional neural network trained to classify fetal ultrasound images and identify what features or patterns the model uses to recognize each class. For that, a model to classify those images has been built using transfer learning from Inception V3 architecture, and an accuracy of 91.7% has been achieved on the test dataset. After that, a bunch of interpretability techniques have been applied on our model in order to interpret the predictions. Below we remark the main results: - The interpretation of the models is a complicated task, which does not have a miracle technique that works for everything. In our case, Occlusion has given us accurate results to explain the correct predictions, while CAM is better for misclassifications. - Gradient based methods like AM or Saliency Maps offer results that are difficult to understand at first and are not valid for an accurate interpretation, but in the end, we have shown that they are related to the results obtained with the most interpretable methods. - A detailed and robust explanation has been provided for each plane, detecting the most important areas of each one. This interpretation is supported mainly by three different methods, Occlusion, CAM and LIME. - Through interpretability techniques, some of the characteristics that negatively influence a class and cause the model go wrong have been found, which could be very useful to improve the taking of images. - We have demonstrated that the model is focusing on the appropriate characteristics to classify the images, which is very important to increase the trustworthiness. - LIME is an interpretability technique sometimes discussed for its black box behavior and unstable results. We have shown that by carrying out a good analysis of the performance of the algorithm, good results can be obtained, which are a positive support for our interpretations. As future work, it would be interesting to replicate these experiments using another model and compare the results. Also, now that we have seen that with our model it is possible to offer a clear explanation of the predictions, it would be interesting to be able to obtain more and better samples, in order to increase its performance and achieve a good detector for images that do not fulfill the requirements of each plane. This tool would surely be of great help to doctors.
78 # BPD images BPD_or_dir_1 = './IMATGES/Originals/Vicent/BPD' BPD_or_dir_2 = './IMATGES/Originals/Safae/BPD/Totes' # CLR images CLR_or_dir_1 = './IMATGES/Originals/Vicent/CLR' CLR_or_dir_2 = './IMATGES/Originals/Safae/CLR/Totes' # NT images NT_or_dir_1 = './IMATGES/Originals/Vicent/NT' NT_or_dir_2 = './IMATGES/Originals/Safae/NT/Totes' BPD = load_images_from_folder(BPD_or_dir_1) + load_images_from_folder(BPD_or_dir_2) CLR = load_images_from_folder(CLR_or_dir_1) + load_images_from_folder(CLR_or_dir_2) NT = load_images_from_folder(NT_or_dir_1) + load_images_from_folder(NT_or_dir_2) # 1BLUE PIXELS ################################################################################ channel1Min_blue = 0.000; channel1Max_blue = 143.000 channel2Min_blue = 0.000; channel2Max_blue = 255.000 channel3Min_blue = 162.000; channel3Max_blue = 255.000 ################################################################################ # 2YELLOW PIXELS ################################################################################ channel1Min_ye = 60.000; channel1Max_ye = 255.000 channel2Min_ye = 190.000; channel2Max_ye = 255.000 channel3Min_ye = 0.000; channel3Max_ye = 189.000 ################################################################################ # 3WHITE PIXELS ################################################################################ channel1Min_wh = 161.000; channel1Max_wh = 255.000; channel2Min_wh = 255.000; channel2Max_wh = 255.000; channel3Min_wh = 0.000; channel3Max_wh = 255.000; ################################################################################ # 4SOME GREEN ONES ################################################################################ channel1Min_green = 0.000; channel1Max_green = 65.000; channel2Min_green = 111.000; channel2Max_green = 238.000; channel3Min_green = 0.000; channel3Max_green = 86.000; ################################################################################ # PROCESSING OF BPD IMAGES for i in range(len(BPD)):
79 # IMAGE I = BPD[i] # GRAY SCALE gray = cv2.cvtColor(I, cv2.COLOR_RGB2GRAY) # AVERAGE FILTER avgImg = cv2.boxFilter(gray, -1, (25,25), borderType=cv2.BORDER_CONSTANT) # SEARCHING THE PIXELS WHICH FULFILLS THE CONDITIONS ABOVE FOR EACH COLOR t_blue = np.where(np.logical_and(I[:,:,0] >= channel1Min_blue, I[:,:,0] <= channel1Max_blue) * np.logical_and(I[:,:,1] >= channel2Min_blue, I[:,:,1] <= channel2Max_blue) * np.logical_and(I[:,:,2] >= channel3Min_blue, I[:,:,2] <= channel3Max_blue)) t_ye = np.where(np.logical_and(I[:,:,0] >= channel1Min_ye, I[:,:,0] <= channel1Max_ye) * np.logical_and(I[:,:,1] >= channel2Min_ye, I[:,:,1] <= channel2Max_ye) * np.logical_and(I[:,:,2] >= channel3Min_ye, I[:,:,2] <= channel3Max_ye)) t_wh = np.where(np.logical_and(I[:,:,0] >= channel1Min_wh, I[:,:,0] <= channel1Max_wh) * np.logical_and(I[:,:,1] >= channel2Min_wh, I[:,:,1] <= channel2Max_wh) * np.logical_and(I[:,:,2] >= channel3Min_wh, I[:,:,2] <= channel3Max_wh)) t_green = np.where(np.logical_and(I[:,:,0] >= channel1Min_green, I[:,:,0] <= channel1Max_green) * np.logical_and(I[:,:,1] >= channel2Min_green, I[:,:,1] <= channel2Max_green) * np.logical_and(I[:,:,2] >= channel3Min_green, I[:,:,2] <= channel3Max_green)) # WE CHANGE THE PIXELS FOUND ABOVE FOR THE AVERAGED PIXEL AFTER FILTERING gray[t_blue] = avgImg[t_blue] gray[t_ye] = avgImg[t_ye] gray[t_wh] = avgImg[t_wh] gray[t_green] = avgImg[t_green] # INTENSITY CORRECTION v_min, v_max = np.percentile(gray, (1.0, 99.0)) gray = exposure.rescale_intensity(gray, in_range = (v_min, v_max)) # MEDIAN FILTER gray = cv2.medianBlur(gray, 5) # CROPPING THE IMAGE if gray.shape == (650,868) or gray.shape == (720,960): gray = gray[59:720, 15:960] elif gray.shape == (768,1024): gray = gray[80:570, 1:760] # RESIZE AND SAVE gray = cv2.resize(gray, image_size) #gray = cv2.cvtColor(gray, cv2.COLOR_GRAY2RGB) cv2.imwrite(processed_path + '/BPD/BPD_%d.jpg' % i, gray) # PROCESSING OF CLR IMAGES
80 for i in range(len(CLR)): # IMAGE I = CLR[i] # GRAY SCALE gray = cv2.cvtColor(I, cv2.COLOR_RGB2GRAY) # AVERAGE FILTER avgImg = cv2.boxFilter(gray, -1, (25,25), borderType=cv2.BORDER_CONSTANT) # SEARCHING THE PIXELS WHICH FULFILLS THE CONDITIONS ABOVE FOR EACH COLOR t_blue = np.where(np.logical_and(I[:,:,0] >= channel1Min_blue, I[:,:,0] <= channel1Max_blue) * np.logical_and(I[:,:,1] >= channel2Min_blue, I[:,:,1] <= channel2Max_blue) * np.logical_and(I[:,:,2] >= channel3Min_blue, I[:,:,2] <= channel3Max_blue)) t_ye = np.where(np.logical_and(I[:,:,0] >= channel1Min_ye, I[:,:,0] <= channel1Max_ye) * np.logical_and(I[:,:,1] >= channel2Min_ye, I[:,:,1] <= channel2Max_ye) * np.logical_and(I[:,:,2] >= channel3Min_ye, I[:,:,2] <= channel3Max_ye)) t_wh = np.where(np.logical_and(I[:,:,0] >= channel1Min_wh, I[:,:,0] <= channel1Max_wh) * np.logical_and(I[:,:,1] >= channel2Min_wh, I[:,:,1] <= channel2Max_wh) * np.logical_and(I[:,:,2] >= channel3Min_wh, I[:,:,2] <= channel3Max_wh)) t_green = np.where(np.logical_and(I[:,:,0] >= channel1Min_green, I[:,:,0] <= channel1Max_green) * np.logical_and(I[:,:,1] >= channel2Min_green, I[:,:,1] <= channel2Max_green) * np.logical_and(I[:,:,2] >= channel3Min_green, I[:,:,2] <= channel3Max_green)) # WE CHANGE THE PIXELS FOUND ABOVE FOR THE AVERAGED PIXEL AFTER FILTERING gray[t_blue] = avgImg[t_blue] gray[t_ye] = avgImg[t_ye] gray[t_wh] = avgImg[t_wh] gray[t_green] = avgImg[t_green] # INTENSITY CORRECTION v_min, v_max = np.percentile(gray, (1.0, 99.0)) gray = exposure.rescale_intensity(gray, in_range = (v_min, v_max)) # MEDIAN FILTER gray = cv2.medianBlur(gray, 5) # CROPPING THE IMAGE if gray.shape == (650,868) or gray.shape == (720,960): gray = gray[59:720, 15:960] elif gray.shape == (768,1024): gray = gray[80:570, 1:760] # RESIZE AND SAVE gray = cv2.resize(gray, image_size) #gray = cv2.cvtColor(gray, cv2.COLOR_GRAY2RGB) cv2.imwrite(processed_path + '/CLR/CLR_%d.jpg' % i, gray) # PROCESSING OF NT IMAGES
81 for i in range(len(NT)): # IMAGE I = NT[i] # GRAY SCALE gray = cv2.cvtColor(I, cv2.COLOR_RGB2GRAY) # AVERAGE FILTER avgImg = cv2.boxFilter(gray, -1, (25,25), borderType=cv2.BORDER_CONSTANT) # SEARCHING THE PIXELS WHICH FULFILLS THE CONDITIONS ABOVE FOR EACH COLOR t_blue = np.where(np.logical_and(I[:,:,0] >= channel1Min_blue, I[:,:,0] <= channel1Max_blue) * np.logical_and(I[:,:,1] >= channel2Min_blue, I[:,:,1] <= channel2Max_blue) * np.logical_and(I[:,:,2] >= channel3Min_blue, I[:,:,2] <= channel3Max_blue)) t_ye = np.where(np.logical_and(I[:,:,0] >= channel1Min_ye, I[:,:,0] <= channel1Max_ye) * np.logical_and(I[:,:,1] >= channel2Min_ye, I[:,:,1] <= channel2Max_ye) * np.logical_and(I[:,:,2] >= channel3Min_ye, I[:,:,2] <= channel3Max_ye)) t_wh = np.where(np.logical_and(I[:,:,0] >= channel1Min_wh, I[:,:,0] <= channel1Max_wh) * np.logical_and(I[:,:,1] >= channel2Min_wh, I[:,:,1] <= channel2Max_wh) * np.logical_and(I[:,:,2] >= channel3Min_wh, I[:,:,2] <= channel3Max_wh)) t_green = np.where(np.logical_and(I[:,:,0] >= channel1Min_green, I[:,:,0] <= channel1Max_green) * np.logical_and(I[:,:,1] >= channel2Min_green, I[:,:,1] <= channel2Max_green) * np.logical_and(I[:,:,2] >= channel3Min_green, I[:,:,2] <= channel3Max_green)) # WE CHANGE THE PIXELS FOUND ABOVE FOR THE AVERAGED PIXEL AFTER FILTERING gray[t_blue] = avgImg[t_blue] gray[t_ye] = avgImg[t_ye] gray[t_wh] = avgImg[t_wh] gray[t_green] = avgImg[t_green] # INTENSITY CORRECTION v_min, v_max = np.percentile(gray, (1.0, 99.0)) gray = exposure.rescale_intensity(gray, in_range = (v_min, v_max)) # MEDIAN FILTER gray = cv2.medianBlur(gray, 5) # CROPPING THE IMAGE if gray.shape == (650,868) or gray.shape == (720,960): gray = gray[59:720, 15:960] elif gray.shape == (768,1024): gray = gray[80:570, 1:760] # RESIZE AND SAVE gray = cv2.resize(gray, image_size) #gray = cv2.cvtColor(gray, cv2.COLOR_GRAY2RGB) cv2.imwrite(processed_path + '/NT/NT_%d.jpg' % i, gray)
82 A.2. Data Augmentation Code Function used to generate the synthetic images: #!/usr/bin/env python # coding: utf-8 import numpy as np import cv2 import os import matplotlib.pyplot as plt from keras.preprocessing.image import ImageDataGenerator from sklearn.utils import shuffle ################################################################################### # SYNTHETIC IMAGES ################################################################################### def generate_images(images, N): images2 = [] datagen = ImageDataGenerator( rotation_range=15, width_shift_range=0.1, height_shift_range=0.1, shear_range=0.2, horizontal_flip=True) datagen.fit(images) i=0 for x_batch in datagen.flow(images, batch_size=1): if i < N: images2.append(x_batch[0]) i += 1 else: break images_syn = np.ndarray((len(images2), images2[0].shape[0], images2[0].shape[1], images2[0].shape[2]), dtype = 'float32') for i in range(len(images2)): images_syn[i] = images2[i] return images_syn
83 A.3. Transfer Learninng with VGG19 from keras.applications.vgg19 import VGG19 model = VGG19(weights='imagenet') model = Model(inputs=model.input, outputs=model.get_layer('fc2').output) model.save('./MODELS/VGG19/extractor.h5') feature_extractor = load_model('./MODELS/VGG19/extractor.h5') x_train, y_train = load_train_images() x_test, y_test = load_test_images() x_train = feature_extractor.predict(x_train, verbose=1) x_test = feature_extractor.predict(x_test, verbose=1) def define_classifier(): sgd = optimizers.SGD(lr=0.001, momentum=0.5, nesterov=False) classifier = Sequential() classifier.add(Dropout(0.3)) classifier.add(Dense(3, name='classificador', activation='softmax')) classifier.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy']) return classifier classifier = define_classifier() history=classifier.fit(x_train, np_utils.to_categorical(y_train), epochs=60, batch_size=80, validation_data=(x_test, np_utils.to_categorical(y_test)), verbose=2) x = feature_extractor.output x = classifier(x) model = Model(inputs=feature_extractor.input, outputs=x) model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy']) Figure A1 Evolution of accuracy and loss during the training process
84 Table A1 Confusion matrix of the trained model BPD CRL NT BPD 0.90 0.05 0.05 CRL 0.01 0.90 0.09 NT 0.03 0.06 0.91 Table A2 TPR and TNR of the trained model Sensitivity (TPR) Specificity (TNR) BPD 0.90 0.98 CRL 0.90 0.95 NT 0.91 0.93 A.4. Activation Maximization Algorithm: import numpy as np import cv2 import matplotlib.pyplot as plt import keras from keras.models import load_model from keras.applications.inception_v3 import preprocess_input import tensorflow as tf model = load_model('./MODELS/InceptionV3/linearOutput.h5') def compute_loss(input_image, filter_index): activation = model(input_image) filter_activation = activation[..., filter_index] return filter_activation @tf.function def gradient_ascent_step(img, filter_index, learning_rate): with tf.GradientTape() as tape: tape.watch(img) loss = compute_loss(img, filter_index) # Compute gradients. grads = tape.gradient(loss, img) # Normalize gradients. img += learning_rate * grads img = tf.math.l2_normalize(img) return loss, img loss_vector = []
85 def initialize_image(): # We start from a gray image with some random noise img = tf.random.uniform((1, 299, 299, 3)) # Inception expects inputs in the range [-1, +1]. # Here we scale our random inputs to [-0.125, +0.125] return (img - 0.5) * 0.25 def visualize_filter(filter_index): # We run gradient ascent for 20 steps iterations = 60 learning_rate = 0.005 img = initialize_image() for iteration in range(iterations): loss, img = gradient_ascent_step(img, filter_index, learning_rate) loss_vector.append(loss.numpy()) if iteration%10 == 0: # print(str(iteration) + " loss: " + str(loss)) # Decode the resulting input image img = deprocess_image(img[0].numpy()) return loss, img def deprocess_image(img): # Normalize array: center on 0., ensure variance is 0.15 img -= img.mean() img /= img.std() + 1e-5 img *= 0.15 # Clip to [0, 1] img += 0.5 img = np.clip(img, 0, 1) # Convert to RGB array img *= 255 img = np.clip(img, 0, 255).astype("uint8") return img # Call the function: the argument is the class that we want to maximize the input loss, img = visualize_filter(0) Plots of the evolution of the score of each class during the process:
86 Figure A2 Score evolution during the Activation Maximization process A.5. Saliency Maps Algorithm: from tensorflow.keras import backend as K from tf_keras_vis.saliency import Saliency from tf_keras_vis.utils import normalize from keras.models import load_model from keras.applications.inception_v3 import preprocess_input import cv2 import matplotlib.pyplot as plt import tensorflow as tf import numpy as np model = load_model('../MODELS/InceptionV3/linearOutput.h5') image = cv2.imread('../IMATGES/InceptionV3/TEST/CLR/CLR_tst_4.jpg') def loss(output): # indicate the class here return output[:, 2] saliency = Saliency(model, model_modifier=None, clone=False) saliency_map = saliency(loss, preprocess_input(image)) More Saliency Maps:
87 Figure A3 More Saliency Maps of BPD class Figure A4 More Saliency Maps of CRL class
94 Figure A12 More results of BPD class using LIME method Figure A13 More results of CRL class using LIME method
95 Figure A14 More results of NT class using LIME method