ResNet

ResNet Isabel Amaya-Rodriguez, Javier Civit-Masot, Francisco Luna-Perejon, Lourdes Duran-Lopez, Alexander Rakhlin, Sergey Nikolenko, Satoshi Kondo, Pablo Laiz, Jordi Vitrià, Santi Seguí, and Patrick Brandao General Motivation In this chapter, all groups have used Residual Network (ResNet) (He et al. 2016)as part of different architectures with the purpose of solving the GIANA challenge. In some cases like RTC-ATC group ResNet-50 was used as a layer in Faster Convolutional Neural Network (FCNN) in order to build an automated recognition system to detect the presence of polyps in colonoscopy images. The main reason to use this network is because ResNet models try to solve the overload of the accuracy which comes from network depth. The accuracy saturation is not due to overfitting or the quantity of layers is because of the named Vanishing Gradient (Hochreiter 1998) this effect try to explain when the network is deep the loss functions in gradients value are near to zero after several chain rule applications. Then weights are not updated and consequently no learning is being performed. To I. Amaya-Rodriguez (B) Vicomtech, Paseo Mikeletegi 57, Donostia-San Sebastian, Spain e-mail: [email protected] J. Civit-Masot · F. Luna-Perejon · L. Duran-Lopez Robotics and Computer Technology Lab., University of Seville, Seville, Spain A. Rakhlin · S. Nikolenko Steklov Institute of Mathematics at St. Petersburg, Russia nab. r. Fontanki, 27, St. Petersburg 191023, Russia S. Kondo Konica Minolta, Inc., 1-2 Sakura-machi, Takatsuki, Osaka 569-8503, Japan P. Laiz · J. Vitrià · S. Seguí Departament de Matemàtiques i Informàtica de la Universitat de Barcelona, Barcelona, Spain P. Brandao Wellcome/EPSRC Centre for Interventional & Surgical Sciences, University College London, London, UK Fig. 1 Residual learning block solve this problem, Microsoft created a new deep learning concept based on residual leaning which allows gradients to flow between layers. Introduction to ResNet Architecture In this section, the basic concepts of ResNet architecture is explained. As it was introduced in the Sect. 12.1 ResNet architecture makes it possible to implement hundreds or even thousands of layers and still achieves compelling performance. Residual Network works substracting features learned from input of that layer. The main characteristic introduced by ResNet is the identity shortcut connection defined as F(x) := H(x) − x shown in Fig. 1. This shortcut connections X are identity mappings and their outputs are added to the following stacked layers. Then ResNet apply simply stacked identity mappings and the residual of X in H(x) is learned. It solves problems like training error increase when the depth increases too. In the case of ResNet-12 contains five Residual Blocks as shows Fig. 2.For each two-convolutional layer there is one identity shorcut connection. Methodologies RTC-ATC Group In this section, RTC-ATC group shows an application of Faster R-CNNs (FRCNN) in order to build an automated recognition system to detect the presence of polyps in colonoscopy images presented in GIANA challenge 2019. To realize this goal, they used an implementation of FRCNN with ResNet-50 as Fully Convolutional Network (FCN) architecture. FRCNN builds on the idea of Region Proposals by sharing intermediate features with the classification network. For example, the ResNet takes an Fig. 2 ResNet-12 architecture input image and produces a series of transformations before arriving at the prediction. The FRCNN will use the intermediate features of ResNet to aid in region proposal. Brief Methodology Introduction Faster R-CNNs have been used for different purposes: face detection (Jiang and Learned-Miller 2017), driver’s cell-phone usage and hands on steering wheel detection (Hoang Ngan Le et al. 2016) are some application examples of this algorithm, which has proven to show good results. As it was mentioned in the introduction, a FRCNN was used in this work for the polyp detection task. This algorithm is divided into two modules (Fig. 3): 1. First of all, a deep Fully Convolutional Network (FCN) (Ren et al. 2015) receives the images from the dataset. Then, it extracts feature maps or descriptive characteristics and analyzes them to propose regions of interest. The novel step that this architecture introduced is the way to determine the regions of interest. Region Proposal Network (RPN) is computed base on the output feature map of the previous step. Then, RPN is connected to a convolutional layer with 3 × 3 filters, 1 padding, 512 output channels. The output is connected to two 1 × 1 convo- Fig. 3 Block diagram of the implemented approach lutional layer for classification and box-regression (Note that the classification here is to determine if the box is an object or not). 2. Next, as shows Fig. 4 ROI pooling layer is used for these proposed regions in order to ensure the standard and pre-defined output size. These valid outputs are passed to a fully connected layer as inputs. In our case, by using a neural network that takes advantage of the mathematical operations made in the convolutional layers. In our architecture we have used the ResNet-50 model (He et al. 2016) as FCN. ResNet models try to solve the saturation of the accuracy caused by increasing the network depth (Fig. 5). 3. Finally, the proposed regions that are the input of the second module, called Fast R-CNN detector, composed of two fully connected layers, a regression layer and a classification layer (Ren et al. 2015). Fig. 4 ResNet-50 in faster convolutional neural network Fig. 5 Block diagram of the implemented approach Fig. 6 Processing applied to the original images. First, black edges are removed in a preprocessing step. Then data augmentation is applied, generating three different new images Architecture and Parameters Tuning Basic architecture was modified to achieve the better results, aiming to observe the benefit of using different datasets by training the network with background examples and augmented dataset. Firstly, with the aim of reducing the number of false positives, a technique called hard-negative mining was used (Felzenszwalb et al. 2009). It consists of adding negative samples, which means, including examples of images that do not contain polyps in the training step, labeling them as background. The dataset was augmented using a series of transformations so that the model would never train twice the exact same image. For each original preprocessed image, an horizontal flip, a vertical flip, and a blur filter have been applied. Thus, we obtain three new images from each original sample. After this data augmentation step, we obtain a dataset that consists of 47.816 images in total (Fig. 6). Our model contains several parameters to be defined in order to improve the results training with some invariant parameters as learning rate 10−5, 1000 iterations per epoch, 32 number of Regional Object Interest (ROIs) and the increased image dataset with rotations and flips. Tests were performed every 50 epochs, selecting different confidence thresholds in order to obtain the best results. Neuromation In this section, the Neuromation team discusses their model architecture and segmentation uncertainty estimation based on Bayesian approximation. Network Architecture. The model architecture stems from the Hourglass and U-Net design principles (Ronneberger et al. 2015; Liu et al. 2017). The contracting branch of the model is based on the Resnet-34 encoder where we introduce useful modifications: ELU activations instead of ReLU, reversed order of batch normalization and activation layers (Mishkin et al. 2017), and He normal weight initialization (He et al. 2015). One major difference from the classical U-Net architecture is meant to deal with the limited dataset size characteristic for the GIANA challenge and for medical imaging problems in general. We use two approaches to alleviate the problem of overfitting to limited training data: (1) extreme data augmentation and (2) Spatial 2D Dropout (Tompson et al. 2015) incorporated into the upsampling branch. The upsampling branch is implemented as a Feature Pyramid Network (FPN) (Lin et al. 2016), reconstructing high-level semantic feature maps at 4 scales simultaneously. We implement a Feature Pyramid block as a convolutional layer with 64 activation maps followed by upsampling to the original resolution with upsampling rate of 8, 4, 2, or 1 depending on the feature map depth (see Fig. 7). We concatenate upsampled maps into a single layer of 64 × 4 = 256 maps and finalize it with the Spatial 2D Dropout layer. Spatial 2D Dropout acts like a regularizer and prevents co-adaptation of the network weights, but unlike conventional dropout it drops out not individual neurons but entire activation maps. In all experiments, we use dropout rate 0.5, i.e., drop 128 out of 256 activation maps. Finally, the output of the model is a sigmoid layer that assigns to every pixel a continuous probability from 0 to 1 of being a polyp region. Loss functions. It is known that the categorical cross entropy (CCE), while convenient for training, does not directly translate into the metric of interest, Jaccard index (Rakhlin et al. 2018; Iglovikov et al. 2017; Rakhlin et al. 2019). Hence, as the loss function we use L(w) =(1−α)CCE(w) −αJ(w), (1) a weighted sum of CCE and the soft Jaccard loss J=1 P P  p=1ypˆyp yp+ˆyp−ypˆyp,(2) Fig. 7 Neuromation architecture where yp is the binary label for pixel p, yˆp is the predicted probability for p, and P is the number of pixels in the image. Segmentation uncertainty estimation. In the domain of medical imaging, it is particularly important to tell whether a model is confident about its estimate or not. One distinctive feature of our approach is an innovative application of dropout as a Bayesian approximation, as recently proposed by Gal and Ghahramani (2016), http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html. Classical deep learning tools do not capture model uncertainty, returning only a point estimate at the output. Using softmax to get probabilities is actually insufficient to obtain model uncertainty (http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce. html). Bayesian models, on the other hand, offer a framework suitable to reason about model uncertainty, but usually do it with a prohibitive computational cost. Gal et al. show that dropout neural networks are identical—under certain, not too restrictive, assumptions—to variational inference in Gaussian processes. In particular, they demonstrate “that averaging forward passes through the dropout network is equivalent to Monte Carlo integration over a Gaussian process posterior approximation” (http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce.html). Traditionally, dropout is considered as model averaging, and it was originally explained that scaling the weights at test time without dropout gives a reasonable approximation to the “average” model (Srivastava et al. 2014). However, for convolutional networks this approximation is not sufficient and can be improved considerably (Gal and Ghahramani 2016). Fig. 8 Basic network architecture Initial Block Residusl + SE Block #1 Residual + SE Block #2 Residual + SE Block #3 Residual + SE Block #4 Final Block Decoder Block #1 Decoder Block #2 Decoder Block #3 Decoder Block #4 Input Image Segmentation Mask DecoderEncoder Konica Minolta Figure 8 shows the basic structure of our network. We use U-Net (Ronneberger et al. 2015) or Link Net (Chaurasia and Culurciello 2017) type deep neural networks with different encoders from the original U-Net and Link Net. U-Net and Link Net both have an encoder-decoder structure and intermediate feature maps in the encoder are concatenated or summed to intermediate feature maps in the decoder, respectively. Modification of Base Architecture Polyp detection, localization and segmentation tasks Our encoder is based on 101 layer ResNeXt (Xie et al. 2017) with Squeeze-and-Excitation blocks (Hu et al. 2018). The decoder is almost same as the original U-Net and Link Net networks except the number of feature maps. We use Link Net type for the polyp detection task, and use U-Net type network for the polyp localization and segmetation tasks. WCE detection and localization tasks Figure 9 shows the whole structure of our network for the WCE detection and localization tasks. We use Link Net type for the WCE detection and localization tasks. Our encoder is based on 101 layer ResNeXt(Xieetal.2017) with Squeeze-and-Excitation blocks (Hu et al. 2018). The decoder is almost same as the original U-Net and Link Net networks except the number of feature maps. We add two fully connected layers on top of the last residual block of the encoder (“Residual + SE Block #4” in Fig. 9) and obtain the Ronneberger, O., Fischer, P., & Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical Image Computing and Computer- Assisted Intervention (pp. 234–241). Springer. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15, 1929–1958. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., & Bregler, C. (2015). Efficient object localization using convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 648–656). “What My Deep Model Doesn’t Know....” http://mlg.eng.cam.ac.uk/yarin/blog_3d801aa532c1ce. html. Xie, S., Girshick, R., Dollár, P., Tu, Z., & He, K. (2017). Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1492–1500). Zhang, H., Cisse, M., Dauphin, Y. N., & Lopez-Paz, D. (2017). mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.