A deep learning fusion model for accurate classification of brain tumours in Magnetic Resonance images

Received: 2 June 2023 - Revised: 30 July 2023 - Accepted: 24 August 2023 - CAAI Transactions on Intelligence Technology DOI: 10.1049/cit2.12276 ORIGINAL RESEARCH A deep learning fusion model for accurate classification of brain tumours in Magnetic Resonance images Nechirvan Asaad Zebari 1 |Chira Nadheef Mohammed 2 |Dilovan Asaad Zebari 3 | Mazin Abed Mohammed 4,5,6 |Diyar Qader Zeebaree 7 | Haydar Abdulameer Marhoon 8,9 |Karrar Hameed Abdulkareem 10 | Seifedine Kadry 11 |Wattana Viriyasitavat 12 |Jan Nedoma 5 |Radek Martinek 6 1 Department of Information Technology, Lebanese French University, Erbil, Iraq 2 Department of Computer Science, University of Zakho, Zakho, Kurdistan Region, Iraq 3 Department of Computer Science, College of Science, Nawroz University, Duhok, Kurdistan Region, Iraq 4 Department of Artificial Intelligence, College of Computer Science and Information Technology, University of Anbar, Ramadi, Iraq 5 Department of Telecommunications, VSB‐Technical University of Ostrava, Ostrava, Czech Republic 6 Department of Cybernetics and Biomedical Engineering, VSB‐Technical University of Ostrava, Ostrava, Czech Republic 7 Department of Computer Network and Information Security, Technical College of Informatics – Akre, Duhok Polytechnic University, Duhok, Iraq 8 Information and Communication Technology Research Group, Scientific Research Center, Al‐Ayen University, Thi‐Qar, Iraq 9 College of Computer Sciences and Information Technology, University of Kerbala, Karbala, Iraq 10 College of Agriculture, Al‐Muthanna University, Samawah, Iraq 11 Department of Applied Data Science, Noroff University College, Kristiansand, Norway 12 Faculty of Commerce and Accountancy, Chulalongkorn Business School, Chulalongkorn University, Bangkok, Thailand Correspondence Mazin Abed Mohammed. Email: [email protected] Funding information Ministry of Education, Youth and Sports of the Chezk Republic, Grant/Award Numbers: SP2023/ 039, SP2023/042; the European Union under the REFRESH, Grant/Award Number: CZ.10.03.01/ 00/22_003/0000048 [Correction added on 23 Jan 2024, after first online publication. The affiliation and affiliation designator is updated]. Abstract Detecting brain tumours is complex due to the natural variation in their location, shape, and intensity in images. While having accurate detection and segmentation of brain tumours would be beneficial, current methods still need to solve this problem despite the numerous available approaches. Precise analysis of Magnetic Resonance Imaging (MRI) is crucial for detecting, segmenting, and classifying brain tumours in medical diagnostics. Magnetic Resonance Imaging is a vital component in medical diagnosis, and it requires precise, efficient, careful, efficient, and reliable image analysis techniques. The authors developed a Deep Learning (DL) fusion model to classify brain tumours reliably. Deep Learning models require large amounts of training data to achieve good results, so the researchers utilised data augmentation techniques to increase the dataset size for training models. VGG16, ResNet50, and convolutional deep belief networks networks extracted deep features from MRI images. Softmax was used as the classifier, and the training set was supplemented with intentionally created MRI images of brain tumours in addition to the genuine ones. The features of two DL models were combined in the proposed model to generate a fusion model, which significantly increased classification accuracy. An openly accessible dataset from the internet was used to test the model's performance, and the experimental results showed that the proposed fusion model achieved a classification accuracy of 98.98%. Finally, the results were compared with existing methods, and the proposed model outperformed them significantly. This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. © 2024 The Authors. CAAI Transactions on Intelligence Technology published by John Wiley & Sons Ltd on behalf of The Institution of Engineering and Technology and Chongqing University of Technology. 790 - CAAI Trans. Intell. Technol. 2024;9:790–804. wileyonlinelibrary.com/journal/cit2 KEYWORDS brain tumour, deep learning, feature fusion model, MRI images, multi‐classification 1 | INTRODUCTION The brain serves as the control and communication hub of the body. It does this by utilising a vast network of connections and neurons to control all of the essential processes that occur within the body. A brain tumour is a condition that could cause life‐threatening consequences. The brain's malfunction is caused by the multiplication of abnormal cells within the brain, which also affects the neurological system and the central spine. It expands slowly, has clearly delineated borders, and rarely moves beyond them [1]. Because there are billions of activated cells in the brain, analysis can be pretty challenging. Brain tumours are one of the leading causes of mortality among people. Around 250,000 people worldwide are diagnosed with primary brain tumours every year; however, these tumours account for fewer than 2% of all cancers [2]. Consistency is present in benign tumours, which do not include any cancer cells actively spreading; in contrast, malignant tumours have structural heterogeneity and active cancer cells. Malignant tumours also contain active cancer cells. Even though benign tumours grow slowly, there is always a chance that they will progress into dangerous malignant tumours. Low‐grade tumours are another term that may refer to benign tumours. Low‐grade tumours, referred to as benign tumours, can be further divided into two distinct types: gliomas and meningiomas. Glioblastoma and astrocytoma are the two types of high‐ grade brain tumours that fall into the same category as malignant tumours, also known as high‐grade brain tumours. A benign tumour is not hazardous since it is not prevalent in other portions of the brain. However, malignant tumours are incredibly harmful because of their quick growth. They can be removed, and once they are gone, they rarely come back. Cancerous tumours can be divided into several unique subtypes according to various criteria, including the growth site, the degree of its malignancy, and the kind of tissue from which it originated [3]. Many treatments are available for brain tumours, and the treatment chosen will rely on the position, size, and kind of tumour. Due to the lack of unintended side effects associated with surgery, it is now the most common and preferred remedy for brain tumours compared to other treatments [4]. The Magnetic Resonance Imaging (MRI) technique is the most effective and widespread in brain tumour diagnosis. The MRI technique provides information on the structure of human soft tissue, which can be enlarged to offer precise views in every direction. In the field of medical imaging, MRI is applied to display distinctions in various soft tissues [5]. The MRI image can be interpreted to indicate the presence of brain tumours. Magnetic Resonance Imaging images can have various characteristics, depending on the internal anatomical structures being examined. The MRI technique is well‐known for its superior picture clarity and precise tumour appearance [6]. Diagnosing a brain tumour requires a significant investment of time and depends significantly on the experience and expertise of the radiologist. Increasing the patient number will cause a significant increase in the amount of data that has to be processed. This has caused conventional methods to be inefficient as well as expensive. Problems arise due to significant variations in tumour size, form, and severity and the presentation of other diseases. These variances can occur even within the same patient. Conventional manual methods of identifying brain tumours and monitoring their progression over time are laborious and error‐prone [7]. As a result, automated solutions are necessary to replace conventional human procedures. Recent evolutions in artificial intelligence and machine learning (ML) have made it possible to diagnose brain malignancies earlier. There has been a recent uptick in interest in the evolution of automated systems for the analysis of images to overcome the restrictions that come with manually diagnosing patients. Recent years have seen the development of several computer‐aided diagnosis systems intended to diagnose brain tumours automatically. Many academics have researched how to classify brain tumours and implemented various methods to evaluate MRI scans to extract potential characteristics from the dataset using ML and Deep Learning (DL) techniques. The fundamental objective of this research and analysis is to improve processes for the early detection of brain tumours. When detecting brain cancers, predicting the course of the tumour, and treating brain tumours, the automatic classification of medical images is significant [8]. When a brain tumour is found at an early stage, it is more likely to have a quick reaction to treatment, which assists in increasing the survival rate of patients. It requires significant time and labour to search for and categorise images of brain tumours manually contained in sizeable medical image archives [9]. Misdiagnosing a brain tumour can have significant repercussions and lower the individual's likelihood of survival. In light of the gravity of the situation, there is an urgent need for a technique that relies entirely on automated processes to identify brain tumours. The manual procedure of assessing several photographs produced in a clinic is a labour‐intensive, time‐ consuming approach that must be revised to comprehend how various cancers behave fully. It is necessary to develop technology for detecting and identifying tumours that are more accurate and computer‐based to understand and solve this complicated issue. Nowadays, various attempts have been undertaken, all to digitise this process to investigate the many different machine‐learning approaches that are now available. There has been a renaissance in interest in the utilisation of deep networks for the identification of cancer cells in a manner that is both more accurate and dependable [10]. Numerous methods have been attempted to develop an accurate and reliable computerised diagnosis of brain tumours. Yet, it remains challenging due to significant inter‐and intra‐shape variances, texture differences, and contrast differences. Traditional ML methods rely on manually produced features, ZEBARI ET AL. - 791 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License limiting the solution's ability to withstand challenges, while DL‐based methods can automatically extract useful information for improved performance. The novelty of this work lies in the innovative combination of convolutional deep belief networks (CDBN) and ResNet50 models as part of a fusion model for brain tumour classification. The fusion process intelligently combines local and global features captured by CDBN and ResNet50, respectively, resulting in a more comprehensive and discriminative representation of brain tumours. This study developed a model‐based DL method to classify brain cancers into gliomas, meningiomas, pituitary tumours, or no‐tumour classes. Our contributions include the development of a DL‐based process that automatically extracts valuable knowledge for improved performance in classifying brain cancers, addressing the challenges posed by traditional ML methods that rely on manually produced features. The key contributions of this study are: �The study developed a fully automatic approach for multi‐ classifying brain tumours using deep convolutional neural networks (CNN) for feature extraction. �Deep CNN networks were utilised to extract the most relevant information features from the image and for better generalisation. A model was built to incorporate deep features and develop the fusion model to achieve better performance in the multi‐classification accuracy of brain tumours. �Development of a DL fusion model to reliably classify brain tumours, significantly increasing classification accuracy performance. �Utilisation of data augmentation techniques to increase the dataset size for training models, enabling the DL model to achieve good results. �Comparison of the proposed model with existing methods demonstrated that the proposed model outperformed them significantly, indicating the effectiveness of the proposed DL fusion model for brain tumour classification. The remaining sections of the paper are laid out as follows. A discussion of the most recent related works is presented in Section 2. The proposed mechanism includes all its phases in‐ depth, as discussed in Section 3. The experimental results of the proposed model and a comparison with the most recent existing works in the literature are given in Section 4. A discussion related to the proposed method and provided results have been presented in Section 5. The last Section of this paper discusses this research's conclusion and feature work. 2 | RELATED WORK Over the past decade, ML and DL methods have significantly contributed to medical image analysis, making medical personnel's work more effortless and efficient. When addressing issues related to brain tumours, most research publications have focused on DL methods such as CNNs, transfer learning, and classical neural networks. These approaches have proven effective in providing solutions for brain tumour classification. A model‐based CNN was presented by Khwaldeh et al. [11] to distinguish between tumour and non‐tumour brain MRI images in addition to high and low grades of glioma tumours. They achieved an accuracy rate of 91% by modifying the AlexNet network and utilising it as the foundation of their network architecture. Despite the critical work being done in this field, more effort is still required to establish a robust and practical strategy for classifying MRI images of the brain. Das et al. [12] successfully identified many forms of brain malignancies using a CNN containing 3064 MRI images. These tumours included glioma tumours, meningioma tumours, and pituitary tumours. In training, the CNN network was trained to use several convolutional and pooling processes. They could do this by scaling the convolutional network to correspond with convolutional filters and kernels of varying sizes. As a result, they achieved an accuracy rate of 94.39%. In an integrated approach, Hashemzehi et al. [13] examined the ability of a CNN and neural autoregressive distribution estimation model to detect brain tumours from MRI data. They employed 3064 images, which they assessed to determine with a 96% level of accuracy three distinct types of brain cancer. A multi‐pathway CNN design was presented by Francisco et al. [14] to segment several kinds of tumours automatically. They tested their suggested model by applying it to a T1‐ weighted, contrast‐enhanced MRI dataset made accessible to the public and found that it had an accuracy of 97.3%. In contrast, the process of model training was quite expensive. A deep inception residual network was proposed in reference [15], which discussed a three‐class classification system for brain tumours. The layer that is considered to be the network's output in ResNet V2 has been modified. The proposed approach achieves the highest possible accuracy for classifying brain tumours. The suggested model was validated by applying it to a brain imaging dataset that can be accessed openly online and contains 3064 images. The performance result of the presented method is 99.69% better than the most advanced techniques in the literature. Abd El Kader et al. [16] created a deep‐CNN model to identify brain tumours as normal and abnormal for MRI images. Employing a differential operator in the trained network allows for the generation of an additional differential feature map, which can be obtained from the CNN feature map that was initially used. The consequence of using the approach for derivation is an improvement in the effectiveness of the method being provided, which is determined by the outcome of the evaluative parameter. To classify brain tumours, ImageNet‐based Vision Transformers (ViT) that have been learnt and optimised were suggested by Tummala et al. [17]. Performance validation, as well as testing, was conducted using three different classes of tumour dataset that was obtained from Figshare. This dataset includes 3064 T1w contrast‐enhanced MRI images that contained three types of tumours. At a size of 384 �384 pixels, the L/32 model performed the best, earning a score of 98.2% on overall accuracy. The efficiency of the ensembles of all four ViT methods at the exact resolution exceeded the efficiency of 792 - ZEBARI ET AL. 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License each technique and their ensemble at resolution 224 �224. The costumes of the four different ViT algorithms achieved a testing accuracy of around 98.7%. To divide 3260 images into four distinct classes, Nayak et al. [18] developed an EfficientNet model that uses normalisation. This was done to classify the images. The newly constructed model is a modification of EfficientNet that includes adding two other layers: dense and drop‐out. Similarly, they utilised data augmentation in conjunction with normalisation. One of the advantages of using a dense CNN model is that it can effectively classify a constrained image database. As a consequence of this, the technique that was suggested offers outstanding overall performance. According to the findings of the experiments, the proposed model had an accuracy of 99.97% when it was being trained, and it had an accuracy of 98.78% when it was being tested. The region‐based convolutional neural network (RCNN) method was utilised to create a new network for brain tumour identification, and this architecture was validated by using two datasets freely available on Kaggle [19]. Researchers provided a technique for diagnosing brain tumours that uses a less complex network to minimise the required time for processing by a conventional RCNN. They first employed a two‐ channel CNN, a low‐complexity model that enhances accuracy by 98.21%. This allowed them to distinguish between healthy and unhealthy tumour MRI images. After that, this model is used as a feature extraction model in an RCNN to detect tumour regions in a glioma class from the dataset classified from an earlier phase. Then, the tumour region is enclosed in boxes. This method has also been applied to treat meningiomas and malignancies originating in the pituitary gland. Their approach, which had an overall confidence level of 98.8%, was able to attain a short execution time compared to other methods considered to be state‐of‐the‐art. There are numerous methods for identifying brain tumours, but several limitations have yet to be determined. The primary objective of this research is to provide an effective diagnosis method for brain tumours by utilising different CNN networks. Due to a lack of readily available knowledge and data, many radiologists need help with this classification. Previous research has shown that DL approaches provide much higher accuracy for brain MRI classification than classical ML techniques. However, a vast amount of data is required to train DL models, which cannot be achieved with traditional ML techniques. Recent research has established DL approaches as a mainstream component of specialist and intelligent systems and medical image analysis. When dealing with brain tumour classification, it is essential to consider the many constraints associated with the discussed methodologies. 3 | PROPOSED MODEL This section describes our proposed approach in general. Then, in the succeeding subsections, we described the specifics of four critical components in detail. The proposed system includes preprocessing (such as image smoothing, cropping, resizing, and normalisation), data augmentation, DL feature extraction, deep feature fusion, and classification. These five fundamental phases make up the proposed approach. Figure 1 presents a diagrammatic representation of the general diagram FIGURE 1 Representation of all steps of the proposed model for brain tumour identification. ZEBARI ET AL. - 793 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License of our model for identifying brain tumours. The performance evaluation of the suggested model has been tested using T1‐ weighted datasets. Standard performance measurements, such as accuracy, recall, precision, and F1‐score, were employed to test the performance of the proposed model. 3.1 | Preprocessing Image preprocessing can also address other significant intrinsic MRI acquisition abnormalities, such as intensity non‐ uniformity and Gaussian noise [20]. This is possible because of the nature of MRI. Intensity non‐uniformity, intensity inhomogeneity, or bias of the magnetic field will show up as a low‐frequency signal on an MRI due to various causes, such as fluctuation of the magnetic field. Several different things can cause this signal. In other volumes, different preprocessing pipelines were given so that the relationship between image preprocessing patterns and noise filtering for MRI could be studied. This investigation aimed to improve the reproducibility and reliability of radionics characteristics. In the MRI‐ based radiomics analysis, the lack of pre‐specific image normalisation methods may impact the features' consistency and dependability. The Gaussian filter (GF) and the skull stripping method are the primary steps during the pre‐processing of the proposed model. The GF method has several advantages, including reduced noise, a more manageable design process, automatic filtering, and rotational symmetry [20]. The image can have Gaussian noise, salt and pepper noise etc. The information contained in our dataset is preserved in a way that is analogous to noise removal applications. To reduce noise from the image, the GF is used. This filter utilises a 2D Gaussian distribution function, and this function can be described as follows: Gði;jÞ ¼ 1 2πσ2e−i2þj2 2σ2ð1Þ Where the standard deviation distribution is represented by σ, the value of a Gaussian kernel is used when designing a convolutional filter to achieve the desired effect. Magnetic Resonance Imaging images need the application of convolutional filters to every single pixel in the image. In MRI images, matrix multiplications are performed on each pixel's brightness and kernel elements. This helps detect brain tumours earlier. As a consequence, the noise present in the MRI is eliminated, and after that, the image is refined with the utilisation of Gaussian filtering. The performance of identifying almost all of the images in MRI datasets of brain tumours could be better due to the presence of unwanted voids and areas in the images. Thus, it is essential to crop the images to remove any unnecessary parts and extract only the relevant data from the image. This work employed the cropping method described in ref. [21], which involves calculating extreme points. It is advised that the MRI images in our dataset be resized to have the same width and height to obtain the best possible results. This is because the widths, heights, and sizes of the MRI images in our dataset vary. Since the size of the images fed into Deep networks is 224 �224 pixels, we reduced the MRI images to 224 �224 pixels for this task. The exception is CDBN, which requires the input images to have a size of 128 �128 pixels. The MRI image from the patient's database needs to provide more clarity. The brain tumours in MRI images contain a certain amount of ambiguity. As a result, brain images require normalisation before any further processing can take place. Usually, images obtained from an MRI are grayscale in appearance. As a result, the images may be readily normalised, which assists in easing image quality and reduces the likelihood of making errors in classification. Nayak et al. [18] used the membership function with the morphological notion to identify brain cancers. The following is an example of the membership function that was utilised for the study: r¼d−mn mx −mn ð2Þ While ris considered a normalised image, it equals a double image and is equal to min (min(image)) and max(max (image)), respectively. With a range from 0 to 1, this membership function's primary purpose is to standardise the image in preparation for enhancement. 3.2 | Deep feature extraction The deep features were extracted from the MRI scans of brain tumours using three different deep‐learning networks: VGG16, ResNet50, and CDBN. After that, the features are integrated to produce data representations that are more accurate. Because feature descriptors based on DL have the potential to learn fundamental features automatically contained within an MRI image, the utilisation of these descriptors can considerably lessen the necessity for handcrafted feature extraction. This work aims to devise an approach for distinguishing between MRI images of various types of brain tumours. At the end of the process, these representations of the features are fed to the classifier layer, which then categorises them into one of several categories [22, 23]. 3.2.1 | ResNet50 One of the most sophisticated deep networks currently available for image classification is called ResNet50. ResNet50 can handle the difficulty of improving performance accuracy while concurrently decreasing the parameters whenever the model cannot be trained. Thus, a feature extractor‐based deep convolutional that we utilise is ResNet50 [24]. ResNet50 increases the network's depth, and the classification performance based on residual learning is not impacted. The remaining modules each have two possible outputs: one produces the residual of the input feature by performing two or three convolutions on 794 - ZEBARI ET AL. 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License that feature, and the other makes the output directly from the supplied feature. These two paths' combined results are considered the residual module's output. Figure 2reveals that the extraction of features was accomplished with the help of ResNet50. When the network is initialised, the weights previously pre‐trained on MRI brain tumour datasets are employed. The learned weights of deeper layers, such as ResNet50's entirely linked layer, tend to be more class‐specific. We were curious about how well the output vectors of the former convolutional layers might be categorised, and then we ran some tests on them. When utilised properly, networks that have deep convolutional layers are key features. As feature vectors, we used the final residual unit outputs generated by the convolutional layers 3, 4, and 5. The dimensions of the features found in the third layer are less than those found in the fifth layer. Table 1demonstrates that we conducted our analysis using ResNet50. 3.2.2 | CDBN model When utilising a Convolutional DBN network, it is possible to create valid probabilistic conclusions efficiently from the bottom up and the top down [25]. On top of this structure are numerous layers of max‐pooling convolutional restricted boltzmann machines (CRBMs), and training is carried out using the greedy layer‐wise approach, just as it would be in a typical DBN. Using a CDBN model, one can learn high‐level properties such as stroke groups or object sections. During the test phase, we trained the CDBN utilising two layers of CRBM and relied on feed‐forward approximation to make our inferences. On top of the CRBM is where the CDBN was constructed. It is feasible to teach the CDBN method by carrying out a sequence of CRBMs, each feeding into another CRBM in the training process. The structure of the CDBN is depicted in Figure 3; its visible and hidden layers are linked by groups of local and shared characteristics of the CRBM's architectural design. To derive deep features from an image, the DBN applies several layers of RBM on top of each other in a multilayer. Equation (3) provides this information visually by graphically representing the joint potential allocation in the visible layer between the input data v and the l‐layer hidden layer hk. The unsupervised greedy method is used to calculate the weight of the data. Training the RBM's first layer to calibrate its first layer's training parameters is the first thing that must be done. After that, the output of the hidden layer of the first RBM layer is utilised as the input for the RBM of the second layer, and the parameters of the first layer are gradually learnt. The final hidden layer is linked to the SoftMax regression classifier, and the supervised Gradient Descent algorithm is used to finish the fine‐tuning [26]. W�x;y1 ;y2 ;…;yl�¼ X l−2 n¼0 Wyn��ynþ1�!W�yl−1 ;yl�ð3Þ P(y l−1 ,y l ) has been described as the distribution of the probability between the two levels of the topmost RBM, which are known as the visible and the buried layers. This work utilises three convolutional layers with three max‐pooling layers. Table 2summarises the parameters of the CDBN model in this study. We determined 2 �2 as the kernel window size in this work. For training images, the number of filters in each layer has been increased by including a more FIGURE 2 The representation of implemented architecture of ResNet‐50 network. TABLE 1Architecture of implemented RenNet‐50 model for brain tumour classification. Layer Output size 50‐Layer Conv1 112 �112 7 �7�64, stride 2 3�3 max pooling, stride 2 Conv2‐�56 �56 2 4 1�1�64 3�3�64 1�1�256 3 5�3 Conv3‐�28 �28 2 4 1�1�128 3�3�128 1�1�512 3 5�4 Conv4‐�14 �14 2 4 1�1�256 3�3�256 1�1�1024 3 5�6 Conv5‐�7�72 4 1�1�512 3�3�512 1�1�2048 3 5�4 FLOPs 1 �1 Average pooling, 1000 FC, softmax Abbreviation: FC, fully connected. ZEBARI ET AL. - 795 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License complex pattern. In the testing phase, the CDBN model utilised an image size of 128 �128 and a batch size 200. 3.2.3 | VGG16 model VGG16 is a deep convolutional neural network with 16 layers, and it is known for its simplicity and effectiveness. The network takes an input image of a fixed size (224 �224 �3). VGG16 consists of multiple convolutional blocks, each containing convolutional layers followed by max‐pooling layers [8]. The convolutional layers have small 3 �3 filters, and the max‐ pooling layers use 2 �2 filters with a stride of 2 to downsample the spatial dimensions. After several convolutional blocks, the network has fully connected layers to make the final predictions. The final layer of the network produces the classification probabilities for the different classes. The representation of the implemented architecture of the VGG16 Network is presented in Figure 4. 3.3 | Fusion model A more robust classification is achieved through the fusion technique, which in turn helps to increase the performance of the classification. Recent studies demonstrate that the fusion process improves classification performance; nevertheless, the most significant downside is that it takes much time to compute, although our primary objective is to facilitate the performance evaluation of brain tumours. Figure 5provides a visual representation of the proposed technique for the fusion process. The classification scores are combined into a single final score using this research‐based score‐level fusion method, which more definitively determines the class. Through decision‐level fusion, which may be utilised to conclude, it is feasible to combine the classification decisions generated by several feature vectors and different classifiers into a single decision. The outcomes of the three models' decisions may be incorporated into a single vector. The features of two DL models (ResNet50 and CDBN) were integrated to achieve fusion at the decision level. Many of the problems that have been plaguing ML have found an answer in fusion modelling. This is because it improves overall performance by combining the predictive skills of several models into one model. Several training datasets or approaches are utilised in the fusion process, which additionally incorporates the predicted results of each base model to offer a single anticipated performance. As a result, the facts may be more faithfully portrayed. The purpose of combining many models is to reduce the amount of information that can be generalised about the forecast. When many models are used, the inaccuracy in the classification diminishes, provided that the base models are diverse and independent of one another. Deep Learning synthesis is an approach that aims to increase productivity by consolidating the diagnostic conclusions drawn from various models into a single, unified decision. Additionally, data fusion is a subset of fusion training, and the degree of integration between the two data types affects the classifier used. Incorporation at this level is anticipated to improve classification performance because the feature set of a model contains more information about the MRI images than all integrated classifiers. A decision fusion, on the other hand, comprises a projected decision that is provided for classifying the outcome. In the brain tumour classification work described in the provided information, SoftMax is used as the final classifier in the DL fusion model. After the feature extraction and fusion process using CDBN and ResNet50, the fused feature vector is fed into the SoftMax function in the output layer for multiclass classification. The SoftMax function takes the fused feature FIGURE 3 The representation of implemented architecture of CDBN network. CDBN, convolutional deep belief networks. TABLE 2Implemented CDBN architecture. Layers Input size Input size 128 �128 Number of layers 2 Conv‐1 7 �7 [32 �124 �124] Max‐pooling layer 2 �2 [32 �62 �62] Conv‐2 5 �5 [64 �58 �58] Max‐pooling layer 2 �2 [64 �29 �29] Conv‐3 6 �6 [128 �24 �24] Max‐pooling layer 2 �2 [128 �12 �12] Batch size 200 Epoch 50 Learning rate 0.001 Abbreviation: CDBN, convolutional deep belief networks. 796 - ZEBARI ET AL. 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License vector as input. It transforms it into a probability distribution over the four brain tumour classes: gliomas, meningiomas, pituitary tumours, and no‐tumour class [18]. The SoftMax function calculates the probability that the input image belongs to each class, ensuring that the sum of all probabilities equals 1. The class with the highest probability output by the SoftMax function is considered the predicted class for the input brain tumour image. This SoftMax classification allows the model to make confident and calibrated predictions across multiple classes, providing the probabilities associated with each class for a more informative output. By employing SoftMax as the classifier, the model can effectively handle the multi‐class nature of the brain tumour classification task, providing accurate and interpretable predictions for different tumour types. The use of SoftMax as the final activation function aligns with best practices for multiclass classification tasks in DL. It is likely a key factor contributing to the high evaluation metrics achieved in the study. The SoftMax function is a mathematical function used for multiclass classification. Given a vector of raw scores for each class, the SoftMax function converts these scores into a probability distribution, representing the likelihood of the input belonging to each class; in comparison, the input logits for each class are denoted as a vector z=[z1, z2, …, zK], where K is the number of classes. The SoftMax function for the class kis defined as follows: Additionally, data fusion is a subset of fusion training, and the degree of integration between the two data types affects the classifier used. Incorporation at this level is anticipated to improve classification FIGURE 4 The representation of implemented architecture of VGG16 network. FIGURE 5 Representation of the training steps for VGG16, ResNet50, CDBN, and proposed fusion models. CDBN, convolutional deep belief networks. ZEBARI ET AL. - 797 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License performance because the feature set of a model contains more information about the MRI images than all integrated classifiers. A decision fusion, on the other hand, comprises a projected decision that is provided for classifying the outcome. SoftmaxðzkÞ ¼ expðzkÞ PðexpðziÞÞ;i¼1to K ð4Þ Here, exp(zk) is the exponential function. The SoftMax function applies the exponential function to each logit and then divides it by the sum of the exponentials of all logits. This normalisation step ensures that the output values are non‐ negative and sum up to 1, creating a valid probability distribution. The output of the SoftMax function is a vector of probabilities [p1, p2, …,pK], where each element p_irepresents the probability of the input belonging to class i. The class with the highest chance is typically chosen as the predicted class for the information. Mathematically, SoftMax multi‐class classification can be given as follows: Input: z=[z1, z2, …, zK] (logits for each class) Output: Probability distribution [p1, p2, …, pK]where p_i=exp(z_i)/∑(exp(z_i)) for i =1to K. 4 | RESULTS AND DISCUSSIONS 4.1 | Dataset A dataset downloaded from the Kaggle website was used in this study to validate the performance evaluation of the suggested model [27]. Table 3provides an overview of the information associated with the dataset. Figure 6provides illustrative examples of some samples. The collection contains 3264 MRI images, each representing one of four different classes of brain tumours. The validation of the experiment is carried out in a specific manner by partitioning the dataset into two halves according to the proportion of the size of the training dataset to the testing data, which was 2870 training data samples to 394 testing data samples. Python 3.6.5 was used as the tool to do the simulation of the suggested model. Some libraries have been used to write the code, including Keras and TensorFlow. The hardware system employed had 32 GB of RAM and an i5 9th Generation processor. The following values have been specified for each parameter: learning rate was 0.01, dropout was 0.5, batch size was 5, epoch count was 50, and activation was ReLU. 4.2 | Performance metrics This section contains the four efficiency metrics applied to assess how well the proposed method performed. The other performance measures that are accessible are all different iterations of the specified performance measurements. Accuracy is one of the four leading performance indicators because it provides information about the overall performance of the suggested approach. By utilising Equations (5)–(8), additional performance metrics, including Recall, Precision, and F‐Score, are employed to analyse the suggested model's valid positive and actual negative rates. The accuracy measure offers definitions for the glioma, meningioma, pituitary, and non‐tumour classes depicted in the labelled images. Equation (5) is the factor that determines how accurate the proposed model is. In Equation (5), True Positive denotes cases that have been successfully identified as positive, and True Negative indicates cases that have been successfully recognised as negative. Samples that have been successfully identified as being improperly classified are denoted by the notation False Negative. In contrast, False Positive represents cases that have been wrongly identified as being correctly identified. Recall that it is utilised to quantify the percentage of genuine positive cases we could successfully foresee by employing our model. In other words, it can show us how accurate our predictions were. To compute the recall metric, Equation (6) is utilised. Precision is a measurement used to determine how many of the accurately classified cases actually turned out to be positive. Equation (7) is employed to calculate the rate of precision. F‐Score is a measure for representing the TABLE 3The distribution of the dataset. Class Training Augmented Testing Glioma‐tumour 826 2478 100 Meningioma‐tumour 822 2466 115 Pituitary‐tumour 827 2481 74 No‐tumour 395 1185 105 Total 2870 8610 394 FIGURE 6 Dataset samples used to evaluate the proposed model. 798 - ZEBARI ET AL. 24682322, 2024, 4, Downloaded from https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/cit2.12276 by Technical University Ostrava, Wiley Online Library on [25/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License