The CrowdGleason dataset: Learning the Gleason grade from crowds and experts

Contents lists available at ScienceDirect Computer Methods and Programs in Biomedicine journal homepage: https://www.sciencedirect.com/journal/computer-methods-andprograms-in-biomedicine The CrowdGleason dataset: Learning the Gleason grade from crowds and experts✩ Miguel López-Péreza,∗, Alba Morquechob, Arne Schmidtb, Fernando Pérez-Buenod, Aurelio Martín-Castroc, Javier Mateosb, Rafael Molinab aInstituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Spain bDepartment of Computer Science and Artificial Intelligence, Universidad de Granada, Granada, Spain cDepartment of Pathology, Virgen de las Nieves University Hospital, 18014 Granada, Spain dBasque Center on Cognition, Brain and Language, Donostia - San Sebastián, Spain ARTICLE INFO Keywords: Computational pathology Crowdsourcing Prostate cancer Gleason grade Gaussian processes Medical image analysis ABSTRACT Background: Currently, prostate cancer (PCa) diagnosis relies on the human analysis of prostate biopsy Whole Slide Images (WSIs) using the Gleason score. Since this process is error-prone and time-consuming, recent advances in machine learning have promoted the use of automated systems to assist pathologists. Unfortunately, labeled datasets for training and validation are scarce due to the need for expert pathologists to provide ground-truth labels. Methods: This work introduces a new prostate histopathological dataset named CrowdGleason, which consists of 19,077 patches from 1045 WSIs with various Gleason grades. The dataset was annotated using a crowdsourcing protocol involving seven pathologists-in-training to distribute the labeling effort. To provide a baseline analysis, two crowdsourcing methods based on Gaussian Processes (GPs) were evaluated for Gleason grade prediction: SVGPCR, which learns a model from the CrowdGleason dataset, and SVGPMIX, which combines data from the public dataset SICAPv2 and the CrowdGleason dataset. The performance of these methods was compared with other crowdsourcing and expert label-based methods through comprehensive experiments. Results: The results demonstrate that our GP-based crowdsourcing approach outperforms other methods for aggregating crowdsourced labels (𝜅= 0.7048 ± 0.0207) for SVGPCR vs.(𝜅= 0.6576 ± 0.0086) for SVGP with majority voting). SVGPCR trained with crowdsourced labels performs better than GP trained with expert labels from SICAPv2 (𝜅= 0.6583 ± 0.0220) and outperforms most individual pathologists-in-training (mean 𝜅= 0.5432). Additionally, SVGPMIX trained with a combination of SICAPv2 and CrowdGleason achieves the highest performance on both datasets (𝜅= 0.7814 ± 0.0083 and 𝜅= 0.7276 ± 0.0260). Conclusion: The experiments show that the CrowdGleason dataset can be successfully used for training and validating supervised and crowdsourcing methods. Furthermore, the crowdsourcing methods trained on this dataset obtain competitive results against those using expert labels. Interestingly, the combination of expert and non-expert labels opens the door to a future of massive labeling by incorporating both expert and non-expert pathologist annotators. 1. Introduction Prostate cancer is a prevalent cancer and the fifth leading cause of cancer-related deaths worldwide [1]. Timely and precise diagnosis is crucial for effective treatment and reducing mortality rates [2]. Currently, the gold standard for diagnosis and prognosis is to analyze ✩This work was supported in part by FEDER/Junta de Andalucía under project P20_00286, grant PID2022-140189OB-C22 funded by MICIU/AEI/10.13039/501100011033 and by ‘‘ERDF/EU’’. The work by Miguel López-Pérez and Fernando Pérez-Bueno was supported by the grants JDC2022-048318-I and JDC2022-048784-I, respectively, funded by MICIU/AEI/10.13039/501100011033 the European Union ‘‘NextGenerationEU’’/PRTR. ∗Corresponding author. E-mail addresses: [email protected] (M. López-Pérez), [email protected] (A. Morquecho), [email protected] (A. Schmidt), [email protected] (F. Pérez-Bueno), [email protected] (A. Martín-Castro), [email protected] (J. Mateos), [email protected] (R. Molina). a biopsy of prostate tissue by the Gleason grading (GG) system which assesses the cancer stage and aggressiveness based on gland morphology. However, the assessment of GG is inherently subjective with high intraand inter-observer variability [3,4]. https://doi.org/10.1016/j.cmpb.2024.108472 Received 2 May 2024; Received in revised form 30 September 2024; Accepted 20 October 2024 Computer Methods and Programs in Biomedicine 257 (2024) 108472 Available online 28 October 2024 0169-2607/© 2024 The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC license ( http://creativecommons.org/licenses/bync/4.0/ ). M. López-Pérez et al. Computer-Aided Diagnosis (CAD) systems assist pathologists and aim to minimize human variability in decision-making. These systems utilize WSIs and computer vision and machine learning (ML) algorithms to detect and grade cancerous regions. The main bottleneck in training and validating ML methods for GG prediction is the scarcity of large-scale public datasets [5]. Creating these datasets is costly and time-consuming and, together with the scarcity of expert pathologists, explain why there are few annotated datasets and even fewer public datasets. Crowdsourcing has emerged as a cost-effective and efficient method for labeling histopathological datasets by leveraging a large pool of annotators with varying levels of expertise [6,7]. While crowdsourcing has shown success in tasks like nuclei detection [8] and cancer cell identification [9], the labels generated are frequently noisy, limiting their direct application to complex tasks such as GG. To address this challenge, probabilistic models like GPs have become popular [10,11]. GPs for crowdsourcing have demonstrated excellent performance in various tasks [12–14] and have been successfully applied in histopathological image classification studies, including breast cancer [15,16] and skin cancer detection [17]. These methods offer competitive performance compared to methods trained with expert labels, indicating that crowdsourced labeling of histopathological images could be a feasible option for cancer classification with minimal reliance on expert pathologists. Regarding GG classification, there are no previous studies involving non-expert annotators. However, it has been shown that learning from the opinion of multiple expert pathologists, despite high interand intra-observer variability, results in strong performance when effectively modeling this variability [18,19]. The objective of this work is twofold. First, we present and make publicly available the first prostate dataset labeled by non-experts for GG prediction. Second, we explore the learning from crowds framework with this novel dataset, assessing and analyzing two state-of-the-art methods based on GP for crowdsourcing. This paper also demonstrates the viability of integrating this new dataset with existing datasets containing expert labels, to create a larger and more diverse dataset. Our experiments indicate that the noisy non-expert labels from the presented dataset can improve previous models in the literature. Below, we outline our contributions in detail: •Introduction of new crowdsourcing protocol for the annotation of patches from WSIs, outlined in Fig. 1, which cheapens and speeds up the labeling process by dramatically reducing the intervention of expert pathologists. •Creation of a new dataset, called CrowdGleason, comprising 19,077 patches from 1045 WSIs of PCa with different GG. This dataset was annotated by seven pathologist-in-training without expert supervision. Note that not all annotators labeled all patches. To the best of our knowledge, this is the first PCa dataset annotated by non-expert pathologists. •Development of a curated test set annotated by each pathologistin-training and two expert PCa pathologists to evaluate automated ML methods and assess bias, expertise, and discrepancies between participants. •Comprehensive experiments to evaluate two GP-based methods for GG prediction: SVGPCR [7] and SVGPMIX [14]. SVGPCR learns from the CrowdGleason dataset, while SVGPMIX combines expert labels from the public SICAPv2 [20] dataset with the CrowdGleason dataset. Results demonstrate that these GP-based crowdsourcing methods outperform popular techniques for label aggregation, with SVGPMIX achieving the best performance in both datasets. The remainder of the work is organized as follows. Section 2describes related work. Section 3presents the CrowdGleason dataset and its annotation protocol. Section 4describes the experimental setup and the methods evaluated. The experimental results are shown in Section 5, and Section 6discusses them. Finally, Section 7presents the conclusions and future work. For further information on the code and dataset: https://github.com/vipgugr/CrowdGleason. 2. Related work Public datasets are essential to develop precise ML methods for GG prediction. Hence, Section 2.1 delves into the current publicly available PCa datasets and Section 2.2 provides an overview of the core work of crowdsourcing and its applications in the context of histopathology. 2.1. Public PCa histopathological datasets The current publicly available PCa histopathological datasets have been typically created by staining tissue biopsies with hematoxylin and eosin (H&E) and scanning them as WSI for histopathological examination. In clinical practice, a WSI usually contains one or a few tissue samples. The use of Tissue Micro Arrays (TMAs) allows many tissue samples to be arranged on a grid and processed simultaneously to obtain a single slide. These datasets are labeled at pixel, patch, or WSI levels. The labeling process at pixel level consists of manually delineating tumor areas and assigning GG classes. This meticulous procedure provides comprehensive tumor information but it is timeconsuming. In WSI level labeling, pathologists assign a label to the entire image without specific tumor location information. Patch level labeling divides WSIs into small regions, named patches, and a label is assigned to a selected set of patches, thus reducing the need to examine the entire WSI. We briefly examine popular public datasets for GG prediction, including Arvaniti, SICAP, GLEASON2019, and PANDA. Table 1provides an overview of these datasets and our proposed CrowdGleason. Arvaniti et al. [21] dataset comprises TMAs annotated at pixel level by an expert pathologist, while SICAPv1 [11] and SICAPv2 [20] datasets offer pixel-level annotations on WSIs. The WSIs were downsampled at 10× magnification and divided into patches of 5122pixels with 50% overlap obtaining patch-level annotations. To our knowledge, SICAPv2 is the largest fully annotated dataset at patch level in the literature. Challenges, such as Gleason2019 and PANDA, have been a popular way of promoting research in GG prediction by providing benchmark datasets for evaluating ML algorithms. The Gleason2019 challenge [18] dataset provides TMA images annotated by a panel of 5 expert pathologists, and the PANDA Challenge [22] dataset includes WSIs annotated at the WSI level by consensus among a large panel of highly experienced expert pathologists, with some samples annotated at pixel level. 2.2. Crowdsourcing To the best of our knowledge, all works for GG prediction from patches addressed the problem with ground-truth labels provided by either a single expert or a panel of expert pathologists. Crowdsourcing presents an opportunity to scale datasets by engaging non-expert annotators in computational pathology-related tasks [8]. Various studies explored the use of labels from non-expert annotators for tasks like mitosis detection [23] or histopathological image classification [24]. Previous works [25,26] have demonstrated promising results in the field of histopathology using crowdsourcing, but they required strong supervision from senior pathologists to review the annotations provided by the crowd. To reduce the need for expert supervision, label aggregation techniques [27] have been developed to automatically curate crowdsourcing labels, enabling the creation of datasets suitable for ML without expert supervision. Various label aggregation methods have been proposed, including majority voting (MV) and more elaborated methods that consider the biases of the different annotators, yielding a better-calibrated set of training labels [7]. They include Dawid-Skene (DS) [28], GLAD [29] and MACE [30] models. Recent studies show that jointly learning ground-truth labels, annotator expertise, and the latent classifier leads to superior performance [31]. Models like SVGPCR [7] have successfully combined Computer Methods and Programs in Biomedicine 257 (2024) 108472 2 M. López-Pérez et al. Fig. 1. Dataset creation and annotation protocol. We collect 1045 WSIs, of which 783 are used exclusively for crowd labeling and 262 for crowd and expert labeling. We divide all WSIs into patches and distribute them among the non-expert annotators to obtain the training set. We create a curated test set with patches where the experts and majority of the non-experts agree. Table 1 Publicly available datasets for GG prediction. MA refers to multiple annotators. Biopsy # Samples Annotations Experts MA Arvaniti [21] TMA 895 Pixel-level Yes No SICAPv1 [11] WSI 79 Pixel-level Yes No SICAPv2 [20] WSI 182 Patch-/pixel-level Yes No GLEASON19 [18] TMA 331 Pixel-level Yes Yes PANDA [22] WSI 12,625 WSI-/pixel-level Yes No CrowdGleason (proposed) WSI 1,045 Patch-level No Yes sparse GPs with a crowdsourcing probabilistic framework, demonstrating competitive performance to GPs trained with expert labels in breast cancer detection from histopathological images [32]. Moreover, SVGPMIX method [14] is the first probabilistic approach based on GPs for fusing expert and non-expert labels, leveraging the confidence offered by expert labels and the larger volume of data provided by nonexpert annotators. To the best of our knowledge, this model has not yet been applied in the biomedical domain. 3. CrowdGleason dataset The dataset presented in this paper, named CrowdGleason, has been partially annotated by different pathologists in-training with varying degrees of expertise. A subset of the dataset was also annotated by expert pathologists, which helped to obtain a test set. 3.1. Data acquisition and annotation by expert pathologists To create CrowdGleason, 1045 WSIs of H&E-stained prostate tissue samples from different patients, were collected by medical experts from the archive of the Hospital Universitario San Cecilio (HUSC) in Granada. All WSIs were digitally scanned at 40× magnification factor. Two expert pathologists exhaustively annotated 262 of those 1045 WSIs at the pixel level. Each image was annotated by only one of the pathologists independently, using the online application described in [20]. Experts thoroughly marked all pathological areas with their GG and delineated artifacts. 3.2. Patch extraction All WSIs were divided into patches of size 2048 ×2048 pixels at a magnification of 40x, without overlapping. This size and magnification were selected in agreement with expert pathologists to provide sufficient context and detail to facilitate the identification of cancerous lesions. Patches containing less than 20% of tissue were discarded, as they do not contain enough tissue to make an accurate diagnosis. Tissue presence was detected by thresholding the magenta channel by the Otsu method [33]. From images with pathological areas marked by the experts, we selected patches containing at least 15% of pathological tissue, labeled with GG of the area marked by the expert: Gleason grade 3 (G3), Gleason grade 4 (G4), or Gleason grade 5 (G5). Patches containing more than one pathological area were discarded since it was not possible to assign a single label to the patch. From images labeled as non-cancerous (NC) by the expert pathologist, on the other side, we could use all tissue to extract patches. To reduce the number of candidate patches, we discarded patches having less than 30% of tissue. A total of 4573 patches, which form the so called expert-labeled set, were obtained from the 262 images annotated by experts. For the remaining 783 images not annotated by expert pathologists, a large number of patches without ground-truth labels were extracted to be annotated by non-expert pathologists at a later stage. To expedite the labeling process, we reduced the number of patches. As gland structure is crucial in PCa diagnosis, we chose patches with a substantial presence of nuclei as representative of tissue with glands. Since nuclei stain with hematoxylin, which is prominent in the cyan component, to extract patches rich in nuclei, we selected those patches where at least 40% of the tissue’s pixels had a high cyan value. Still the number of patches was overwhelming. Due to the huge class imbalance in histopathological data, with large areas of non-pathological tissue, and cancerous tissue that is only sparsely represented, to select a set of patches representing the different PCa grades, we proceeded as follows. We trained the classification algorithm in [34] that combines semisupervised and multiple instance learning on the public PANDA dataset for PCa classification, following the successful setting in [34]. Note that the PANDA dataset is labeled at the WSI level; hence, standard supervised learning techniques cannot be used. Although segmentation masks are provided for some images, they can be only used to develop strategies for selecting the most significant subsamples of the Computer Methods and Programs in Biomedicine 257 (2024) 108472 3 M. López-Pérez et al. Fig. 2. Examples of patches and annotations by each participant. Table 2 Metrics for the patch selection algorithm in a small set of patches extracted from the expert-labeled set. Accuracy F1 score Kappa 0.732 0.648 0.606 images [35]. The algorithm in [34] uses an EfficientNet-B5 neural network architecture [36] that was trained with a learning rate of 0.01 for 10 epochs on the classes NC, G3, G4, and G5. To validate this approach, we classified a small set of patches extracted from the expertlabeled set. Note that these patches were only used to obtain the metrics shown in Table 2and were not used in model training. These figures of merit show that the method is good enough to distinguish patches from the different classes. Using the learned model, the patches from nonannotated WSIs were classified in the class with the highest probability, selecting a total of 16,151 patches. This collection constitutes a roughly balanced set that will serve as training set of our study. Since this set was designed for annotation by non-expert pathologists, the labels provided by the classification algorithm were discarded after patch selection and not further used in this study. 3.3. Annotation by non-expert pathologists Seven pathologists in-training with different expertise levels participated in the annotation of the dataset. Two of them were in their fourth year of medical residency, two in their third, two in their second, and one was a first-year medical resident. Following the Spanish Official Specialist Training Program in Anatomic Pathology, third and fourthyear pathologists in-training have completed specific training in the subspecialty of Uropathology, which includes the study of the prostate, with an approximate duration of 2–3 months. First and second-year students do not have specific training in this subspecialty. In collaboration with expert pathologists, we designed an annotation protocol based on the well-known PCa grading of the Gleason Score, originally introduced by Donald F. Gleason [37] for grading prostatic carcinoma based solely on the architectural pattern of the tumor. Non-expert pathologists were instructed to label as NC, G3, G4, or G5 the patches in the expert-labeled and training sets, described in the previous section, rather than thoroughly examine the WSI and exhaustively delineate the tumor areas. Patches that could not be labeled due to the presence of artifacts, blurriness, tissue from other organs, folded tissue, etc. have been labeled as ‘‘Other’’. Patches with more than one GG, which have not a clear label, have also been labeled as ‘‘Other’’. All pathologists in-training labeled the 4573 patches in the expertlabeled set. Table 3presents a summary of the distribution of the labels in this set. Note that some patches were labeled as ‘‘Other’’ by some residents and, hence, the total of each column may not add up to the total number of patches. An example of patches and the annotations provided by the crowd and the experts is shown in Fig. 2. To minimize the influence of inherent pathologist variability in labeling, we created acurated test set where the ground-truth label for each sample was established by consensus between the majority of pathologists in-training and the expert pathologist. Using this curated set, whose distribution of samples for each class is shown in Table 4, we will estimate the degree of reliability of each resident as well as evaluate ML methods. Recall that expert intervention has only been necessary for the creation of the curated test set, not for the training set. The 16,151 patches in the training set were labeled, on average, by more than two resident pathologists, with each pathologist intraining labeling approximately 5000 patches. Table 5summarizes the training set. Patches were provided to the residents in 4 batches of approximately equal size over a 6-month period. Finally, as a post-processing step, the patches of both the training set and the curated test set were downsampled using bicubic interpolation to a size of 512 ×512 pixels. This is equivalent to obtaining the patches at a magnification factor of 10×, and it was necessary to accommodate the patches into the GPU memory. In summary, the CrowdGleason consists of a crowdsourcing annotated training set with 16,151 patches of size 512 ×512 extracted from 783 WSIs, annotated by one or more of the seven pathologists in-training, and a curated test set with 2926 patches of size 512 ×512 extracted from other 262 WSIs, annotated by expert pathologists and all the pathologists in-training. Ground-truth labels for the curated test set were obtained by consensus between the expert pathologists and the majority of the pathologists in-training. 3.4. Ethical consent and data availability The Research Ethics Committee of the Universidad de Granada approved the study with code 4096/CEIH/2024 as part of the project P20_00286, funded by FEDER/Junta de Andalucía, following the principles established in international and national biomedical international and national legislation in the field of biomedicine, bioethics and bioethics, as well as the rights derived from the protection of personal data. The complete CrowdGleason dataset will be available in Figshare upon acceptance of the paper. 4. Materials and methods Datasets. We present and utilize the dataset, described in Section 3.1, and combine it with the public dataset SICAPv2 [20]. We normalize both datasets via the BKSVD method [38], and use them for training and evaluation to demonstrate the utility of the proposed CrowdGleason dataset with respect to another popular dataset in the literature. Our approach also allows for the generalization ability of Computer Methods and Programs in Biomedicine 257 (2024) 108472 4 M. López-Pérez et al. Table 3 Distribution of labels in the expert-labeled set labeled by each resident pathologist and the expert. We refer to each annotator as A#, where the number is an anonymized ID. Class A1 A2 A3 A4 A5 A6 A7 Expert Total NC 1891 2290 2968 2983 2941 2868 389 2438 14,439 G3 1024 1267 507 601 763 693 1402 1498 5233 G4 1155 674 808 663 413 666 1513 449 4737 G5 503 341 281 308 431 317 1074 188 2752 Total 4573 4572 4564 4555 4548 4544 4378 4573 27,161 Table 4 Distribution of patches in each class of the curated test set. NC G3 G4 G5 Total 2157 548 164 57 2926 the classifiers on an external cohort. The task is to learn the GG of each patch, i.e. to classify the patches as ‘NC’, ‘G3’, ‘G4’ and ‘G5’. Feature extraction. A reduced set of features is extracted and used as input to the GP-based methods. For this, we utilize the 18-layer variant of ResNet [39], i.e., ResNet18, pre-trained on Imagenet and fine-tuned on SICAPv2. We use the output of the last convolutional layer as a feature extractor. Since SICAPv2 is the largest publicly available PCa dataset with patch labels, we assume that the learned feature extractor generalizes well to other datasets. The experimental results will validate this assumption. We utilize these 512-dimensional feature vectors as input to the GP classifiers and also report the results of the end-to-end training of ResNet18 for comparison. We perform five independent runs of all the presented experiments, including the mean performance and 95% confidence interval (CI). Note that for each run, we also run the feature extractor to obtain a different set the features and ensure the robustness of the whole pipeline proposed in this work. To train the network, we use the SGD optimizer with a learning rate of 10−3, a momentum of 0.9, and a batch size of 32 patches. Common data augmentation transformations, such as horizontal and vertical flips, blur, and brightness, contrast, hue, and saturation variations, are applied to the training dataset. The CNN is implemented using Pytorch 2.0.1 and is run on an NVIDIA GeForce RTX 3090 GPU. Supervised Learning: Gaussian Processes. We use the features extracted by ResNet18 and the ground-truth label as inputs of a stochastic variational Gaussian process (SVGP) model to learn the GG from the SICAPv2 dataset. SVGPs are scalable GP models that use variational inference to approximate the posterior distribution. See a detailed description in [40] and an intuitive review in [32]. We utilize a squared exponential kernel to compute the correlation matrix. We initialize the kernel hyperparameters, lengthscale and variance, to 2. We train the SVGP for 50 epochs and save the parameters that obtain the best Cohen’s Quadratic Kappa (𝜅) on the validation set. In all cases, we use the Adam Optimizer with a learning rate of 10−2and a batch size of 128. Based on the experimental results (see Section 5.1), we fixed a value of 512 inducing points, which provides a good trade-off between the generalization and complexity of the model, for all GP based methods. SVGP is implemented using GPflow 1.2.0 and is run on an NVIDIA GeForce RTX 3090 GPU. The code will be released in GitHub upon acceptance of the paper. Label aggregation. We utilize and compare Majority Voting (MV), DS [28], MACE [30], and GLAD [29] aggregation methods to curate the multiple noisy labels available in the CrowdGleason dataset. The aggregated labels can be used as the single ground-truth label and, therefore, used by supervised learning methods. All methods, implemented in the popular Python library for crowdsourcing tasks crowd-kit [41], are run with the default hyperparameters. The aggregated labels and the features extracted by ResNet18 are fed to SVGP to learn a GG classifier. Crowdsourcing models. We utilize, as an enhancement of the label aggregation methods, the learning from crowds model based on GPs, SVGPCR [7]. This model extends the GPs to the crowdsourcing scenario and jointly learns the expertise of the annotators and the GP classifier. The main assumption is that multiple annotators provide noisy labels that are corrupted observations of the ground-truth label. This corruption is modeled with a confusion matrix for each annotator, which reflects the probability of providing a given label for each ground-truth class (as in the DS model [28]). Once trained, the model can predict ground-truth labels in unseen instances using the GP classifier. Furthermore, we analyze how crowdsourcing labeled datasets can be used in conjunction with expert labeled datasets to learn a classifier. For this purpose, we use the SVGPMIX model [14]. This model, used here for the first time in medical imaging, generalizes SVGPCR to cases where labels have been provided either by a noisy annotator or by an expert. We utilize this model to study the combination of CrowdGleason with SICAPv2. SVGPCR and SVGPMIX use the same training procedure and software framework as supervised GPs. Evaluation Metrics. To assess the quality of the learned models, we report the numerical results of three different metrics: Accuracy, Cohen’s Quadratic Kappa (𝜅), and the F1-score. The accuracy is the rate of success of the classifiers. The F1 can be defined per class as the harmonic mean between precision and recall. We only report multiclass F1, which can be defined as the average across class-wise F1. Recall that this score is of special importance in imbalanced scenarios, which are common in medical imaging. Finally, the 𝜅score is increasingly popular for GG assessment [20,21,42,43]. It measures the level of agreement between the output of the classifier and the ground-truth label [44]. We can also use it to measure the agreement between annotators. The 𝜅metric ranges from −1 to 1, being directly proportional to the level of agreement between observers (−1 complete disagreement, 0 no agreement beyond what would be expected by chance, 1 total agreement). It is commonly argued that a moderate agreement is achieved if 𝜅is higher than 0.6, whereas a strong agreement is attained when 𝜅is higher than 0.8. Furthermore, this metric also penalizes disagreements depending on class differences (in a quadratic manner). For example, a disagreement between classes NC and G5 implies a stronger penalization than between classes NC and G3. 5. Experimental results In this section, we report the results of a set of experiments. They compare different GP-based approaches that learn from expert labels, crowdsourcing labels, and a combination of both. 5.1. Experiment 1: Expert labels In this experiment, we present the results of models trained on expert SICAPv2 labels. The model is validated using the validation set of SICAPv2. For comparison purposes, we also train the CNN-based method ResNet18 with the same data. To select the number of inducing points for the SVGP method, we run the method with several values: 64, 128, 256, 512, and 1024. Fig. 3 shows that the F1 and Kappa scores are stable in the SICAPv2 validation set across different numbers of inducing points. This result means that the information can be summarized in a few points of the feature space, and adding more flexibility does not improve the performance. Furthermore, we can see that a large number of inducing points does not lead to overfitting. Hence, we fix a value of 512 inducing points for Computer Methods and Programs in Biomedicine 257 (2024) 108472 5 M. López-Pérez et al. Table 5 Distribution of patches in the training set labeled by each resident pathologist. We refer to each annotator as A#, where the number is an anonymized ID. Class A1 A2 A3 A4 A5 A6 A7 Total NC 2165 2747 3659 4186 3476 3452 866 20,551 G3 1462 1618 604 149 1024 667 682 6206 G4 995 510 544 580 405 641 2580 6255 G5 400 205 140 111 250 337 677 2120 Total 5022 5080 4947 5026 5155 5097 4805 35,132 Table 6 Results of the methods trained on SICAPv2 (expert labels) when tested on SICAPv2 and CrowdGleason test sets. Method SICAPv2 CrowdGleason Accuracy F1 score Kappa Accuracy F1 score Kappa ResNet18 𝟎.𝟕𝟔𝟒𝟖 ±𝟎.𝟎𝟏𝟎𝟐 𝟎.𝟕𝟏𝟒𝟓 ±𝟎.𝟎𝟏𝟒𝟑 0.6611 ± 0.0149 𝟎.𝟖𝟖𝟑𝟗 ±𝟎.𝟎𝟏𝟐𝟐 𝟎.𝟔𝟔𝟗𝟖 ±𝟎.𝟎𝟑𝟏𝟕 𝟎.𝟕𝟎𝟗𝟓 ±𝟎.𝟎𝟒𝟐𝟔 SVGP-SICAP 0.7515 ± 0.0048 0.6912 ± 0.0119 𝟎.𝟕𝟕𝟑𝟔 ±𝟎.𝟎𝟏𝟑𝟗 0.8736 ± 0.0075 0.6628 ± 0.0061 0.6583 ± 0.0220 Table 7 Results of the methods trained on CrowdGleason (crowdsourcing labels) when tested on SICAPv2 and CrowdGleason test sets. Method SICAPv2 CrowdGleason Accuracy F1 score Kappa Accuracy F1 score Kappa ResNet18-MV 0.5910 ± 0.0568 0.5251 ± 0.0694 0.4139 ± 0.0763 0.8815 ± 0.0149 0.6791 ± 0.0316 0.6958 ± 0.0415 SVGP-DS 0.4960 ± 0.0233 0.4345 ± 0.0282 0.4965 ± 0.0283 0.8402 ± 0.0121 0.5499 ± 0.0360 0.6152 ± 0.0236 SVGP-MACE 0.4980 ± 0.0212 0.4345 ± 0.0263 0.4759 ± 0.0347 0.8486 ± 0.0070 0.5363 ± 0.0300 0.5574 ± 0.0325 SVGP-GLAD 0.4909 ± 0.0256 0.4342 ± 0.0344 0.5052 ± 0.0533 0.8539 ± 0.0091 0.5410 ± 0.0260 0.5776 ± 0.0338 SVGP-MV 0.6861 ± 0.0138 0.6331 ± 0.0169 0.6242 ± 0.0277 0.8649 ± 0.0016 0.6287 ± 0.0123 0.6576 ± 0.0086 SVGPCR 𝟎.𝟕𝟏𝟐𝟑 ±𝟎.𝟎𝟎𝟕𝟐 𝟎.𝟔𝟖𝟓𝟎 ±𝟎.𝟎𝟎𝟕𝟓 𝟎.𝟔𝟗𝟓𝟑 ±𝟎.𝟎𝟏𝟕𝟔 𝟎.𝟗𝟎𝟐𝟑 ±𝟎.𝟎𝟎𝟑𝟕 𝟎.𝟕𝟎𝟔𝟖 ±𝟎.𝟎𝟏𝟒𝟐 𝟎.𝟕𝟎𝟒𝟖 ±𝟎.𝟎𝟐𝟎𝟕 Fig. 3. Variation of the F1 and Kappa scores with the number of inducing points for the SVGP model trained on the SICAPv2 dataset. These results are reported on the validation set. all experiments as a trade-off between complexity and generalization capability. Table 6shows the results of the SVGP method trained on SICAPv2 (denoted as SVGP-SICAP) tested on the SICAPv2 and CrowdGleason curated test sets. Additionally, Table 6includes the test results of the end-to-end trained ResNet18 network for comparative analysis. The SVGP-SICAP classifier achieves better figures-of-merit for the Kappa score in SICAPv2 than using ResNet18 and is competitive in the rest of the metrics. 5.2. Experiment 2: Crowdsourcing labels In this experiment, we train the methods with the CrowdGleason dataset. The dataset is split into 13824 training samples and 2327 validation samples. For validation, we use the MV strategy for aggregating the labels. We first use different label aggregation strategies (DS, MACE, GLAD, and MV) to train the SVGP classifier. Note that the input features, as we have already indicated, are extracted using ResNet18 trained on SICAPv2. Results in Table 7show that MV produces the best result among the label aggregation strategies followed by DS. For comparison purposes, we also report the results on ResNet18 trained end-to-end with the MV labels. Finally, we report the results of the SVGPCR method trained with the crowdsourcing labels of CrowdGleason. From Table 7it is clear that SVGPCR outperforms the rest of the methods in the literature in both datasets. The MV aggregation strategy can reduce the bias of the annotations but the noisy labels hinder the classifier’s performance. 5.3. Experiment 3: Combining expert and crowdsourcing labels Until now, information from experts and crowds was not used simultaneously. In this experiment, we explore the possibility of enhancing expert-labeled datasets with crowdsourcing-labeled ones. We add the CrowdGleason training set to the SICAPv2 training set for this purpose. For supervised methods (SVGP and ResNet18), we consider MV aggregation, since it achieved the best results in experiment 2. All methods use SICAPv2 as the validation set since it already provides ground-truth labels. Results are shown in Table 8. SVGPMIX outperforms the competing methods on the SICAPv2 and CrowdGleason test datasets, showing that the combination of expert and crowdsourcing labels is feasible and beneficial. Although ResNet18-MV achieves a slightly higher F1 score value than SVGPMIX on SICAPv2 (F1 = 0.7137 ± 0.0119 vs. F1 = 0.7216 ± 0.0152), its Kappa value is much lower (𝜅= 0.6748 ± 0.0085) compared to SVGPMIX (𝜅= 0.7814 ± 0.0083). We observe a similar behavior in the SVGP-MV performance. We believe that this is due to the presence of noisy labels in the combined dataset. In contrast, SVGPMIX achieves a satisfying Kappa value on the SICAPv2 (𝜅= 0.7814 ± 0.0083) and CrowdGleason (𝜅= 0.7276 ± 0.0260) datasets, demonstrating its robustness. To assess the statistical significance of our results, we apply the Almost Stochastic Order (ASO) test [45,46] (implemented in the deep significance library1) on the five random runs of both SVGP-SICAP and 1https://deep-significance.readthedocs.io/en/latest/ Computer Methods and Programs in Biomedicine 257 (2024) 108472 6 M. López-Pérez et al. Table 8 Results of the methods trained on SICAPv2 and CrowdGleason (expert and crowdsourcing labels, respectively) combined when tested on SICAPv2 and CrowdGleason test sets. Method SICAPv2 CrowdGleason Accuracy F1 score Kappa Accuracy F1 score Kappa ResNet18-MV 𝟎.𝟕𝟕𝟒𝟑 ±𝟎.𝟎𝟎𝟓𝟔 𝟎.𝟕𝟐𝟏𝟔 ±𝟎.𝟎𝟏𝟓𝟐 0.6748 ± 0.0085 0.8804 ± 0.0192 0.6843 ± 0.0355 0.7042 ± 0.0381 SVGP-MV 0.6861 ± 0.0138 0.6331 ± 0.0169 0.6242 ± 0.0277 0.8649 ± 0.0016 0.6287 ± 0.0123 0.6576 ± 0.0086 SVGPMIX 0.7660 ± 0.0056 0.7137 ± 0.0119 𝟎.𝟕𝟖𝟏𝟒 ±𝟎.𝟎𝟎𝟖𝟑 𝟎.𝟗𝟎𝟐𝟕 ±𝟎.𝟎𝟎𝟗𝟔 𝟎.𝟕𝟏𝟕𝟔 ±𝟎.𝟎𝟐𝟕𝟎 𝟎.𝟕𝟐𝟕𝟔 ±𝟎.𝟎𝟐𝟔𝟎 the proposed SVGPMIX models. The test was performed on the F1 score metric since it takes into account the imbalanced scenario presented in this paper. The ASO test outputs a value between 0 and 1 indicating the degree of violation in stochastic order, where a value below 0.5 indicates that the SVGPMIX model performs statistically better than SVGP-SICAP. Using ASO with a confidence level 𝛼= 0.05, we found the score distribution of SVGPMIX to be stochastically dominant over SVGP-SICAP (𝜖min = 0.0615 in SICAPv2 and 𝜖min = 0.0in CrowdGleason). In conclusion, the proposed CrowdGleason dataset outperforms the model trained on SVGP-SICAP. 5.4. Ablation studies on the quality of the labels and the number of annotators We have seen how well the models perform on the test set but have not analyzed in-depth the role of the non-expert annotators in the crowdsourcing model. In this subsection, we provide an ablation study on the crowdsourcing models, highlighting the impact of crowdsourced annotations into the final performance. First, we assess the effect of experience on pathologists in training by dividing them into two groups: junior (residents in their first or second year) and senior (third or fourth year residents). We train the SVGPCR model using the same configuration as in previous experiments, but in two different settings: (i) using only samples labeled by junior participants and (ii) using only samples labeled by senior participants. This experiment is conducted over five independent runs. The results are shown in Fig. 4 which includes the mean performance and 0.95 CI. The performance is comparable across both datasets; however, the model trained with junior-labeled samples performs better on the CrowdGleason dataset, while the model trained with senior-labeled samples excels in the SICAPv2 dataset. Since the junior residents were specifically trained using the CrowdGleason dataset, the model trained with their annotations tends to overfit. In contrast, the senior participants, with greater experience, are able to recognize a broader range of patterns, allowing the model to generalize more effectively to the SICAPv2 dataset. Secondly, in the ablation study on the number of annotators, we investigate how many annotators are sufficient to achieve a satisfactory crowdsourcing model. For this, we trained the SVGPCR model with varying numbers of annotators. For each number of annotators, we performed eight independent runs, randomly sampling different subsets of annotators. For each subset, we run the models five times to ensure stability and consistency. Fig. 5illustrates the results for the SICAPv2 and CrowdGleason datasets, showing the mean performance, the 95% CI, and the SVGPCR performance with all annotators. As we increase the number of annotators, the CI narrows, indicating that the models become more stable and less dependent on the specific annotators selected. On both test datasets, the performance of the models trained with subsets of annotators overlaps with that of the model trained with all annotators. This suggests that fewer annotators can achieve comparable performance. Overall, for this experiment, about five annotators appear sufficient to achieve satisfactory results, although increasing the number of annotators leads to a more stable performance since the results are highly influenced by the expertise of the selected annotators. 5.5. Ablation study on the impact of expert label datasets in the crowdsourcing scheme This section analyzes the impact of the expert labels from SICAPv2 on the SVGPMIX model. We investigate how many expert-labeled samples are necessary to achieve a robust SVGPMIX model. For this, we trained the SVGPMIX model using CrowdGleason and different proportions of expert-labeled samples from SICAPv2. For each proportion, we performed eight independent runs, randomly sampling different subsets of expert-labeled data. For each sampled dataset, we run the model five times to ensure stability and consistency. Fig. 6illustrates the results for each dataset, showing the mean performance, the 95% CI, and both SVGPMIX and SVGP-SICAP performance with all expert-labeled samples. Unlike crowdsourced-labeled samples, increasing the number of expert-labeled samples does not significantly narrow the CI, as expert labels tend to have less variability and are inherently more robust. Notably, when 10% of the expert samples are used, the model stabilizes (and also surpasses the results from SVGP-SICAP in the SICAPv2 dataset), indicating that the samples are highly informative for training the crowdsourcing model. Beyond this point, adding more samples does not provide additional benefits, demonstrating that the model can perform effectively with a relatively small amount of expert data. 5.6. Analysis of annotator behavior We measure the performance of each non-expert annotator by means of the Kappa score. The figures-of-merit, shown in Table 9, indicate the degree of agreement between each annotator and the curated test set. The best-performing annotator is A4 (𝜅= 0.7765), while A7 presents the lowest agreement (𝜅= 0.0899). The disparity of performance among annotators highlights the crowd heterogeneity and complexity of the task. We further depict the per-class behavior of the annotators in Fig. 7. The confusion matrices are normalized row-wise for better visualization and comparison purposes. These matrices can be understood as an estimation (on the test set) of the annotators’ expertise. The crowdsourcing methods aim to estimate these confusion matrices from the noisy labeled training set. Recall that the ground-truth labels are not observed for these models. Figs. 8and 9show the estimated confusion matrices estimated by SVGPCR and SVGPMIX, respectively. These matrices closely approximate the annotators’ behavior, emphasizing the excellent performance of the crowdsourcing methods. 6. Discussion Our experiments have shown (see Tables 6and 7) that SVGP improves the performance of ResNet18 tested on SICAPv2 and is competitive or outperforms ResNet18 when tested on the new CrowdGleason. These results confirm the potential of GPs to perform GG classification. The SVGPCR classifier, used in the learning from crowds framework, achieved a value of 𝜅= 0.7048 ± 0.0207 and 𝜅= 0.6953 ± 0.0176 on CrowdGleason and SICAPv2 test sets, respectively (see Table 7), outperforming label aggregation strategies, such as MV, DS, MACE, and GLAD. The best label aggregation model (i.e., MV) obtains 𝜅= 0.6576 ± 0.0086 and 𝜅= 0.6242 ± 0.0277 (see Table 7) for CrowdGleason and SICAPv2 test sets, respectively. This significant difference highlights the Computer Methods and Programs in Biomedicine 257 (2024) 108472 7 M. López-Pérez et al. Fig. 4. Results of the SVGPCR model trained with labels provided by junior participants (first and second year) or senior participants (third and fourth year). Fig. 5. Results of the SVGPCR model varying the number of annotators. Fig. 6. Results of the SVGPMIX model varying the proportion expert-labeled samples from SICAPv2. enormous impact of noisy labels provided by non-expert annotators on the models performance and the need to use a suitable model to learn from crowds. Furthermore, the SVGPCR results are competitive with SVGP trained on SICAPv2 with expert labels that obtain 𝜅= 0.6583 ± 0.0220 and 𝜅= 0.7736 ± 0.0139 (see Table 6). Regarding the F1 metric, SVGPCR can even improve the performance of SVGP trained with expert labels on both test datasets. These results align with previous works in crowdsourcing [12,15,16,18], and validate the use of the proposed CrowdGleason dataset for further studies on crowdsourcing and GG. We have also explored the combination of the SICAPv2 dataset with our dataset. Recall that learning a model with samples from two Computer Methods and Programs in Biomedicine 257 (2024) 108472 8 M. López-Pérez et al. Fig. 7. Normalized confusion matrices of the seven annotators in the CrowdGleason curated test set. Table 9 Cohen’s Quadratic Kappa (𝜅) coefficient of non-expert annotators on the CrowdGleason curated test set. A1 A2 A3 A4 A5 A6 A7 𝜅0.4120 0.6283 0.5394 0.7765 0.7040 0.6520 0.0899 different centers is difficult due to the heterogeneity between samples and labels. Additionally, noisy labels from non-expert annotators introduce noise into the dataset, which worsens the classifier performance. See, for instance, the decrease from 𝜅= 0.7736 ± 0.0139 of SVGPSICAP in Table 6to 𝜅= 0.6242 ± 0.0277 of SVGP-MV in Table 8. In this work, we propose using SVGPMIX to address this issue. SVGPMIX extends SVGPCR to the scenario where some labels are given by one expert and the rest are given by multiple non-experts. In this case, SVGPMIX improves the results of both SVGP-SICAP and SVGPCR on both datasets (see Table 8). Specifically, it achieves 𝜅= 0.7276 ± 0.0260 and 𝜅= 0.7814 ± 0.0083 on CrowdGleason and SICAPv2, respectively. Remarkably, SVGPMIX achieves stable results with only 10% percent of samples labeled by expert pathologists. The results obtained by both SVGPCR and SVGPMIX are within the range of results reported in the literature for GG classification. For example, Marrón-Esquivel et al. [42] reported 𝜅= 0.826, Xiang et al. [47] reported 𝜅= 0.81 and Arvaniti et al. [21] reported 𝜅= 0.49 and 𝜅= 0.53 for two different pathologists. During the study, we have observed great variability between annotators that is even more accentuated when they have little experience in the area. Table 9shows that the results obtained by non-experts in the CrowdGleason curated test set are very dissimilar ranging from 𝜅= 0.09 to 𝜅= 0.78. In general, we observe a lower mean agreement with the test set (𝜅= 0.5432) than that observed in other works involving only expert pathologists. For example, in [42] the authors reported 𝜅= 0.6946 among expert pathologists, and in [21] two expert pathologists scored 𝜅= 0.71. The annotators classified non-cancerous patches relatively well, but had more confusion between classes G3 and G4 (see Fig. 7). SVGPCR and SVGPMIX automatically estimate these confusion matrices from the noisy training data. The results in Figs. 8and 9 show that the estimated matrices capture the behavior of the noisy annotators. For instance, both models capture the higher sensitivity in the G4 and G5 grades of annotator 7. Furthermore, these models also capture the behavior of the annotators when labeling samples as ‘NC’. The models correctly estimate that annotators 1 and 7 have the lowest sensitivity in this class (as seen in Fig. 7). Note also that SVGPCR achieves a better concordance (Kappa value) on the test set than most pathologists in-training, as seen in Tables 7and 9. This means that SVGPCR outperforms each pathologist in-training individually. As an Computer Methods and Programs in Biomedicine 257 (2024) 108472 9