Multimodal Learning Approaches for Colorectal Cancer

Multimodal Learning Approaches for Colorectal Cancer Miljana SHULAJKOVSKAa, 1 , Jitendra JONNAGADDALAb, and Anton GRADIŠEKa a Jožef Stefan Institute, Ljubljana, Slovenia b UNSW Sydney, NSW Australia ORCiD ID: Miljana Shulajkovska https://orcid.org/0009-0009-8833-9802 Jitendra Jonnagaddala https://orcid.org/0000-0002-9912-2344 Anton Gradišek https://orcid.org/0000-0001-6480-9587 Abstract. The digitalization of tissue samples through whole slide imaging (WSI), coupled with advancements in deep learning (DL), has opened new possibilities for cancer diagnosis and prognosis. The increasing availability of complementary data modalities, such as genomics and clinical information, has driven the adoption of multimodal approaches. By simultaneously processing these diverse data sources, multimodal models can uncover complex patterns and enhance predictive accuracy. Colorectal cancer (CRC) is one of the leading causes of cancer-related mortality worldwide. This workshop highlights recent advancements in the application of multimodal deep learning techniques to CRC, integrating WSI with other data modalities to improve diagnostic and prognostic capabilities. Keywords. Colorectal cancer, whole slide images, multimodal models, deep learning 1. Introduction Colorectal cancer is one of the leading causes of cancer-related mortality worldwide (1). The digitalization of tissue samples into WSI allows for the capture of detailed morphological and histopathological characteristics. These digitized slides provide critical insights into cellular structures and the tumor microenvironment, which are essential for accurate diagnosis and prognosis. A patient’s condition is reflected in multiple data modalities. In addition to tissue-based information, genomic features, such as biomarkers, offer valuable insights into the molecular and genetic landscape of tumors, which are crucial for understanding cancer biology and predicting therapeutic responses. Clinical data further enrich this understanding by providing patient-specific information, including medical history, demographics, and treatment details. Artificial intelligence (AI), especially deep learning (DL) and multimodal learning, has significantly advanced the field of digital pathology (2), enabling the analysis of complex patterns that may be challenging for human experts to discern. Recent developments in DL and AI have demonstrated substantial potential for CRC diagnosis and (3–5) prognosis (6). While traditional approaches often rely on singlemodal data, DL techniques applied to WSI provide deeper insights. Furthermore, the integration of additional data sources, such as clinical records and genomic profiles, enhances the predictive power of these models. 1 Corresponding Author: Miljana Shulajkovska, miljana.sulajkovs[email protected]i. Multimodal deep learning methods process and analyze data from different modalities simultaneously, capturing intricate patterns across tissue, clinical, and genomic data. This integration allows the discovery of novel intermodal relationships, leading to more accurate and comprehensive cancer assessments. Data integration is a critical yet challenging step (7,8). The fusion of diverse modalities is difficult in terms of alignment, interpretability, and explainability. Recent studies have shown promising results for modality fusion in prognosis (9–11) and biomarker prediction (12). The emergence of foundation models marks a significant advancement in this field (13). 2. Workshop objectives and program This workshop aims to explore the transformative impact of AI on healthcare, with a particular focus on CRC. We will showcase recent advancements in multimodal data fusion, combining WSIs with complementary modalities, including clinical and genomic data, and integrating foundation models. We will discuss most widely used CRC datasets for multimodal approaches. Participants will gain insights into various fusion techniques and cutting-edge models applied to CRC diagnosis and prognosis. DL models have proven remarkably effective in analysing WSIs for tasks such as biomarker classification and survival prediction. However, the integration of clinical and genomic data enhances the depth of analysis and provides a more comprehensive understanding of CRC. This workshop will focus on state-of-the-art DL approaches, emphasizing multimodal techniques that fuse clinical, genomic, and histopathological data. We will explore how these diverse data sources are integrated and how such fusion improves diagnostic and prognostic outcomes of patients with cancer. The workshop will span 90 minutes and will be divided into three sessions. • Session 1: Led by Jitendra Jonnagaddala, Senior Research Fellow at the School of Population Health, Faculty of Medicine, University of New South Wales (UNSW), Sydney, Australia. His research focuses on leveraging the secondary use of routinely collected electronic health records (EHRs), with a primary emphasis on the integration of heterogeneous data modalities. In this session, Jitendra will discuss the multimodal analysis of WSIs in CRC. Advances in AI, especially multimodal models, integrate diverse data types to improve CRC diagnosis, prognosis, and treatment. Jitendra will discuss multimodal techniques, their performance, and comparisons with foundation models, along with the challenges in data integration, heterogeneity, generalizability, and interpretability. • Session 2: Led by Miljana Shulajkovska, a Young Researcher at the Jožef Stefan Institute in Ljubljana, Slovenia. Her PhD research focuses on developing multimodal models that integrate diverse data modalities including WSI. She will explore the latest trends in AI for digital pathology, including fusion techniques and the application of foundational models. • Session 3: Led by Assist. Prof. Anton Gradišek, Senior Research Associate at Jožef Stefan Institute, Departments of Intelligent Systems and Solid State Physics. His research focuses on applied AI, particularly in the fields of medicine and biology. He will discuss the explainability and interpretability of AI models in the field of digital pathology. 3. References 1. Ahmedin Jemal FB. Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J Clin. 2024;74(3):229–63. 2. Niazi MKK, Parwani AV, Gurcan MN. Digital pathology and artificial intelligence. Lancet Oncol. 2019 May;20(5):e253–61. 3. Jonnagaddala Jitendra, Croucher Joanne L., Jue Toni Rose, Meagher Nicola S., Caruso Lena, Ward Robyn, et al. Integration and Analysis of Heterogeneous Colorectal Cancer Data for Translational Research. In: Studies in Health Technology and Informatics [Internet]. IOS Press; 2016 [cited 2025 Nov 26]. Available from: https://www.medra.org/servlet/aliasResolver?alias=iospressISBN&isbn=978-1-61499657-6&spage=387&doi=10.3233/978-1-61499-658-3-387 4. Susič D, Syed-Abdul S, Dovgan E, Jonnagaddala J, Gradišek A. Artificial intelligence based personalized predictive survival among colorectal cancer patients. Comput Methods Programs Biomed. 2023 Apr;231:107435. 5. Biomarker Prediction in Colorectal Cancer Using Multiple Instance Learning. In: Proceedings of Slovenian Conference on Artificial Intelligence 2024 [Internet]. Jožef Stefan Instutute; 2024 [cited 2025 Nov 26]. Available from: https://is.ijs.si/?p=16834 6. Unger M, Kather JN. A systematic analysis of deep learning in genomics and histopathology for precision oncology. BMC Med Genomics. 2024 Feb 5;17(1):48. 7. Lipkova J, Chen RJ, Chen B, Lu MY, Barbieri M, Shao D, et al. Artificial intelligence for multimodal data integration in oncology. Cancer Cell. 2022 Oct;40(10):1095–110. 8. Jonnagaddala J, Shulajkovska M, Gradišek A, Jue TR, Zhou Q, Guo Y, et al. Multimodal analysis of whole slide images in colorectal cancer. Npj Digit Med. 2025 Nov 24;8(1):719. 9. Xu Y, Guo J, Yang N, Zhu C, Zheng T, Zhao W, et al. Predicting rectal cancer prognosis from histopathological images and clinical information using multi-modal deep learning. Front Oncol [Internet]. 2024;14. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.085191781850&doi=10.3389%2ffonc.2024.1353446&partnerID=40&md5=413fc6f7d5 c6943bb7b7e4ef4abee46a 10. Lv Z, Lin Y, Yan R, Wang Y, Zhang F. TransSurv: Transformer-Based Survival Analysis Model Integrating Histopathological Images and Genomic Data for Colorectal Cancer. IEEE/ACM Trans Comput Biol Bioinform. 2023 Nov;20(6):3411–20. 11. Zhou J, Foroughi Pour A, Deirawan H, Daaboul F, Aung TN, Beydoun R, et al. Integrative deep learning analysis improves colon adenocarcinoma patient stratification at risk for mortality. eBioMedicine. 2023 Aug;94:104726. 12. Lv Z, Yan R, Lin Y, Gao L, Zhang F, Wang Y. A Disentangled RepresentationBased Multimodal Fusion Framework Integrating Pathomics and Radiomics for KRAS Mutation Detection in Colorectal Cancer. Big Data Min Anal. 2024 Sept;7(3):590–602. 13. Neidlinger P, Nahhas OSME, Muti HS, Lenz T, Hoffmeister M, Brenner H, et al. Benchmarking foundation models as feature extractors for weakly-supervised computational pathology [Internet]. arXiv; 2024 [cited 2025 Mar 18]. Available from: https://arxiv.org/abs/2408.15823