Mul imodal Lea ning App oaches o
Colo ec al Cance
Miljana SHULAJKOVSKAa,
1
, Ji end a JONNAGADDALAb, and An on GRADIŠEKa
a Jože S e an Ins i u e, Ljubljana, Slo enia
b UNSW Sydney, NSW Aus alia
ORCiD ID: Miljana Shulajko ska h ps://o cid.o g/0009-0009-8833-9802
Ji end a Jonnagaddala h ps://o cid.o g/0000-0002-9912-2344
An on G adišek h ps://o cid.o g/0000-0001-6480-9587
Abs ac . The digi aliza ion o issue samples h ough whole slide imaging (WSI),
coupled wi h ad ancemen s in deep lea ning (DL), has opened new possibili ies o
cance diagnosis and p ognosis. The inc easing a ailabili y o complemen a y da a
modali ies, such as genomics and clinical in o ma ion, has d i en he adop ion o
mul imodal app oaches. By simul aneously p ocessing hese di e se da a sou ces,
mul imodal models can unco e complex pa e ns and enhance p edic i e accu acy.
Colo ec al cance (CRC) is one o he leading causes o cance - ela ed mo ali y
wo ldwide. This wo kshop highligh s ecen ad ancemen s in he applica ion o
mul imodal deep lea ning echniques o CRC, in eg a ing WSI wi h o he da a
modali ies o imp o e diagnos ic and p ognos ic capabili ies.
Keywo ds. Colo ec al cance , whole slide images, mul imodal models, deep
lea ning
1. In oduc ion
Colo ec al cance is one o he leading causes o cance - ela ed mo ali y wo ldwide (1).
The digi aliza ion o issue samples in o WSI allows o he cap u e o de ailed
mo phological and his opa hological cha ac e is ics. These digi ized slides p o ide
c i ical insigh s in o cellula s uc u es and he umo mic oen i onmen , which a e
essen ial o accu a e diagnosis and p ognosis.
A pa ien ’s condi ion is e lec ed in mul iple da a modali ies. In addi ion o
issue-based in o ma ion, genomic ea u es, such as bioma ke s, o e aluable insigh s
in o he molecula and gene ic landscape o umo s, which a e c ucial o unde s anding
cance biology and p edic ing he apeu ic esponses. Clinical da a u he en ich his
unde s anding by p o iding pa ien -speci ic in o ma ion, including medical his o y,
demog aphics, and ea men de ails.
A i icial in elligence (AI), especially deep lea ning (DL) and mul imodal
lea ning, has signi ican ly ad anced he ield o digi al pa hology (2), enabling he
analysis o complex pa e ns ha may be challenging o human expe s o disce n.
Recen de elopmen s in DL and AI ha e demons a ed subs an ial po en ial o CRC
diagnosis and (3–5) p ognosis (6). While adi ional app oaches o en ely on single-
modal da a, DL echniques applied o WSI p o ide deepe insigh s. Fu he mo e, he
in eg a ion o addi ional da a sou ces, such as clinical eco ds and genomic p o iles,
enhances he p edic i e powe o hese models.
1
Co esponding Au ho : Miljana Shulajko ska, miljana.sulajko s[email p o ec ed]i.
Mul imodal deep lea ning me hods p ocess and analyze da a om di e en
modali ies simul aneously, cap u ing in ica e pa e ns ac oss issue, clinical, and
genomic da a. This in eg a ion allows he disco e y o no el in e modal ela ionships,
leading o mo e accu a e and comp ehensi e cance assessmen s. Da a in eg a ion is a
c i ical ye challenging s ep (7,8). The usion o di e se modali ies is di icul in e ms
o alignmen , in e p e abili y, and explainabili y. Recen s udies ha e shown p omising
esul s o modali y usion in p ognosis (9–11) and bioma ke p edic ion (12). The
eme gence o ounda ion models ma ks a signi ican ad ancemen in his ield (13).
2. Wo kshop objec i es and p og am
This wo kshop aims o explo e he ans o ma i e impac o AI on heal hca e, wi h a
pa icula ocus on CRC. We will showcase ecen ad ancemen s in mul imodal da a
usion, combining WSIs wi h complemen a y modali ies, including clinical and genomic
da a, and in eg a ing ounda ion models. We will discuss mos widely used CRC da ase s
o mul imodal app oaches. Pa icipan s will gain insigh s in o a ious usion echniques
and cu ing-edge models applied o CRC diagnosis and p ognosis. Fo mo e de ails, you
can ead ou ull blog pos and a icle.
DL models ha e p o en ema kably e ec i e in analysing WSIs o asks such
as bioma ke classi ica ion and su i al p edic ion. Howe e , he in eg a ion o clinical
and genomic da a enhances he dep h o analysis and p o ides a mo e comp ehensi e
unde s anding o CRC. This wo kshop will ocus on s a e-o - he-a DL app oaches,
emphasizing mul imodal echniques ha use clinical, genomic, and his opa hological
da a. We will explo e how hese di e se da a sou ces a e in eg a ed and how such usion
imp o es diagnos ic and p ognos ic ou comes o pa ien s wi h cance . The wo kshop will
span 90 minu es and will be di ided in o h ee sessions.
• Session 1: Led by Ji end a Jonnagaddala, Senio Resea ch Fellow a he
School o Popula ion Heal h, Facul y o Medicine, Uni e si y o New
Sou h Wales (UNSW), Sydney, Aus alia. His esea ch ocuses on
le e aging he seconda y use o ou inely collec ed elec onic heal h
eco ds (EHRs), wi h a p ima y emphasis on he in eg a ion o
he e ogeneous da a modali ies. In his session, Ji end a will discuss he
mul imodal analysis o WSIs in CRC. Ad ances in AI, especially
mul imodal models, in eg a e di e se da a ypes o imp o e CRC diagnosis,
p ognosis, and ea men . Ji end a will discuss mul imodal echniques, hei
pe o mance, and compa isons wi h ounda ion models, along wi h he
challenges in da a in eg a ion, he e ogenei y, gene alizabili y, and
in e p e abili y.
• Session 2: Led by Miljana Shulajko ska, a Young Resea che a he Jože
S e an Ins i u e in Ljubljana, Slo enia. He PhD esea ch ocuses on
de eloping mul imodal models ha in eg a e di e se da a modali ies
including WSI. She will explo e he la es ends in AI o digi al pa hology,
including usion echniques and he applica ion o ounda ional models.
• Session 3: Led by Assis . P o . An on G adišek, Senio Resea ch Associa e
a Jože S e an Ins i u e, Depa men s o In elligen Sys ems and Solid S a e
Physics. His esea ch ocuses on applied AI, pa icula ly in he ields o
medicine and biology. He will discuss he explainabili y and
in e p e abili y o AI models in he ield o digi al pa hology.
3. Re e ences
1. Ahmedin Jemal FB. Global cance s a is ics 2022: GLOBOCAN es ima es o
incidence and mo ali y wo ldwide o 36 cance s in 185 coun ies. CA Cance J Clin.
2024;74(3):229–63.
2. Niazi MKK, Pa wani AV, Gu can MN. Digi al pa hology and a i icial
in elligence. Lance Oncol. 2019 May;20(5):e253–61.
3. Jonnagaddala Ji end a, C ouche Joanne L., Jue Toni Rose, Meaghe Nicola S.,
Ca uso Lena, Wa d Robyn, e al. In eg a ion and Analysis o He e ogeneous Colo ec al
Cance Da a o T ansla ional Resea ch. In: S udies in Heal h Technology and
In o ma ics [In e ne ]. IOS P ess; 2016 [ci ed 2025 No 26]. A ailable om:
h ps://www.med a.o g/se le /aliasResol e ?alias=iosp essISBN&isbn=978-1-61499-
657-6&spage=387&doi=10.3233/978-1-61499-658-3-387
4. Susič D, Syed-Abdul S, Do gan E, Jonnagaddala J, G adišek A. A i icial
in elligence based pe sonalized p edic i e su i al among colo ec al cance pa ien s.
Compu Me hods P og ams Biomed. 2023 Ap ;231:107435.
5. Bioma ke P edic ion in Colo ec al Cance Using Mul iple Ins ance Lea ning.
In: P oceedings o Slo enian Con e ence on A i icial In elligence 2024 [In e ne ]. Jože
S e an Ins u u e; 2024 [ci ed 2025 No 26]. A ailable om: h ps://is.ijs.si/?p=16834
6. Unge M, Ka he JN. A sys ema ic analysis o deep lea ning in genomics and
his opa hology o p ecision oncology. BMC Med Genomics. 2024 Feb 5;17(1):48.
7. Lipko a J, Chen RJ, Chen B, Lu MY, Ba bie i M, Shao D, e al. A i icial
in elligence o mul imodal da a in eg a ion in oncology. Cance Cell. 2022
Oc ;40(10):1095–110.
8. Jonnagaddala J, Shulajko ska M, G adišek A, Jue TR, Zhou Q, Guo Y, e al.
Mul imodal analysis o whole slide images in colo ec al cance . Npj Digi Med. 2025
No 24;8(1):719.
9. Xu Y, Guo J, Yang N, Zhu C, Zheng T, Zhao W, e al. P edic ing ec al cance
p ognosis om his opa hological images and clinical in o ma ion using mul i-modal
deep lea ning. F on Oncol [In e ne ]. 2024;14. A ailable om:
h ps://www.scopus.com/inwa d/ eco d.u i?eid=2-s2.0-
85191781850&doi=10.3389%2 onc.2024.1353446&pa ne ID=40&md5=413 c6 7d5
c6943bb7b7e4e 4abee46a
10. L Z, Lin Y, Yan R, Wang Y, Zhang F. T ansSu : T ans o me -Based Su i al
Analysis Model In eg a ing His opa hological Images and Genomic Da a o Colo ec al
Cance . IEEE/ACM T ans Compu Biol Bioin o m. 2023 No ;20(6):3411–20.
11. Zhou J, Fo oughi Pou A, Dei awan H, Daaboul F, Aung TN, Beydoun R, e al.
In eg a i e deep lea ning analysis imp o es colon adenoca cinoma pa ien s a i ica ion
a isk o mo ali y. eBioMedicine. 2023 Aug;94:104726.
12. L Z, Yan R, Lin Y, Gao L, Zhang F, Wang Y. A Disen angled Rep esen a ion-
Based Mul imodal Fusion F amewo k In eg a ing Pa homics and Radiomics o KRAS
Mu a ion De ec ion in Colo ec al Cance . Big Da a Min Anal. 2024 Sep ;7(3):590–602.
13. Neidlinge P, Nahhas OSME, Mu i HS, Lenz T, Ho meis e M, B enne H, e
al. Benchma king ounda ion models as ea u e ex ac o s o weakly-supe ised
compu a ional pa hology [In e ne ]. a Xi ; 2024 [ci ed 2025 Ma 18]. A ailable om:
h ps://a xi .o g/abs/2408.15823
