PhD P og am in A ificial In elligence
ADVANCES TOWARDS A ROBUST
AI-BASED GLIOMA GRADING SYSTEM
Ca la Pi a ch Abaiga
Thesis Supe iso s:
P o . Al edo Vellido Alcacena
D . Vicen Ribas Ripoll
Sep embe 2024
ABSTRACT
Gliomas ep esen he mos common and agg essi e o m o p ima y b ain umo s in adul s,
posing signi ican diagnos ic challenges due o hei he e ogeneous molecula , his ological,
and adiological cha ac e is ics. The Wo ld Heal h O ganiza ion (WHO) classi ies gliomas
on a ou -s age scale, anging om he mos benign o he mos malignan . The la es WHO
Cen al Ne ous Sys em (CNS) diagnos ic c i e ia eleases inco po a e molecula and
his ological ea u es o sub yping and g ading. Howe e , his p ocess is complex,
ime-consuming, and p one o in e -obse e a iabili y. While his ological assessmen
emains he s anda d me hod o glioma cha ac e iza ion, i is in asi e and limi ed in cases
whe e issue ex ac ion is no easible, highligh ing he need o non-in asi e al e na i es.
This disse a ion in es iga es he applica ion o deep lea ning algo i hms o mul i-sequence
Magne ic Resonance Imaging (MRI) o he au oma ic g ading o gliomas. Fi s , we
in eg a ed and homogenized MRI da a om h ee well-known publicly a ailable da ase s
esul ing in a collec i e da abase exceeding one housand cases. Ini ial e o s ocused on
unde s anding he mul i-sequence MRI da a, assessing a i ac s, and e alua ing echniques
such as da a no maliza ion, da a augmen a ion, ans e lea ning, and umo pa ching, o
enhance he classi ica ion be ween lowe - and high-g ade gliomas, benchma king wi h he
s a e-o - he-a . In he second phase, we ex end he classi ica ion o glioma g ades as de ined
by he WHO CNS 2016 c i e ia. We de eloped and compa ed di e en Con olu ional
Neu al Ne wo ks (CNNs) a chi ec u es based on 2D slices om single ana omical planes,
mul i-plana in eg a ions, and en i e 3D olumes. Recognizing he s ong associa ion
be ween glioma agg essi eness and gene ic ea u es like IDH mu a ions and 1p/19q
co-dele ions, we de eloped a mul i- ask amewo k capable o classi ying he glioma g ade
and hese molecula ea u es simul aneously, allowing he ne wo k o lea n con ex ual
ela ionships among a iables. To p e en heal hca e dispa i ies and os e he eliabili y and
anspa ency o ou indings, we e alua ed model pe o mance ac oss di e en demog aphic
and clinical popula ions, including sex, age, and molecula subg oups.
Ou esul s e eal consis en challenges in accu a ely classi ying g ade 3 gliomas,
unde sco ing he need o u he esea ch, as well as mo e di e se and up- o-da e da a ha
aligns wi h cu en guidelines. While A i icial In elligence (AI) holds immense po en ial o
assis in medical diagnosis, p ognosis, and he apy planning, i s ull adop ion in he medical
ield emains limi ed la gely due o a lack o us and widesp ead accep ance among
heal hca e p o essionals and pa ien s. The ul ima e goal o his hesis is o explo e he
implica ions and limi a ions o in eg a ing AI algo i hms in heal hca e and os e
collabo a ion among clinicians and echnologis s o achie e pa ien -cen ic, eliable AI
wi hin clinical p ac ice.
iii
RESUM
Els gliomes ep esen en la o ma m´
es comuna i ag essi a de umo s ce eb als p ima is en
adul s, p esen an un desa iamen diagn`
os ic signi ica iu a causa de la se a he e ogene¨
ı a
molecula , his ol `
ogica i adiol`
ogica. L’O gani zaci´
o Mundial de la Salu (OMS) classi ica els
gliomes en una escala de qua e es adis, des de benignes ins a malignes. Les ´
ul imes edicions
dels c i e is diagn `
os ics del Sis ema Ne i´
os Cen al (SNC) de l’OMS inco po en
ca ac e ´
ıs iques molecula s i his ol `
ogiques pe a la sub ipi icaci ´
o i g adaci´
o. Tanma eix,
aques p oc´
es ´
es complex, eque eix mol de emps i ´
es p opens a la a iabili a en e
obse ado s. Enca a que l’a aluaci ´
o his ol `
ogica con inua sen el m`
e ode es `
anda d pe a la
ca ac e i zaci ´
o dels gliomes, ´
es in asi a i es `
a limi ada en casos on no ´
es ac ible l’ex acci ´
o de
eixi , cosa que essal a la necessi a d’al e na i es no in asi es.
Aques eball in es iga l’aplicaci ´
o d’algo i mes d’ap enen a ge p o und en ima ges de
esson`
ancia magn`
e ica (IRM) mul isecuencia pe a la classi icaci ´
o au om`
a ica dels gliomes.
P ime , am in eg a i homogene¨
ı za les IRM de es bases de dades p ´
ubliques econegudes,
esul an en una base de dades col·lec i a que supe a els mil casos. Els es o c¸os inicials es an
cen a en comp end e les dades d’IRM mul isecuencia, a alua a e ac es i alo a `
ecniques
com la no mali zaci´
o de dades, l’augmen de dades, l’ap enen a ge pe ans e `
encia i
l’ex acci´
o de umo s, pe millo a la classi icaci´
o en e gliomes de g au baix i al ,
compa an -los amb l’es a de l’a . En la segona ase, am amplia la classi icaci´
o als g aus de
gliomes segons els c i e is del SNC de l’OMS es able s el 2016. Vam desen olupa i
compa a di e en s a qui ec u es de xa xes neu onals con olucionals basades en alls 2D de
plans ana `
omics indi iduals, in eg acions mul iplana es i olums 3D. Reconeguen
l’associaci´
o o a en e l’ag essi i a dels gliomes i ca ac e ´
ıs iques gen`
e iques com les
mu acions d’IDH i la codeleci´
o 1p/19q, am desen olupa una a qui ec u a mul i asca
capac¸ de classi ica simul `
aniamen el g au del glioma i aques es ca ac e ´
ıs iques molecula s,
pe me en que la xa xa ap engui elacions con ex uals en e les a iables. Pe p e eni
dispa i a s en el con ex cl´
ınic i omen a la iabili a i anspa `
encia dels nos es esul a s,
am a alua el endimen del model en di e en s poblacions demog `
a iques i cl´
ıniques,
incloen sexe, eda i subg ups molecula s.
Els nos es esul a s e elen ep es consis en s en la classi icaci ´
o p ecisa dels gliomes de g au 3,
cosa que essal a la necessi a de m´
es in es igaci ´
o, aix´
ı com de dades m´
es di e ses i ac uali zades
que s’aline¨
ın amb les guies ac uals. Enca a que la In el.lig`
encia A i icial (IA) posseeix un
immens po encial pe assis i en el diagn`
os ic m`
edic, el p on`
os ic i la plani icaci´
o e ap`
eu ica,
la se a adopci´
o comple a en el camp m`
edic con inua sen limi ada en g an mesu a a causa
de la manca de con ianc¸a i d’accep aci ´
o gene ali zada en e els p o essionals de la salu i els
pacien s. L’objec iu inald’aques a esi ´
esexplo a lesimplicacions ilimi acionsde lain eg aci ´
o
d’algo i mes de IA en el con ex m`
edic i omen a la col·labo aci ´
o en e cl´
ınics i ecn `
olegs pe
aconsegui una IA iable i cen ada en el pacien dins de la p `
ac ica cl´
ınica.
RESUMEN
Los gliomas ep esen an la o ma m´
as com ´
un y ag esi a de umo es ce eb ales p ima ios en
adul os, p esen ando un signi ica i o desa ´
ıo diagn´
os ico po su he e ogeneidad molecula ,
his ol ´
ogica y adiol´
ogica. La O ganizaci ´
on Mundial de la Salud (OMS) clasi ica los gliomas
en una escala de cua o es adios, de benignos a malignos. Las ´
ul imas ediciones de los
c i e ios diagn´
os icos del Sis ema Ne ioso Cen al (SNC) de la OMS inco po an
ca ac e ´
ıs icas molecula es e his ol´
ogicas pa a la sub ipi icaci ´
on y g adaci´
on. Sin emba go,
es e p oceso es complejo, equie e mucho iempo y es p openso a a iabilidad en e
obse ado es. Aunque la e aluaci ´
on his ol ´
ogica sigue siendo el m´
e odo es ´
anda pa a la
ca ac e izaci ´
on de gliomas, es in asi a y es ´
a limi ada en casos donde no es ac ible la
ex acci´
on de ejido, lo que esal a la necesidad de al e na i as no in asi as.
Es e abajo in es iga la aplicaci ´
on de algo i mos de ap endizaje p o undo en im´
agenes de
esonancia magn´
e ica (IRM) mul i-secuencia pa a la clasi icaci´
on au om´
a ica de gliomas.
P ime o, in eg amos y homogeneizamos IRM de es econocidas bases de da os p ´
ublicas,
esul ando en una base de da os colec i a que supe a los mil casos. Es ue zos iniciales se
cen a on en comp ende los da os de IRM mul isecuencia, e alua a e ac os y e alua
´
ecnicas como la no malizaci´
on de da os, aumen o de da os, el ap endizaje po ans e encia
y la ex acci´
on de umo es, pa a mejo a la clasi icaci´
on en e gliomas de g ado bajo y al o,
compa ´
andolos con el es ado del a e. En la segunda ase, ampliamos la clasi icaci´
on a los
g ados de gliomas seg ´
un los c i e ios del SNC de la OMS es ablecidos en 2016.
Desa ollamos y compa amos di e en es a qui ec u as de edes neu onales con olucionales
basadas en co es 2D de planos ana ´
omicos indi iduales, in eg aciones mul iplano y
ol ´
umenes 3D. Reconociendo la ue e asociaci ´
on en e la ag esi idad de los gliomas y
ca ac e ´
ıs icas gen´
e icas como las mu aciones de IDH y la codeleci´
on 1p/19q, desa ollamos
una a qui ec u a mul i a ea capaz de clasi ica simul ´
aneamen e el g ado del glioma y es as
ca ac e ´
ıs icas molecula es, pe mi iendo que la ed ap enda elaciones con ex uales en e las
a iables. Pa a p e eni dispa idades en el con ex o cl´
ınico y omen a la iabilidad y
anspa encia de nues os hallazgos, e aluamos el endimien o del modelo en di e en es
poblaciones demog ´
a icas y cl´
ınicas, incluyendo sexo, edad y subg upos molecula es.
Los esul ados e elan desa ´
ıos consis en es en la clasi icaci´
on p ecisa de los gliomas de g ado
3 que esal an la necesidad de m´
as in es igaci ´
on, as´
ı como de da os m´
as di e sos y ac ualizados
que se alineen con las gu´
ıas ac uales. Aunque la In eligencia A i icial (IA) posee un inmenso
po encial pa a asis i en el diagn´
os ico m´
edico, el p on ´
os ico y la plani icaci ´
on e ap´
eu ica,
su adopci´
on comple a en el campo m´
edico sigue siendo limi ada en g an medida debido a la
al a de con ianza y acep aci ´
on gene alizada en e los p o esionales de la salud y los pacien es.
El obje i o inal de es a esis es explo a las implicaciones y limi aciones de la in eg aci´
on de
algo i mos de IA en el con ex o m´
edico y omen a la colabo aci ´
on en e cl´
ınicos y ecn ´
ologos
pa a log a una IA con iable y cen ada en el pacien e den o de la p ´
ac ica cl´
ınica.
ii
PRACTICE 075
4
THE DATA 081
4.1 Desc ip ion ............................083
4.2 Explo a o y da a analysis ....................087
4.2.1 MRI da a ........................087
4.2.2 Demog aphic & clinical da a .............092
Key Takeaways ..............................096
5
PRELUDE:
A CLASSIFICATION OF
LOWER AND HIGH GRADES 101
5.1 P eamble .............................103
5.2 Da a ................................104
5.3 Me hods ..............................106
5.3.1 CNN a chi ec u es ...................106
5.3.2 Model aining .....................107
5.4 Resul s ...............................108
5.4.1 Ana omical plane selec ion .............108
5.4.2 P e-p ocessing assessmen .............109
5.4.3 E alua ing ne wo k a chi ec u es ..........116
5.4.4 E alua ing he sample size ..............117
Key Takeaways ..............................118
6
ON THE WHO GLIOMA
GRADE CLASSIFICATION 123
6.1 P eamble .............................125
6.2 Da a ................................127
6.3 Me hods ..............................128
6.3.1 The 2D single-plana classi ie ...........128
6.3.2 The 2.5D mul i-plana classi ie ...........128
6.3.3 The 3D olume ic classi ie .............129
6.3.4 The mul i- ask lea ning amewo k .........130
6.4 Model aining ...........................131
6.5 Resul s ...............................131
6.5.1 Glioma g ade classi ica ion pe o mance .....131
6.5.2 Combining g ade and molecula ea u es .....137
6.5.3 E alua ing he classi ie dispa i ies .........139
Key Takeaways ..............................142
CLOSING 147
DISCUSSION 151
FUTURE DIRECTIONS 156
LIST OF PUBLICATIONS 158
BIBLIOGRAPHY 163
APPENDICES 203
A. Li e a u e e iew .........................203
B. Supplemen a y esul s .....................210
C. Models’ hype pa ame e s ....................226
LIST OF FIGURES
Fig. 0.1 Gene al hesis ou line, including i s pa s and chap e s. 009
Fig. 1.1 Flow diag am ha illus a es he e olu ion o he
WHO 2016 classi ica ion in o he WHO 2021
classi ica ion o gliomas .................. 025
Fig. 1.2 Pa ches ex ac ed om glioblas oma ( i s ow),
as ocy oma (second ow), and oligodend oglioma
( hi d ow) WSI images. ................... 027
Fig. 1.3 MRI scanne (le ) and a adiologis in e p e ing a
b ain MRI ( igh ) ....................... 028
Fig. 1.4 Sagi al, co onal, and axial iews o an MRI scan. ... 029
Fig. 1.5 Con en ional sequences o an MRI scan in axial iew. 030
Fig. 2.1 Illus a ion o in e - and in a-subjec s MRI egis a ion. 036
Fig. 2.2 Raw ( op) and bias ield co ec ed (bo om) images.
A colo map has been applied o acili a e he
isualiza ion o issue di e ences a e BFC. ...... 038
Fig. 2.3 Raw ( op) and skull-s ipped (bo om) images. ..... 039
Fig. 2.4 Raw ( op) and equalized (bo om) images. ....... 040
Fig. 2.5 Visualiza ion o 3D b ain MRI olume. .......... 042
x ii
Fig. 2.6 Visualiza ion o en (a) sagi al, (b) co onal, and (c)
axial 2D slices. ........................ 042
Fig. 2.7 A chi ec u e o a 2D anilla CNN wi h wo
con olu ional laye s, wo pooling laye s, one
la ened laye , and one ully connec ed laye . ..... 048
Fig. 2.8 K- old C oss-Valida ion scheme. ............. 050
Fig. 2.9 Da a in eg a ion s a egies. ................ 051
Fig. 2.10 Mul i- ask lea ning. ..................... 052
Fig. 3.1 Publicly a ailable da ase s usage equency om
2018 o 2024. ........................ 064
Fig. 4.1 Flow diag am o pa icipan numbe s ini ially and
pos -quali y assessmen o MRI modali ies,
alongside he implemen ed da a spli s a egy. ..... 084
Fig. 4.2 Pixel dis ibu ion o EGD lai MRIs ac oss imaging
acquisi ion scans. ...................... 087
Fig. 4.3 His og ams o pixel alues ex ac ed om sagi al 2D
MRIs con aining he la ges a ea o umo . ....... 089
Fig. 4.4 His og ams o pixel alues ex ac ed om co onal 2D
MRIs wi h la ges a ea o umo . ............. 090
Fig. 4.5 His og ams o pixel alues ex ac ed om axial 2D
MRIs con aining he la ges a ea o umo . ....... 091
Fig. 4.6 His og ams o 2D axial slices using (a) min-max
no maliza ion, (b) s anda diza ion on comple e
image egion, and (c) s anda diza ion on he b ain
egion. ............................. 092
x iii
Fig. 4.7 Dis ibu ion o wo-class umo g ade by (a) sex, (b)
age, (c) IDH, and (d) 1p/19q co-dele ion s a us. .... 093
Fig. 4.8 Dis ibu ion o umo g ade by (a) sex, (b) age, (c)
IDH, and (d) 1p/19q co-dele ion s a us. ......... 094
Fig. 4.9 Dis ibu ion o IDH and 1p/19q co-dele ion s a us by
glioma g ade. ......................... 095
Fig. 5.1 2D MRI slices con aining he la ges umo a ea om
(a) sagi al, (b) co onal, and (c) axial iews. ....... 104
Fig. 5.2 Example o he selec ed 10 con iguous slices bo h
be o e and a e he slice con aining he maximum
umo a ea wi h a skip o 2 o (a) LGG and a skip o
5 slices o (b) HGG pa ien s. ............... 105
Fig. 5.3 P oposed ResNe a chi ec u e based on 18 and 34
laye s. ............................. 106
Fig. 5.4 P oposed VGGNe a chi ec u e based on 11 and 16
laye s. ............................. 107
Fig. 5.5 Classi ica ion pe o mance o LGG and HGG ac oss
ana omical planes. ..................... 108
Fig. 5.6 (a) O e all AUC, (b) LGG ecall, and (c) HGG ecall,
using indi idual MRI sequences and hei usion
ac oss a ious p e-p ocessing echniques. ....... 111
Fig. 5.7 G adCAM a en ion maps ac oss di e en
p e-p ocessing and sequences. .............. 113
Fig. 5.8 Model’s ou pu p obabili y dis ibu ion ac oss
independen sequences and hei usion, using
mean-s d s anda dized images wi hin he b ain egion. 114
Fig. 5.9 Mul i-modal model’s ou pu p obabili y dis ibu ion
inco po a ing da a augmen a ion. ............. 115
xix
Fig. 6.1 2D MRI slices con aining he la ges umo a ea
c opped o he umo ROI om (a) sagi al, (b)
co onal, and (c) axial iews. ................ 127
Fig. 6.2 P oposed mul i-plana a chi ec u e using
in e media e agg ega ion o in eg a e he ou pu
ea u es o each plane-speci ic model. .......... 129
Fig. 6.3 P oposed 3D model o olume ic da a. ......... 130
Fig. 6.4 Recall o glioma g ades 2, 3, and 4 ac oss 2D, 2.5D,
and 3D model s a egies. .................. 134
Fig. 6.5 Con usion ma ices ob ained using he mul i-plana
model wi h in e media e ea u e conca ena ion. .... 135
Fig. 6.6 Ou pu p obabili y dis ibu ion ac oss g ades using
he mul i-plana model wi h in e media e ea u e
conca ena ion. ........................ 135
Fig. 6.7 Mean 3- old CV ecall on es se ac oss
demog aphic and clinical popula ions. .......... 140
Fig. A.1 Con usion ma ices using he mul i-plana model o
he emale g oup. ...................... 223
Fig. A.2 Con usion ma ices using he mul i-plana model o
he male g oup. ....................... 223
Fig. A.3 Con usion ma ices using he mul i-plana model o
he age <40 g oup. ..................... 223
Fig. A.4 Con usion ma ices using he mul i-plana model o
he age 40 −59 g oup. ................... 224
Fig. A.5 Con usion ma ices using he mul i-plana model o
he age ≥60 g oup. ..................... 224
xx
Fig. A.6 Con usion ma ices using he mul i-plana model o
he IDH wild- ype g oup. .................. 224
Fig. A.7 Con usion ma ices using he mul iplana model o
he IDH mu a ed g oup. ................... 225
Fig. A.8 Con usion ma ices using he mul i-plana model o
he 1p/19q in ac g oup. .................. 225
Fig. A.9 Con usion ma ices using he mul i-plana model o
he 1p/19q co-dele ed g oup. ............... 225
xxi
LIST OF TABLES
Table 3.1 An o e iew o publicly a ailable MRI da ase s o
b ain umo classi ica ion benchma king. .........062
Table 4.1 G ade dis ibu ion ac oss da ase s. .............084
Table 4.2 Demog aphic cha ac e is ics o e he di e en da a
pa i ions. ............................086
Table 4.3 F equency o imaging acquisi ion scanne
manu ac u e s in EGD da ase . ...............087
Table 4.4 S a is ics compu ed om sagi al 2D MRIs
con aining he la ges a ea o umo . ............089
Table 4.5 S a is ics compu ed om co onal 2D MRIs
con aining he la ges a ea o umo . ............090
Table 4.6 S a is ics compu ed om axial 2D MRIs con aining
he la ges a ea o umo . ...................091
Table 4.7 Dis ibu ion o LGG and HGG by clinical ea u es. ....093
Table 4.8 Dis ibu ion o umo g ade by clinical ea u es. ......094
Table 5.1 Mean 3- old CV 2D classi ica ion pe o mance on
he es se using ResNe -18 o dis inguish be ween
LGG and HGG, e alua ed ac oss he h ee
ana omical planes using he la ges umo ous slice. ...109
xxii
Table 5.2 Mean 3- old CV pe o mance on he es se o he
classi ica ion o LGG s. HGG ob ained using he
umo ROI, mul i-slice o e sampling, and ine- uning
om ImageNe . ........................115
Table 5.3 Mean 3- old CV pe o mance on he es o he
classi ica ion o LGG s. HGG using ResNe -18,
ResNe -34, VGGNe -11, and VGGNe -16. ........116
Table 5.4 Compa ison o model pa ame e s and aining ime
o ResNe -18, ResNe -34, VGGNe -11, and
VGGNe -16. ..........................117
Table 5.5 Mean 3- old CV pe o mance on he es se o he
classi ica ion o LGG s. HGG conside ing di e en
sample sizes. ..........................117
Table 6.1 Mean 3- old CV pe o mance on he es se o
classi y he WHO glioma g ade using 2D
single-plana , 2.5D mul i-plana , and 3D olume models. 133
Table 6.2 Mean 3- old CV pe o mance on he es se o
pai wise g ade compa isons using he in e media e
usion mul i-plana a chi ec u e. ...............136
Table 6.3 Mean 3- old CV pe o mance on he es se o he
classi ica ion o IDH mu a ion s a us and 1p/19q co-
dele ion s a us using he mul i-plana a chi ec u e. ....137
Table 6.4 Mean 3- old CV pe o mance on he es se o he
classi ica ion o he WHO glioma g ades, he IDH
mu a ion s a us, and he 1p/19q co-dele ion s a us
using he mul i- ask amewo k. ...............138
Table 6.5 Mean 3- old CV pe o mance on he es se o he
classi ica ion o he WHO glioma g ades ac oss
demog aphic g oups using he mul i-plana a chi ec u e. 141
xxiii
INTRODUCTION
CLINICAL CONTEXT
In ecen decades, he p e alence o b ain cance in he adul popula ion has inc eased by
app oxima ely 40% [1]. The la es da a a ailable om Global Cance Obse a o y [2] e eals
ha in 2022, b ain umo s we e diagnosed in 321 731 indi iduals globally. This ep esen s
an incidence a e o 4.1 pe 100 000 people, accoun ing o 1.6% o all new cance cases
diagnosed. Fu he mo e, b ain cance was esponsible o 248 500 dea hs, co esponding o
a c ude mo ali y a e o 3.2 pe 100 000 pe son-yea s.
Gliomas ep esen he mos p e alen o m o malignan p ima y b ain umo s, cons i u ing
he 75% [3]. Managing hese umo s is pa icula ly challenging due o hei he e ogenei y,
in il a i e na u e, and a iable esponse o ea men [4]. No ably, Glioblas oma
Mul i o me (GBM), ecognized as one o he mos agg essi e o ms o cance , accoun s o
55% o all gliomas and is o en accompanied by a poo p ognosis, wi h only 5.5% o he
pa ien s es ima ed o achie e a 5-yea ela i e su i al [5]. Glioblas oma, occu ying a a a e
o 3 cases pe 100 000 pe son-yea s [6], is a a e disease, making he collec ion o la ge and
di e se da ase s o la ge-scale popula ion esea ch challenging.
Acco ding o he Wo ld Heal h O ganiza ion (WHO) sys em, gliomas a e g aded on a scale
anging om 1 (mos benign) o 4 (mos malignan ). G ade 1 umo s a e conside ed
non-cance ous and a e a e in adul s. In he ou h edi ion o he WHO Classi ica ion o
Tumo s o he Cen al Ne ous Sys em (CNS) published in 2016, molecula pa ame e s
we e included along wi h his ological ea u es as decisi e ma ke s o glioma classi ica ion,
mos ly based on he p esence o Isoci a e Dehyd ogenase (IDH) mu a ion and co-dele ion
o ch omosome a ms 1p/19q [7]. The 2021 i h edi ion [8] in oduced signi ican changes
003
ha ad ance he ole o molecula diagnos ics in CNS umo classi ica ion. Inconsis encies
a ise in how ce ain c i e ia a e in e p e ed, which can di e among clinicians, consequen ly
esul ing in a misg ading a e o up o 30% [9]. Al hough imaging ea u es a e no included
in he WHO classi ica ion, hey s ill can se e as a powe ul ool o impac clinical p ac ice.
Be o e issue con i ma ion and also in a e cases when issue con i ma ion is no possible,
imaging ea u es a e he key in o ma ion ha d i es he clinical decision [10].
The ea men o a glioma depends on i s g ade and may include obse a ion, su ge y,
adia ion he apy, and chemo he apy [11]. The g ade o gliomas is signi ican in de e mining
hei ea men since high g ades a e ea ed in mo e agg essi e ways, o en wi h adia ion o
chemo he apy. Accu a e pa hological classi ica ion is essen ial o de e mining ea men
and p ognosis p edic ion. Se e al s udies highligh signi ican a ia ions among obse e s in
he his ological analysis o gliomas, a ec ing umo yping and g ading [12]. Such
disc epancies can esul in inapp op ia e ea men decisions, isking bo h o e - and
unde - ea men , especially in de e mining adio he apy and chemo he apy in e en ions.
The di e en ia ion be ween g ades 2 and 3 is pa icula ly challenging due o he subjec i e
na u e o he WHO glioma classi ica ion s anda ds, which ely on mo phological
desc ip ions wi h in e p e i e bounda ies be ween di e en umo g ades, wi h e ms like
“mode a ely” o “inc eased” cellula i y.
Imaging plays a undamen al ole in he managemen o gliomas om he diagnosis o he
pos - ea men ollow-up, which a ies acco ding o he his ological g ade, loca ion,
esec abili y o he umo , and pe o mance s a us o he pa ien [13]. Rega ding he
pos - ea men ollow-up, he gold s anda d o assessing ea men esponse is p o ided by
he Response Assessmen in Neu o-Oncology Wo king G oup (RANO) [14] c i e ia,
which in eg a es bo h imaging ea u es and clinical ac o s. Recen ly, he in oduc ion o
RANO 2.0 c i e ia [15], ep esen s a no able e inemen dis inguishing be ween low-g ade
and high-g ade gliomas. No ably, RANO 2.0 also conside s he IDH mu a ion s a us o
de e mine he assessmen o he su ounding non-enhancing egion.
The cu en gold-s anda d p ocedu e o glioma g ading in ol es su ge y and his ological
e alua ion o he umo , which play a pi o al ole in bo h diagnosis and p ognosis.
None heless, due o he in asi e and ime-consuming na u e o his me hod, he e is a
g owing in e es in explo ing non-in asi e and p e-ope a i e p ocedu es o cha ac e ize he
umo , accele a e he diagnosis, and plan pe sonalized ea men s.
004
METHODOLOGICAL CONTEXT
AI in Heal hca e In ecen decades, signi ican s ides ha e been made in de eloping
obus A i icial In elligence (AI) based models capable o add essing in ica e challenges
such as image classi ica ion [16–19], objec de ec ion [20–24], and image segmen a ion
[25–27]. Wi h he ma u a ion o Machine Lea ning (ML) echniques in ecen decades,
he e has been a no able su ge in in e es ega ding he de elopmen o au oma ed sys ems
o medical diagnosis. The eme gence o Deep Lea ning (DL) has b ough abou new
possibili ies in he a ea o medical imaging analysis and diagnosis. One o i s a guably mos
success ul models is Con olu ional Neu al Ne wo k (CNN), a widely used ype o DL
algo i hm, well known o i s abili y o cap u e spa ial co ela ions wi hin image pixel da a
hie a chically. DL models a e da a-hung y, which means ha he ull po en ial o a model is
o en inhibi ed by a lack o da a. This issue is pa icula ly p onounced in he medical
con ex , whe e da a acquisi ion is cos ly and dependen on medical expe ise. The
a ailabili y o good-quali y da ase s ha a e clinically ele an is one o he key challenges
ha de elope s ace. The lack o good-quali y da ase s o use in he de elopmen o AI
sys ems may hinde hei e ec i eness and po en ial bene i s. Da a ha a e no o su icien
quali y o he in ended pu pose can also lead o many p oblems, such as bias and e o s
[28]. E en hough AI-based models showed p omising esul s, one o he limi a ions in he
cu en in eg a ion o AI in o clinical p ac ice is he di icul y in unde s anding hei
in e nal mechanisms and explaining he a ionale behind hei decisions. Fo AI-based
applica ions o be e ec i ely inco po a ed in o he clinical ou ine, i is essen ial o
comp ehend and in e p e hei ou pu s. AI sys ems should ideally be used o complemen
human in elligence capabili ies and decision-making, in o de o educe e o s in diagnos ic
classi ica ion o p ognosis. They ha e shown p omise in medical imaging asks [29–31],
enabling imp o ed umo de ec ion, classi ica ion, and p ognosis assessmen . In ecen
yea s, he applica ion o hese cu ing-edge me hods o b ain umo assessmen has also seen
a ma ked inc ease in in e es [32–36], u he showcasing he po en ial o DL in
e olu ionizing he ield o neu o-oncology.
E hics o AI in Heal hca e In ecen yea s, Eu ope has seen he eme gence o egula ions
aimed a add essing he complexi ies o he digi al age. The Gene al Da a P o ec ion
Regula ion (GDPR) se he s age in 2018 by es ablishing s ic guidelines o da a p i acy
and p o ec ion. In 2021, he Uni ed Na ions Educa ional, Scien i ic and Cul u al
O ganiza ion (UNESCO) elabo a ed he Recommenda ion on he E hics o AI, which calls
o legal amewo ks o go e n hese echnologies and ensu e hey con ibu e o he public
good, emphasizing anspa en , unbiased, inclusi e, and ai solu ions. Simila ly, he WHO
eleased he guidance on E hics and Go e nance o AI o Heal h [37], which emphasizes
he need o AI e hical egula ions, especially in he heal hca e domain. This guidance
endo ses a se o key e hical p inciples: (1) P o ec ing human au onomy in medical
decisions; (2) P omo ing human well-being and sa e y and he public in e es ; (3) Ensu ing
anspa ency, explainabili y and in elligibili y by de elope s, medical p o essionals, pa ien s,
005
use s, and egula o s; (4) Fos e ing esponsibili y and accoun abili y; (5) Ensu ing
inclusi eness and equi y i espec i e o sex, gende , income, ace, e hnici y, sexual
o ien a ion, among o he s; (6) P omo ing esponsi e and sus ainable AI. The WHO
guidance has been u he ex ended ecen ly in 2024 o include la ge mul imodal models. In
esponse o he g owing need o e hical and secu e AI, he Eu opean Commission has
ecen ly eleased he AI Ac [38,39] as he i s legal amewo k dedica ed o AI ha aims o
mi iga e he isks associa ed wi h AI echnologies while es ablishing measu es o p omo e
he de elopmen o us wo hy AI.
Mul imodal Da a In eg a ion The in eg a ion o di e se da a ypes and modali ies
p esen s oppo uni ies o enhance he obus ness and p ecision o diagnos ic and p ognos ic
models, he eby b idging he gap be ween AI ad ancemen s and clinical applica ion. In
ecen yea s, AI models ha e exhibi ed imp essi e e icacy ac oss nume ous clinically ele an
asks, e en hose ha may pose challenges o human in e p e a ion. These models can d aw
upon a ange o da a sou ces, such as adiology, pa hology, genomics, and elec onic heal h
eco ds, o deli e p ecise pa ien p edic ions. In o de o ge close o he clinical p ac ice
ou ine, models ha can handle mul iple ypes o da a a e gaining popula i y. Cons aining
o a single modali y can signi ican ly educe he clinical po en ial. Acco ding o [40],
mul imodal models could ex end AI-based sys ems h oughou diagnosis and clinical ca e
wi h a long- e m ision o de eloping a “gene alis medical AI”.
Model-cen ic s. Da a-cen ic AI In he ealm o AI esea ch, he p ima y ocus lies in
de eloping a model good enough o handle he noise in he da a, wi h a mino i y ocusing
on unde s anding and imp o ing he da a. As s a ed by And ew NG [41], mo e han 90% o
he esea ch pape s a e model-cen ic. The model-cen ic app oach ocuses on de eloping
expe imen al esea ch o imp o e he ML model pe o mance while keeping he da a he
same and imp o ing he code o model a chi ec u e. On he con a y, in a da a-cen ic iew,
he consis ency o he da a is pa amoun . This app oach in ol es sys ema ically imp o ing
he da ase s o inc ease he pe o mance o ML applica ions. Despi e li ing in an e a whe e
da a is a he co e o e e y decision-making p ocess, AI ini ia i es o en neglec and
mishandle da a. Consequen ly, coun less hou s a e was ed ine- uning models based on
aul y da a, po en ially comp omising accu acy despi e op imiza ion e o s [42]. As da a
se es as he ounda ional ma e ial o da a-d i en app oaches, i is essen ial o p io i ize
high-quali y, adequa ely sized, and consis en ly labeled da ase s. While singula iewpoin s
domina e, adop ing a hyb id app oach ha inco po a es bo h model and da a
conside a ions may o e he mos e ec i e s a egy. Acco ding o he WHO Regula o y
Conside a ions on AI o heal h [28], da a quali y g ea ly in luences he success o AI
solu ions’ sa e y and e ec i eness.
006
MOTIVATION & OBJECTIVES
Glioma umo s, wi h hei complex and he e ogeneous na u e, pose signi ican challenges
wi hin he medical ield, demanding he de elopmen o inno a i e and obus diagnos ic
and p ognos ic solu ions. Au oma ic eliable iden i ica ion and cha ac e iza ion o b ain
umo s o hei la e emo al is a g owing public heal h conce n wo ldwide. The cu en
p ac ice o glioma diagnosis has some associa ed limi a ions, such as clinicians’ skills,
in e -expe a iabili y, and la ge wai ing imes o ob ain esul s. These issues highligh he
impo ance o de eloping compu e -aided diagnosis sys ems o help adiologis s in e p e
and quan i y abno mali ies om b ain images h ough he au oma ion and s anda diza ion
o edious and ime-consuming asks in ol ing iden i ica ion, classi ica ion, o
segmen a ion.
Algo i hms ha could a leas pa ially au oma e he p ocess o umo diagnosis, malignancy
quan i ica ion, and p ognosis assessmen would be e y aluable. AI-d i en app oaches ha e
eme ged as p omising ools in his ega d. By ex ac ing quan i a i e ea u es om medical
imaging da a, hey can con ibu e o he de elopmen o non-in asi e and p e-ope a i e
decision-making sys ems.
This hesis was o iginally concei ed o de elop an accu a e au oma ic sys em o suppo
clinicians in he complex ealm o glioma diagnosis and g ading. Howe e , his esea ch
aced unexpec ed hu dles. Despi e genuine in en ions o con ibu e meaning ully o he
ield, e o s o p ocu e essen ial da a om hospi als and secu e he coope a ion o clinicians
un o una ely esul ed in se backs. These challenges un eiled a c i ical aspec o he b oade
landscape - he indispensable need o collabo a ion among di e se s akeholde s. The
in eg a ion o AI in heal hca e holds immense p omise, ye i s success ul implemen a ion
elies hea ily on mul idisciplina y and in e disciplina y collabo a ion in ol ing domain and
AI expe s.
Al hough da a is a he co ne s one o e e y decision-making p ocess, he ocus is o en
p ima ily on he models. Howe e , despi e he po en ial o ans o ma i e change o AI in
heal hca e applica ions, signi ican challenges emain. One majo conce n is he opaci y o
AI sys ems, which can hinde us and adop ion among heal hca e p o essionals and
pa ien s alike. This hesis emphasizes he c i ical impo ance o la ge-scale, obus da ase s
and he es ablishmen o eliable p ocesses as ounda ional elemen s o de eloping solu ions
ha can be seamlessly in eg a ed in o heal hca e ou ines. By p io i izing anspa ency, da a
quali y, and p ocess eliabili y, we aim o build AI sys ems ha no only pe o m well bu a e
also us ed and widely adop ed in clinical se ings.
007
This hesis is s uc u ed a ound he ollowing key objec i es:
•Re iew he openly a ailable da abases o b ain umo classi ica ion esea ch.
This e iewaims oassess hequali yo heopenda abases o suppo ingdeeplea ning
esea ch in b ain umo classi ica ion, pa icula ly o glioma g ading, and o iden i y
po en ial gaps o he need o new da ase s.
•Re iew he cu en s a e-o - he-a deep lea ning-based me hodologies o
b ain umo classi ica ion om MRI, wi h a ocus on glioma g ading. This
e iew aims o iden i y gaps in he cu en me hods and highligh a eas o u he
esea ch.
•E alua e a ious MRI p e-p ocessing me hods o enhance he dis inc ion o
glioma g ades using deep lea ning. This e alua ion aims o iden i y he mos
e ec i e s a egies o an accu a e and obus glioma classi ica ion, conside ing
echniques u ilized in he cu en s a e o he a .
•De elop an accu a e and obus deep lea ning-based model o classi y he
WHO glioma g ade om MRI. This in ol es compa ing a ious da a p epa a ion
and a chi ec u e s a egies o ad ance owa d a clea unde s anding o how hese
ac o s impac he pe o mance o deep lea ning-d i en glioma g ading sys ems.
•Compa e he e icacy o 2D-based and 3D-based me hodologies o ex ac
meaning ul pa e ns in MRI using deep lea ning. The goal is o de elop a
easible, comp ehensi e, and obus model sui able o clinical implemen a ion,
balancing compu a ional e iciency, pe o mance, and in e p e abili y.
•De elop and e alua e a mul i- ask amewo k design o di e en ia e glioma
g ades and he p esence o molecula ea u es simul aneously. This goal seeks
o explo e he con ibu ion o molecula cha ac e is ics o glioma g ade classi ica ion
aiming o align mo e closely wi h clinical p ac ice.
•E alua e possible demog aphic dispa i ies in he p oposed models. The aim is
o p omo e anspa ency and ai ness in pe o mance e alua ion and esul epo ing,
ensu ing he esponsible u iliza ion and dis ibu ion o AI echnologies and mi iga ing
biases p esen in he algo i hms.
•Fos e collabo a i e mul idisciplina y and in e disciplina y e o s. The goal is
o aise awa eness o he need o u he esea ch and o p omo e he c ea ion o
meaning ul, up- o-da e, la ge-scale, and obus da ase s.
008
THESIS OUTLINE & CONTRIBUTIONS
This disse a ion is s uc u ed in o h ee pa s. The i s , backg ound, p o ides essen ial
clinical and me hodological concep s necessa y o unde s anding he hesis, accompanied
by an ex ensi e e iew o he cu en cu ing-edge esea ch in he ield. The second pa ,
p ac ice, del es in o he speci ic me hods employed and he co esponding esul s ob ained.
Las ly, he closing pa , p o ides an in-dep h discussion o he esea ch conclusions and
po en ial u u e di ec ions. The gene al ou line is illus a ed in Figu e 0.1.
Figu e 0.1. Gene al hesis ou line, including i s pa s and chap e s.
009
This pa is o ganized in o h ee chap e s, each laying he ounda ional g oundwo k essen ial
o his hesis.
Chap e 1p o ides a ho ough o e iew o he clinical ounda ion o his s udy. I begins
wi h he de ini ion o b ain cance and aces he e olu ion o he Wo ld Heal h O ganiza ion
glioma classi ica ion o i s cu en o m. I subsequen ly del es in o MRI, highligh ing i s ole
as he p incipal non-in asi e diagnos ic ool in neu o-oncology.
Chap e 2explo es he me hodological ounda ion o he hesis. Fi s ly, we del e in o he
undamen als o deep lea ning o image classi ica ion, co e ing essen ial me hods and he
mos success ul ne wo k a chi ec u es. Secondly, we discuss a ious s a egies employed in
deep lea ning o medical imaging classi ica ion assessmen . Nex , we explo e app oaches
aimed a add essing he challenge o da a sca ci y in deep lea ning. Las ly, we examine
in e p e abili y mechanisms.
Chap e 3 e iews he la es ad ancemen s in MRI- and DL-based me hodologies o
b ain umo classi ica ion, wi h a pa icula ocus on glioma g ading. This chap e p o ides
an in-dep h analysis o he s a e-o - he-a echniques, he in icacies o he me hodologies
employed, and he da ase s u ilized in hese s udies.
017
1
CLINICAL
BACKGROUND
1.1 THE BRAIN CANCER
The b ain is comp ised o wo dis inc ca ego ies o specialized cells: neu ons and glial cells.
Neu ons a e highly specialized in unc ion and in e connec ed, while glial cells p o ide
s uc u al suppo . Heal hy cells exhibi a egula ed g ow h pa e n, hal ing p oli e a ion
when a su icien numbe o cells a e p esen . This ensu es he eplacemen o old o
damaged cells. In con as , umo cells p oli e a e uncon ollably and ail o adhe e o he
ypical in e ac ions obse ed be ween no mal cells.
A p ima y b ain umo a ises om an uncon ollable p oli e a ion o cells o igina ing
wi hin he b ain i sel , which is di e en om b ain me as ases, as he la e o igina e om
cells elsewhe e in he body. Mos b ain umo s a e no linked wi h any speci ic cause, bu
some ac o s can aise he isk, including age, gene ics, li es yle, and en i onmen al ac o s.
Radia ion exposu e is he bes -known isk ac o o b ain umo s and o he common
ac o s include age, amily his o y, o bad habi s (alcohol, smoking, o poo
nu i ion) [1,43]. B ain umo s cause di e en symp oms by p essing on he b ain o he
spinal co d, aking up space inside he skull when hey g ow. Headaches, seizu es, ision loss,
weakness, and speech impai men a e among he mos common symp oms [1,43,44].
The e a e hund eds o ypes o b ain umo s. Among he benign a ie ies, meningioma
o igina es in he meninges and accoun s o mo e han 30% o all b ain umo s.
App oxima ely 85% o meningiomas a e non-cance ous and exhibi slow g ow h a es.
Pi ui a y adenomas, de i ing om he pi ui a y glands, cons i u e a ound 10% o b ain
umo s, cha ac e ized by hei g adual de elopmen [45,46]. Gliomas o igina e om glial
cells and accoun o o e 30% o all b ain umo s, encompassing 75% o malignan p ima y
umo s [3] and a e o en accompanied by a poo clinical p ognosis. Unlike meningiomas
and pi ui a y adenomas, gliomas possess a di use na u e, sp eading widely h oughou he
b ain and ex ensi ely in il a ing su ounding b ain issue. Gliomas can occu e e ywhe e,
wi h he denses occu ence ound in he on al lobe [47]. The male- o- emale a io among
a ec ed pa ien s is abou 3:2. The peak age a onse o anaplas ic as ocy omas is in he
ou h o i h decade, whe eas glioblas omas usually p esen in he six h o se en h
decade [10,48].
023
1.2 THE WHO CNS CRITERIA
Acco ding o he ini ial WHO CNS classi ica ion, gliomas we e p ima ily diagnosed
his opa hologically concep ually based on he p esumed cell o o igin [49]. B ain umo s
exhibi ing cell cha ac e is ics simila o as ocy es we e classi ied as as ocy omas, while hose
esembling oligodend ocy es we e ca ego ized as oligodend ogliomas. WHO CNS
classi ica ion has been e ol ing since 1979, whe e only mio ic ac i i y, nec osis, and umo
in il a ion we e de e minan elemen s. In 1993, he immune his ochemis y s a us was
added o he diagnosis c i e ia. La e in 2000, he gene ic p o ile s a ed o play a ole in
de e mining he g ade o he CNS umo s, ollowed by he addi ion o he his ological
a ia ion in 2007, and u he ex ended o molecula ea u es in 2016. Un il he 2016 WHO
classi ica ion, di use gliomas we e classi ied based on hei mo phologic ea u es, wi h
molecula es ing me ely playing an ancilla y ole [10]. The WHO 2021 CNS i h edi ion,
ea u ed subs an ial changes by mo ing u he o ad ance he ole o molecula
diagnos ics [8]. Fo he i s ime in he WHO classi ica ion, adul - ype gliomas we e
sepa a ed om he pedia ic- ype.
The molecula pa ame e s ha ha e now been added as bioma ke s o g ading and o
u he es ima ing p ognosis wi hin mul iple umo ypes include IDH mu a ion, 1p/19q
co-dele ion, O-6-me hylguanine-DNA me hyl ans e ase (MGMT) me hyla ion, elome ase
e e se ansc ip ase (TERT) p omo e mu a ion, epide mal g ow h ac o ecep o (EGFR)
ampli ica ion, and umo p o ein TP53 mu a ion. IDH mu a ion and 1p/19q co-dele ion
a e linked o imp o ed p ognosis and ea men esponse. On he o he hand, he MGMT
p omo e has been shown inc eased sensi i i y o alkyla ing agen s such as emozolomide,
whe eas al e a ions in he TERT p omo e , EGFR ampli ica ion, and TP53 mu a ion a e
associa ed wi h mo e agg essi e pheno ype and poo e ea men esponse [50,51].
P e ious WHO CNS classi ica ion eleases ha e conside ed a ou -s age g ading sys em
ac oss di e en umo ypes, om he mos benign o he mos malignan . Fo example, an
anaplas ic1as ocy oma and an anaplas ic meningioma would be assigned g ade 3 e en
hough hey a e biologically un ela ed and ha e di e en clinical cou ses. Con e sely, he
WHO CNS i h edi ion (2021) mo ed g ades wi hin ypes aligning wi h umo s in o he
o gan sys ems, whe e he umo is g aded acco ding o i s pa icula g ading
sys em [8,10,49]. The scheme p esen ed in Figu e 1.1 shows how he en i ies om WHO
CNS 2016 e ol ed in WHO CNS 2021 c i e ia.
1Anaplas ic cells p oli e a e apidly and ha e li le o no esemblance o no mal cells
024
Figu e 1.1. Flow diag am ha illus a es he e olu ion o he WHO 2016 classi ica ion
in o he WHO 2021 classi ica ion o gliomas. Solid lines deno e s ong co ela ions
be ween he wo classi ica ions, while do ed lines deno e how a WHO 2016 disease
en i y would likely, bu no de ini i ely, be de ined. NOS: No O he wise Speci ied.
Adap ed om Whi ield and Huse [49].
025
2
METHODOLOGICAL
BACKGROUND
2.1 MRI PRE-PROCESSING
Image p e-p ocessing plays a c i ical ole in enhancing he quali y, consis ency, and
compa abili y o MRI da a o u he analysis. In his sec ion we p esen a se ies o essen ial
s eps designed o mi iga e a ious sou ces o noise inhe en in MRI acquisi ion. These s eps
in ol e image alignmen o egis a ion, image esampling o a common esolu ion, bias ield
co ec ion, skull emo al, con as enhancemen , and in ensi y no maliza ion.
2.1.1 REGISTRATION
Image egis a ion is a c ucial s ep in medical imaging, ha in ol es aligning and combining
da a om mul iple sou ces in o a uni ied coo dina e sys em. This p ocess geome ically
aligns mul iple images cap u ed a saying imes, iewpoin s, o by di e en senso s [62,63].
Image egis a ion in ol es iden i ying a spa ial ans o ma ion ha maps mul iple images o
a common coo dina e ame [64]. The e a e h ee di e en coo dina e sys ems and spaces.
The wo ld coo dina es e e o a Ca esian coo dina e sys em in which he MRI scanne o
he pa ien is posi ioned. The ana omical space consis s o he h ee planes ha desc ibe he
s anda d ana omical posi ion o a human: axial, co onal, and sagi al. The image coo dina e
sys em (i, j, k)desc ibes how an image ( oxels) was acqui ed conce ning he ana omy.
In essence, egis a ion in ol es modi ying an image so ha i aligns wi h he physical space
and shape o a e e ence image. The alignmen o images be ween di e en coo dina e
spaces can occu h ough wo p ima y app oaches: in e -subjec and in a-subjec
egis a ion. In e -subjec egis a ion, commonly known as spa ial no maliza ion, in ol es
mapping MRI da a om mul iple subjec s in o a s anda dized ana omical empla e.
F equen ly used empla es include he Talai ach and MNI a lases, p o iding a common
e e ence o analysis ac oss subjec s. Con e sely, in a-subjec egis a ion, o en e med
co- egis a ion, ocuses on aligning da a om a single subjec ’s dis inc ana omic imaging
s udies, such as di e en modali ies o ime poin s, o ensu e in e nal consis ency wi hin
ha indi idual’s da a. The idea o in e - and in a-subjec egis a ion is illus a ed in
Figu e 2.1. Ana omical empla es e e o an a e age MR olume ha con ains an a e age
035
p obabili y o di e en issues a a pa icula spa ial loca ion in hei oxels. Since empla e
gene a ion is compu a ionally expensi e, i is common o use publicly a ailable empla es
such as MNI305[65] in which 305 T1-weigh ed MRI b ains we e linea ly co- egis e ed o
241 b ains ha had been co- egis e ed o he Talai ach a las.
The p ocess o image egis a ion is ypically i e a i e, adjus ing he ans o ma ion
pa ame e s o minimize a cos unc ion ha quan i ies he quali y o alignmen be ween wo
images [66]. Commonly used cos unc ions include he Sum o Squa e Di e ences (SSD)
and No malized C oss-Co ela ion (NCC) o in a-modali y egis a ion, and Mu ual
In o ma ion (MI) and No malized Mu ual In o ma ion (NMI) o in e -modali y
egis a ion. The spa ial ela ionships be ween he images can be modeled as igid
ans o ma ions (in ol ing o a ions and ansla ions), a ine ans o ma ions (including
scaling and shea ing), o de o mable ans o ma ions (allowing non-linea adjus emen s).
The pa ame e s op imized du ing egis a ion co espond o he Deg ees o F eedom (DoF),
which de ine he numbe o independen ways ha he ans o ma ion can a y [67]. Fo
igid egis a ion, ypically used o in a-subjec alignmen , he ans o ma ion in ol es 6
DoF: 3 o o a ions and 3 o ansla ions. Fo a ine egis a ion, commonly applied in
in e -subjec egis a ion, he ans o ma ion expands o 12 DoF, adding 3 o scaling and 3
o shea ing o he igid pa ame e s [68]. De o mable ans o ma ions a e used o complex
alignmen s and ha e a iable and o en in he o de o hund eds o housands DoF.
Figu e 2.1. Illus a ion o in e - and in a-subjec s MRI egis a ion.
036
2.1.2 RESAMPLING
MRI scans can a y in esolu ion and oxel sizes depending on he acquisi ion sys em. Voxel
size, analogous o a pixel in h ee dimensions, is c ucial o image quali y. Voxel size is
de e mined by bo h he pixel size and slice hickness. Pixel size, dependen on bo h he ield
o iew and he image ma ix, di ec ly impac s in-plane spa ial esolu ion. Typically
exp essed in millime e s, pixel size commonly alls wi hin he ange o 0.5 o 1.5 mm. The
smalle he pixel size, he g ea e he image spa ial esolu ion [69].
In MR image p ocessing, a ia ions in oxel size and esolu ion ac oss scans can pose
challenges o subsequen analysis and compa ison, especially when using analy ical
echniques ha equi e uni o m inpu dimensions. To mi iga e hese dispa i ies, esampling
in ol es he s anda diza ion o he spa ial esolu ion ac oss he MRI images o ensu e
uni o m dimensions. The p ocess in ol es in e pola ing he in ensi y alues o he o iginal
image o gene a e a new image wi h desi ed spa ial cha ac e is ics, such as pixel size, slice
hickness, o o e all dimensions.
2.1.3 BIAS FIELD CORRECTION
MRI images migh exhibi a ia ions in in ensi y, which a e commonly e e ed o as
inhomogenei y a i ac s o bias ields, along wi h noise. The image o ma ion model can be
ep esen ed by he equa ion:
ν(x) = u(x) (x) + n(x)(2.1)
whe e he image ν(x)is a combina ion o he o iginal unco up ed image u(x), he bias ield
(x), and he noise n(x), assumed o be independen and Gaussian.
Bias Field Co ec ion (BFC) consis s o an i e a i e algo i hm ha g adually es ima es and
ec i ies in ensi y a ia ions ac oss he image. This me hod elies on a mul i- esolu ion
app oach, whe e he image is successi ely smoo hed a a ious esolu ions o es ima e he
bias ield. N4ITK’s (N4 Bias Field Co ec ion) [70], e inemen o e N3 (non-pa ame ic
non-uni o m no maliza ion) [71] includes a mo e obus B-spline i ing s a egy o bias
ield es ima ion, p o iding enhanced accu acy and e iciency in co ec ing in ensi y
inhomogenei ies. By add essing hese a ia ions, BFC no only imp o es he o e all image
quali y bu also acili a es mo e accu a e and consis en quan i a i e analysis in MRI-based
s udies.
037
Figu e 2.2. Raw ( op) and bias ield co ec ed (bo om) images. A colo map has been
applied o acili a e he isualiza ion o issue di e ences a e BFC.
2.1.4 SKULL-STRIPPING
The main objec i e o he skull-s ipping s ep is o e icien ly isola e he ce eb al egion o
in e es om non-ce eb al issues, enabling DL models o ocus exclusi ely on hose b ain
issues. Skull-s ipping is no only an essen ial p e-p ocessing s ep bu also c ucial o p i acy
p ese a ion. T adi ionally, his s ep was pe o med h ough manual segmen a ion, bu in
he pas decade, signi ican e o s ha e been made o au oma e his p ocess wi h
conside able success. Fo ins ance, BET (B ain Ex ac ion Tool) [72] employs a de o mable
model wi h adap i e in ensi y h esholds o isola e he b ain om non-b ain issues. Mo e
ecen ly, DL-based me hods ha e been p oposed in he li e a u e. HD-BET, implemen ed by
Isensee e al. [73], is an ANN inspi ed in he UNe [25] capable o ex ac ing he b ain om
he ou con en ional MRI modali ies ( lai , 1ce, 1, and 2). This model was ained using
bo h heal hy and pa hological 3D olumes, demons a ing i s obus ness and e sa ili y in
a ious clinical scena ios. Ano he impo an con ibu ion was made by [74], who
compa ed mul iple DL-based models ained on a la ge b ain umo coho and p oposed a
“modali y-agnos ic” app oach, ha is obus o he p esence o missing modali ies.
038
Figu e 2.3. Raw ( op) and skull-s ipped (bo om) images.
2.1.5 CONTRAST ENHANCEMENT
Con as enhancemen me hods play a pi o al ole in highligh ing in ica e de ails and
enabling imp o ed analysis o images. One widely used echnique in image p ocessing o
his pu pose is his og am equaliza ion. This me hod aims o s e ch he his og am o
in ensi y alues wi hin an image, e ec i ely dis ibu ing hem o achie e a mo e uni o m
dis ibu ion. This edis ibu ion o in ensi ies helps un eil sub le de ails, enhancing he
o e all cla i y o he image. The implemen a ion o his og am equaliza ion is as ollows.
Conside I(x, y)as an image wi h in ensi y alues a pixel (x, y). The his og am h(i)
ep esen s he coun o occu ences o each in ensi y le el iin he image I(x, y).
his (i) = X
pixels(x,y)
p(I(x, y) = i)(2.2)
whe e p(I(x, y) = 0) is he p obabili y o an occu ence o a pixel o le el iin he image
I(x, y). The Cumula i e Dis ibu ion Func ion CDF(i)accumula es he his og am alues
up o a pa icula in ensi y le el i:
CDF(i) = X
j≤i
h(j≤i)(2.3)
039
The no maliza ion o he CDF ensu es ha i s ange emains be ween 0 and 1.
CDF′(i) = CDF(i)
CDF(max(i)) (2.4)
The o iginal in ensi y alues I(x, y)a e mapped o new in ensi y alues I′(x, y)using he
in e se o he no malized CDF unc ion, edis ibu ing he in ensi y alues and hus
enhancing he con as .
I′(x, y) = CDF ′−1(CDF ′(I(x, y))) (2.5)
which maps o iginal in ensi y alues I(x, y) o new in ensi y alues I′(x, y).
Figu e 2.4. Raw ( op) and equalized (bo om) images.
2.1.6 INTENSITY NORMALIZATION
A echnique adop ed o escale in ensi y alues o MRI scans o a nume ic ange, ende ing
hem consis en ac oss he da ase . This p ocess mi iga es scale- ela ed dispa i ies. Two
p ominen app oaches commonly applied o MRI da a as inpu o DL models a e min-max
040
no maliza ion and z-sco e no maliza ion. Min-max achie es i s goal by escaling in ensi y
alues wi hin MRI scans, spanning hei ange be ween 0 and 1, h ough he o mula
Xno m =X−Xmin
Xmax −Xmin
(2.6)
In con as , z-sco e no maliza ion o s anda diza ion, ans o ms he dis ibu ion o
in ensi y alues by cen e ing i a ound a ze o mean and s anda d de ia ion o alue 1,
h ough he o mula
Xs d =X−µ
σ(2.7)
No malizing in ensi ies is a widely acknowledged p ocess o enhance he e iciency o DL
ne wo ks. By adjus ing inpu s o a ange a ound ze o, his p ac ice enables mo e consis en
weigh upda es wi hin he model. Consequen ly, i mi iga es he issue o luc ua ing s ep
sizes, he eby alle ia ing p oblems associa ed wi h exploding g adien s, a phenomenon ha
occu s when he g adien s o he ne wo k loss become excessi ely la ge conce ning he
weigh s. Since he ne wo k ou pu is a linea combina ion o each ea u e ec o , his means
ha he ne wo k lea ns weigh s o each ea u e ha a e also on di e en scales. O he wise,
he la ge ea u e will simply d own ou he small ea u e. Then du ing g adien descen , o
“mo e he needle” o he Loss, he ne wo k would ha e o make a la ge upda e o one
weigh compa ed o he o he weigh . This can cause he g adien descen ajec o y o
oscilla e back and o h along one dimension, hus aking mo e s eps o each he minimum.
Da a no maliza ion is no only impo an be ween samples bu also wi hin. When images
ha e mul iple channels, he in ensi ies in one channel can be high compa ed o he o he
channels and he alues o his channel would become dominan , diminishing he
o he s [75].
2.2 MRI PROCESSING
MRI da a cons i u es h ee-dimensional (3D) olumes ha enable isualiza ion ac oss axial,
co onal, and sagi al ana omical planes. These olumes a e composed o 3D cubes known as
oxels, in con as o he s anda d wo-dimensional (2D) images which a e made up o 2D
squa es called pixels. While 3D olumes o e comp ehensi e da a, hey can also be
decomposed in o 2D slices. MRI images a e commonly p o ided in NI TI (Neu oimaging
In o ma ics Technology Ini ia i e) o ma o DICOM (Digi al Imaging and
Communica ions in Medicine). NI TI is a s anda d o ma o s o ing and exchanging
neu oimaging da a, while DICOM is a widely used s anda d o medical imaging.
041
Figu e 2.7. A chi ec u e o a 2D anilla CNN wi h wo con olu ional laye s, wo pooling
laye s, one la ened laye , and one ully connec ed laye .
Se e al ne wo ks ha e made signi ican con ibu ions o he wo ld o DL. The i s CNN,
LeNe 5, was p esen ed by LeCun e al. [83] in 1989 and i was called LeNe 5. I consis s o
mul iple con olu ional and pooling laye s, ollowed by a ully-connec ed laye . Fu he o
LeNe ’s eme gence, se e al CNNs made signi ican con ibu ions o he wo ld o DL.
AlexNe [16], was he i s a chi ec u e o employ max-pooling laye s, ReLU ac i a ion
unc ions, and d opou egula iza ion me hod. I was he i s deep CNN ained on
ImageNe and won he ImageNe La ge Scale Visual Recogni ion Challenge (ILSVRC) in
2012. In he ILSVRC 2014, Szegedy2014 won wi h a CNN called GoogleNe , achie ing a
signi ican ly low e o a e. I s main con ibu ion was he implemen a ion o an incep ion
module ha allows o mo e e icien compu a ion and a subs an ial educ ion in he
numbe o pa ame e s in he model. The e ol ed e sion o GoogLeNe is
calledIncep ionNe . Ano he ele an ne wo k, VGGNe , was in oduced by Simonyan and
Zisse man [18] wi h he pa icula i y o using small-size con olu ion il e s ha allow
inc easing he dep h o he ne wo k, while signi ican ly educing he numbe o necessa y
pa ame e s. Among he mos popula CNNs, ResNe was de eloped by He e al. [19] o win
he ILSVRC challenge in 2015. This wo k p oposed he use o esidual blocks, which help
o add ess he p oblem o anishing g adien s when inc easing he dep h o he ne wo k. In
2017, he concep o dense blocks was in oduced wi h DenseNe [84], in which he ea u e
maps ob ained om each laye a e conca ena ed a e eed he subsequen laye . In 2019, Tan
and Le [85] in oduced E icien Ne , which in oduces he idea o iden i ying op imal
dep h, wid h, and esolu ion. Mo e ecen ly, T ans o me s, ini ially wi h he ViT [27],
gained popula i y in he ield o compu e ision h ough he addi ion o sel -a en ion
laye s.
DL models a e conside ed da a-hung y since hey equi e subs an ial amoun s o da a o
e ec i e aining. In he ealm o medical da a analysis, a p ima y challenge, as p e iously
men ioned, is he inhe en da a sca ci y. To add ess his challenge, common solu ions
include he applica ion o Da a Augmen a ion me hods and T ans e Lea ning (TL)
048
echniques. Da a Augmen a ion echniques a e a c ucial s a egy o mi iga e he challenge
o limi ed anno a ed da a in medical image analysis. These me hods encompass a ange o
ans o ma ions applied o exis ing images, e ec i ely expanding he da ase in e ms o bo h
size and di e si y. Fo me app oaches in ol e a wide ange o geome ic modi ica ions such
as o a ion, scaling, lipping, c opping, zooming, o colo changes. Beyond adi ional
augmen a ions, ad anced me hods like Gene a i e Ad e sa ial Ne wo ks (GANs) [86] a e
used o gene a e new syn he ic and ealis ic examples.
The idea behind TL is o ans e he knowledge om a sou ce ask o a new domain. TL
in ol es using a p e- ained model, ypically ained on la ge and di e se da ase s, as a
s a ing poin and adap ing i o a new ask, especially when he a ailable da a may lack
di e si y and ep esen a ion. Di e en s a egies a e a ailable depending on he simila i y
be ween he wo domains. When he new ask closely esembles he o iginal ask, a p ac ical
app oach is o ini ia e wi h he p e- ained model, eezing ce ain laye s ha cap u e
low-le el ea u es, while ine- uning high-le el laye s ha adap o he new domain and
cap u e mo e complex ea u es. Con e sely, when he new ask di e ges signi ican ly om
he o iginal ask, he app oach in ol es un eezing all he laye s o he p e- ained model,
allowing hem o adap and lea n speci ic ea u es om he new da ase , cons i u ing a
weigh ini ializa ion s a egy.
Widely used p e- ained CNNs, such as ImageNe [87] o MS-COCO [88], ha e been
o iginally de eloped om 2D la ge-scale da ase s. Howe e , a no able challenge when
dealing wi h medical image da a is he limi ed a ailabili y o la ge and di e se 3D da ase s o
uni e sal p e aining [89]. T ans e ing he knowledge acqui ed om he 2D o he 3D
domain p o es o be a non- i ial ask, p ima ily due o he undamen al di e ences in da a
s uc u e and ep esen a ion be ween hese wo con ex s. To ackle his challenge and
add ess he limi a ion o limi ed da a, a b oadly used s a egy is o decompose 3D olumes
in o indi idual 2D slices wi hin a de e mined ana omical plane. Howe e , he
decomposi ion o 3D olumes in o indi idual 2D slices in oduces a po en ial da a leakage
conce n. This issue a ises when 2D slices om he same indi idual inad e en ly end up in
bo h he aining and es ing da ase s in an analy ical pipeline. Such da a leakage can lead o
o e es ima ions o model pe o mance and a ec he alidi y o expe imen al esul s. In
addi ion, i is impo an o no e ha his app oach comes wi h he ade-o o losing he 3D
con ex p esen in he o iginal da a.
2.5 MODEL EVALUATION
In o de o e alua e he gene aliza ion abili y in ML, a ious me hodologies a e a ailable.
The hold-ou me hod in ol es di iding he da ase in o wo subse s, a aining se used o
ain he model and a es ing se o e alua e he pe o mance. I is also common o add a
hi d se , known as alida ion, o moni o ing and uning hype pa ame e s o op imize
049
model pe o mance. Howe e , a p ima y d awback o his me hod is i s suscep ibili y o
selec ion bias, pa icula ly when he sample size is small. To add ess his limi a ion, K- old
C oss-Valida ion (CV) eme ges as a powe ul al e na i e. In K- old CV, he da ase is
pa i ioned in o K subse s (o olds) and he model is ained and e alua ed K imes, each
ime u ilizing a di e en old as he alida ion se and he emaining olds o aining.
Pe o mance me ics a e compu ed o each alida ion se and, ypically, he a e age
pe o mance is epo ed. This app oach p o ides a mo e obus es ima e o model
pe o mance compa ed o he hold-ou me hod, especially in scena ios wi h limi ed da a. I
is also common o ha e an independen es se o ensu e ha he inal model e alua ion is
unbiased and ep esen a i e o i s ue gene aliza ion pe o mance.
Figu e 2.8. K- old C oss-Valida ion scheme.
2.6 DATA INTEGRATION
Da a in eg a ion in ol es combining in o ma ion om a ious modali ies, sou ces, o
pe spec i es. By le e aging mul iple da a sou ces, models can cap u e a iche se o ea u es
and complemen a y con ex ual in o ma ion, enhancing he obus ness and accu acy o he
p edic ions [90,91]. Da a in eg a ion s a egies a e di ided in e ms o he s age a which he
in o ma ion usion occu s.
Ea ly Fusion 1in eg a es aw da a om a ious modali ies a he ini ial s age o he
p ocessing pipeline. The used ep esen a ion is hen ed in o a single model. An ad an age
o ea ly usion is i s compu a ional e iciency due o a single model. Howe e , a d awback is
ha i migh s uggle o e ec i ely cap u e complex in e ac ions be ween di e en
modali ies, po en ially leading o in o ma ion dilu ion.
In e media e Fusion 2Occu s a e some ini ial p ocessing has been pe o med
independen ly on each modali y. The ex ac ed ea u es a e hen combined, and he used
050
ep esen a ion unde goes u he p ocessing o p oduce he ou pu sco es. The ad an age o
in e media e usion is ha he model can lea n ich in e ac ions ac oss each modali y.
La e Fusion 3In ol es aining sepa a e models a e ained o each modali y
independen ly, and he ou pu sco es a e combined a he inal s age. This app oach is akin
o an ensemble me hod. While la e usion can inc ease he obus ness o he model by
mi iga ing he impac o e o s om a single model, i does no enable he model o lea n
complex in e ac ions ac oss modali ies. Addi ionally, i can be compu a ionally in ensi e, as
i equi es aining mul iple models.
Figu e 2.9. Da a in eg a ion s a egies.
1Also known as inpu -le el usion.
2Also known as ea u e-le el usion.
3Also known as decision-le el usion.
051
2.7 MULTI-TASK LEARNING
In adi ional DL, a sepa a e model is ypically ained o each speci ic ask. Mul i- ask
Lea ning (MTL) was i s in oduced by Ca uana [92] and consis s o aining a single
model o pe o m mul iple asks simul aneously. In he con ex o DL, his in ol es aining
a neu al ne wo k o lea n ep esen a ions ha a e use ul o mul iple ela ed asks. The
concep is inspi ed by human lea ning, o new asks we apply he knowledge we ha e
acqui ed by lea ning ela ed asks. The main ad an age is ha he model can lea n he
con ex be ween di e en ea u es. By sha ing ep esen a ions be ween ela ed asks, we can
enable ou model o gene alize be e on ou o iginal ask [93]. Mul i- ask lea ning imp o es
gene aliza ion by le e aging he domain-speci ic in o ma ion con ained in he aining
signals o ela ed asks [94]. By join ly lea ning mul iple asks he model acili a es TL, in he
sense ha he knowledge gained om lea ning one ask can be ans e ed o imp o e
pe o mance on ela ed asks, bu i can also implici ly lea n he ela ionship be ween asks.
As di e en asks ha e di e en noise pa e ns, a model ha lea ns mul iple asks
simul aneously can lea n a mo e gene al ep esen a ion educing he isk o o e i ing [93].
Figu e 2.10. Mul i- ask lea ning.
In mul i- ask DL we use one ne wo k ha pe o ms mul iple asks simul aneously which
in ol es ha d o so pa ame e sha ing o hidden laye s. Ha d pa ame e sha ing in ol es
he pa ame e s o hidden laye s being sha ed ac oss all asks while keeping se e al
ask-speci ic ou pu laye s. In so pa ame e sha ing, each ask has i s model wi h i s
pa ame e s, bu he pa ame e s a e encou aged o be simila ac oss asks h ough
egula iza ion echniques. This las app oach equi es high compu a ional cos . Mul i- ask
lea ning eme ges as a use ul app oach when he e is an abundance o labeled da a o one
ask ha can be sha ed wi h ano he ask wi h less labeled da a [95].
052
2.8 EXPLAINABLE DEEP LEARNING
E en hough sophis ica ed algo i hms can p oduce ancy ou comes, hey equen ly come a
a cos – opaci y. A i icial Neu al Ne wo k models, and DL models in pa icula , a e o en
e e ed o as “black boxes” due o he opaci y o hei in e nal mechanisms. This lack o
anspa ency poses a ba ie o hei accep ance and app o al o clinical use [96,97]. Clinical
decisions can be conside ed high- isk asks since hey ha e a s ong impac and he model
beha io comp ehensibili y is desi ed o be high. The mo e complex he a chi ec u es, he
mo e di icul he explana ion o how and why a pa icula ne wo k p edic ion is ob ained,
o he elucida ion o which componen s o he complex sys em con ibu ed essen ially o he
ob ained decision [98].
In ecen yea s, he e has been an inc easing emphasis on he necessi y o hese models o
be bo h anspa en and in e p e able. In e p e able ML e e s o me hods and models ha
make he beha io and p edic ions o machine lea ning sys ems unde s andable o humans.
In e p e abili y can be de ined as he deg ee o which a human can unde s and he cause o a
decision [99]. An ML model is conside ed in e p e able due o i s simple s uc u e, such as
linea models o decision ees. Explainabili y is closely ied o in e p e abili y bu in ol es he
useo addi ional pos -hoc me hods ocomp ehend hemodel’sdecision. I ackles he p oblem
whe einhumanuse s acedi icul yindi ec lycomp ehending hein ica ebeha io so Deep
Neu al Ne wo k (DNN) o elucida ing hei unde lying decision-making mechanisms.
Explainabili y in CNNs is equen ly add essed h ough ea u e isualiza ion. Pixel
A ibu ion me hods, also known as Saliency Maps, highligh he pixels ha a e mos
ele an o a speci ic classi ica ion ask. Fo enhancing image classi ica ion anspa ency and
unde s anding, se e al echniques ha e eme ged, b oadly ca ego ized in o wo main ypes:
hose ha ely on pe u bing ea u es o obse e changes in he model ou pu and hose ha
u ilize g adien in o ma ion o asce ain he in luence o each pixel on he decision-making
p ocess. A popula pe u ba ion-based me hod is Local In e p e able Model-Agnos ic
Explana ion (LIME), as in oduced by Ribei o e al. [100] in 2016. The key in ui ion
behind LIME is o gene a e a new da ase con aining pe u bed samples by u ning some o
he componen s “o ” and he co esponding p edic ions o he model. Fo each pe u bed
sample, LIME compu es he class p obabili y and hen lea ns an in e p e able model (e.g.
linea eg ession) which is weigh ed by he dis ance be ween he pe u bed image and he
o iginal image [101]. The bigge he weigh , he bigge he impo ance o he ea u e.
Ma hema ically,
ξ(x) = a g min
g∈G
L( , g, πx) + Ω(g)(2.24)
whe e Lis a measu e o how un ai h ul gis in app oxima ing in he locali y de ined by πx.
Ω(g)is a measu e o he complexi y o he explana ion g∈G.
053
G adien -based me hods u ilize he g adien s o he ou pu wi h espec o he ex ac ed
ea u es calcula ed h ough backp opaga ion o e eal which pa s o he inpu con ibu e
he mos o he inal decision. A widely used g adien -based me hod is G adien -weigh ed
Class Ac i a ion Mapping (G adCAM) [102]. An in ui i e idea o G adCAM is o
unde s and which pa s o he image he con olu ional laye gi es mo e a en ion o a
ce ain decision by analyzing which egions a e ac i a ed in he ea u e maps. In o de o
decide he impo ance o each ea u e map o he class o in e es , G adCAM weigh s each
pixel o each ea u e map wi h he g adien a e aging o e he ea u e maps [101]. Then, o
each inpu ea u e (pixel) i p oduces a alue be ween 0 and 1 ep esen ing a low o high
con ibu ion in he class p edic ion, which esul s in a hea map ha highligh s he egions
ha con ibu e ei he nega i ely o posi i ely o he decision. The objec i e is o p oduce a
localiza ion map, ep esen ed ma hema ically as:
Lc
G adCAM ∈Rux =ReLU
|{z}
Pick posi i e alues X
k
αc
kAk!(2.25)
He e, uand ep esen he wid h and he heigh , espec i ely, o he explana ion map o a
gi en classc. The e mαkdeno es heimpo anceo hek- h ea u e map o he a ge classc,
and Ak e e s o he ac i a ions o a con olu ional laye ’s ea u e map. The applica ion o he
ReLU unc ion ensu es ha only posi i e con ibu ions o he classo in e es a e conside ed.
To calcula e αc
k he equa ion is as ollows:
αc
k=
global a e age pooling
z }| {
1
ZX
iX
j
δyc
δAk
ij
|{z}
g adien s ia backp op
(2.26)
This ep esen s he global a e age pooling o he g adien s compu ed ia backp opaga ion
ac oss he ea u e map Ak.Ak
ij e e s o he ac i a ion a loca ion (i, j)o he ea u e map Ak
and Yc e e s o he sco e o each class c. The a iable Zaccoun s o he o al numbe o
pixels in he ea u e map. The class sco e yccan be exp essed as:
Yc=X
k
wc
k
|{z}
class ea u e weigh s
GAP
z }| {
1
ZX
iX
j
Ak
ij
|{z}
ea u e map
(2.27)
whe e wc
ka e weigh s indica ing he ele ance o ea u es wi hin he map Ak o he class c.
054
2.9 BIAS IN MACHINE LEARNING MODELS
F om a legal pe spec i e, bias e e s o an inclina ion o p ejudice owa ds a pa icula
indi idual o en i y. In s a is ics, bias e e s o he sys ema ic de ia ion o an es ima o om
he ue alue o he pa ame e being es ima ed. In he ealm o ML, bias ansla es o he
sys ema ic e o in he model’s p edic ions om he ac ual ou comes. This implies he
possibili y o ha ing an unbiased model in e ms o ML while s ill possessing bias in legal o
s a is ical e ms. Algo i hmic decisions a e o en pe cei ed as ai o unbiased due o
minimal human in e en ion, which heo e ically elimina es human biases om he
decision-making p ocess. When we obse e dispa i ies, i does no imply ha he designe o
he sys em in ended o such inequali ies o a ise, howe e , algo i hmic decision-making
may unin en ionally e lec exis ing inequali ies p esen in he socie y, which a e inhe en ly
p esen in he da a [103]. Machines a e no inhe en ly une hical [104], howe e , an
algo i hm is only as good as he da a i wo ks wi h [105]. I he da ase is skewed o
imbalanced, he model may de elop a p opensi y owa ds ce ain pa e ns o ou comes,
po en ially yielding un ai o disc imina o y esul s o speci ic popula ions. This bias poses
a signi ican isk, as i can lead o ou comes ha un ai ly impac ce ain g oups o
indi iduals, which in heal hca e would be ansla ed o inequi able heal hca e ou comes.
Recen wo ks ha e demons a ed ha ML models can exhibi bias based on ce ain ea u es
such as sex [106–108] o ace [109,110]. As ML inc easingly pe mea es decision-making
p ocesses, i becomes c ucial o de elop models ha a e no only accu a e bu also objec i e
and ai . Th ee common no ions o (un) ai ness can be de ined as ollows [111,112]:
Dispa a e ea men 1A decision-making p ocess su e s om dispa a e ea men i i s
ou comes change based on a change in he sensi i e ea u e alue wi h all o he ea u es being
he same.
Dispa a e impac 2A decision-making p ocess su e s om dispa a e impac i i g an s a
disp opo iona ely la ge ac ion o bene icial (o posi i e classi ica ion) ou comes o ce ain
sensi i e ea u e g oups.
Dispa a e mis ea men 3A decision-making p ocess su e s om dispa a e mis ea men
i i s misclassi ica ion a es a e di e en o di e en sensi i e g oups.
While dispa a e ea men accoun s o di ec un ai ness whe e a decision-making p ocess
in en ionally uses he sensi i e in o ma ion o he classi ica ion ou come, dispa a e impac
1Dispa a e ea men has been also e e ed o as di ec disc imina ion [113].
2Dispa a e impac has been also e e ed o as s a is ical pa i y [114] o demog aphic pa i y [115].
3Dispa a e mis ea men has been also e e ed o as equali y o oppo uni y [116] and p edic i e
equali y [114].
055
and dispa a e mis ea men accoun o indi ec un ai ness, whe e he model
unin en ionally uses he co ela ion be ween sensi i e ea u es and class labels in a way ha
places a sensi i e ea u e g oup a a ela i e disad an age. In he con ex o ai ness in bina y
classi ica ion, we can exp ess he absence o un ai ness concep s as ollows [111]:
No dispa a e ea men . A bina y classi ie does no su e om dispa a e ea men i
he p obabili y ha he classi ie ou pu s a speci ic alue o ˆygi en a ea u e ec o xdoes no
change a e obse ing he sensi i e ea u e z,
P(ˆy|x, z) = P(ˆy|x)(2.28)
No dispa a e impac . A bina y classi ie does no su e om dispa a e impac i he
p obabili y ha a classi ie assigns a use o he posi i e class is he same o bo h alues o
he sensi i e ea u e z,
P(ˆy= 1|z= 0) = P(ˆy= 1|z= 1) (2.29)
No ice ha nei he dispa a e ea men no dispa a e impac depends on he subjec s’ g ound
u h label (y).
No dispa a e mis ea men . A bina y classi ie does no su e om dispa a e
mis ea men i he misclassi ica ion a es o di e en g oups o people ha ing di e en
alues o he sensi i e ea u e za e he same.
App oaches o mi iga e bias can be classi ied in o h ee ca ego ies: p e-p ocessing,
in-p ocessing, and pos -p ocessing. P e-p ocessing in ol es emo ing inequi ies p io o
model aining, o example by inco po a ing new da a. In-p ocessing me hods a e applied
du ing he lea ning p ocess, by o example imposing cons ain s o egula iza ion. Finally,
pos -p ocessing in ol es adjus ing he model’s p edic ions o co ec any emaining
biases [117].
056
Figu e 3.1. Publicly a ailable da ase s usage equency om 2018 o 2024.
Fo a de ailed e iew, he au ho e e s o Pi a ch e al. [257], whe e Table A1 o e s a de ailed
o e iew o heda ase s, ocusing onessen ialaspec s suchas hedimensionali yo heimages,
sample size, MRI de ails, and p e-p ocessing me hods used. Table A2 in he e e ed a icle,
del es in o he speci ica ions o he employed DL models, highligh ing he b ain umo
classi ica ion ask, da a pa i ioning, a chi ec u e, and he epo ed pe o mance me ics.
3.2.1 BRAIN TUMOR CLASSIFICATION OVERVIEW
The pe o mance, gene alizabili y, and obus ness o machine lea ning models a e signi i-
can ly impac by he size and di e si y o he aining da a. Se e al s udies ha e explo ed he
impac o a ying he size o he aining da a se on model pe o mance [130,135,143,157,
162,164,218,235,248]. Thei indings highligh he alue o ensu ing ha a subs an ial
olume o da a is a ailable o aining, as i signi ican ly con ibu es o he model’s abili y o
make mo e accu a e and eliable p edic ions. Recen e o s ha e aimed a o e coming da a
sca ci y challenges. App oxima ely 60% o he examined s udies employed da a augmen a ion
echniques, and 40% inco po a ed ans e lea ning in a 2D domain as a iable solu ion. A
numbe o hese in es iga ions [139,144,153,159,170,216,220,236] ha e demons a ed
he ad an ages o inc easing bo h he quan i y and a iabili y o he samples h ough he
064
inclusion o augmen ed images. Applying adi ional da a augmen a ion echniques, such
as geome ic a ia ions om o iginal images, was he mos widely used s a egy, while only
a ew s udies op ed o he use o DL gene a i e models [147,159,168,228]. Se e al s ud-
ies[183,220,239,258–260]ha ein eg a edda aaugmen a ionasan o e sampling echnique
o add ess he p oblem o imbalanced da a in he con ex o b ain umo classi ica ion. O he
wo ks ha e explo ed he inclusion o mul i- iew 2D slices om sagi al, co onal, and axial
planes, in addi ion o employing image lipping and o a ions o augmen he da ase [221].
P e- ained models ha e demons a ed pe o mance enhancemen s in he classi ica ion o
glioma g ades in se e al s udies [223,224,235]. Howe e , no all in es iga ions ha e epo ed
equi alen ad an ages when employing p e- ained models o disc imina e be ween heal hy
and umo ous samples[187] o o di e en ia e umo ypes[141]. These a ia ions in indings
unde sco e he complexi y o he obse ed pe o mance dispa i ies, which may no be solely
asc ibed o he classi ica ion ask i sel , bu also be in luenced by in insic da ase a ia ions.
The p edominan app oach o da a pa i ioning in ol es he use o hold-ou alida ion, wi h
aining and alida ion se s. This was ollowed by he adop ion o K- old c oss- alida ion,
which enhances he obus ness o model e alua ion. A less equen ly employed me hod was
he h ee-way spli , which includes aining, alida ion, and es ing se s. In o al, only 36%
o he s udies assessed hei inal esul s using an independen es se . Alanazi e al. [160],
Decuype e al. [233], Gilanie e al. [234] ook a s ep u he by assessing he gene alizabili y
and obus ness o hei models using ex e nal es ing se s. While he au ho s o he Figsha e
da ase hough ully included a 5- old CV se up alongside he da a o p omo e compa abili y
and ep oducibili y, i is s ill impo an o ema k ha a subs an ial majo i y o s udies using
his da abase con inue o p e e cus om da a pa i ioning me hods.
P e-p ocessing echniques a e pi o al in medical image analysis. A subs an ial 80% o he
conside ed pape s, p o ide insigh s in o he speci ic p e-p ocessing me hodologies ha we e
employed. Wi hin hissubse ,35%employed egis a ion echniques,in ol ing egis a ion o
a common ana omical empla e and co- egis a ion o he same MRI modali y. Fu he mo e,
40% employed segmen a ion as a c i ical s ep o isola e he b ain om he su ounding skull
s uc u es. Nea ly hal o he pape s emb aced no maliza ion echniques o s anda dize he
in ensi y o he image da a be o e i was ed o he models. A en ion has also been di ec ed
owa d iden i ying he op imal MR image in ensi y no maliza ion app oach, which plays a
c i ical ole in enhancing model pe o mance in DL models. Many s udies p e e mean-s d
s anda diza ion [54,57,143,146,160,176,232,236,237,240,243,253,254,260–267],
while o he s lean owa ds min-max no maliza ion [138,155,162,173,177,182,183,212,215,
218,247,268]. Addi ionally, some s udies inco po a e con as enhancemen me hods o
imp o e he con as and isibili y o c ucial ana omical s uc u es. [143,149,151,154,156,
158,176,181,214,229,231,251]. Addi ionally, 30% o he pape s ex ac he b ain umo
egion h ough me hods such as bounding box delinea ion o umo segmen a ion. In se e al
s udies [222,258,269], esea che s in es iga ed he ad an ages o u ilizing he umo a ea
as opposed o he en i e image, highligh ing he signi ican bene i s o concen a ing on he
umo egion a he han he en i e image.
065
The abili y o CNNs o au oma ically ex ac meaning ul ea u es om b ain MRI images,
as opposed o he con en ional need o manual ea u e enginee ing in ce ain ML
algo i hms like RF, G B, and SVM, has been emphasized by nume ous s udies. These
s udies unde sco e he po en ial o CNNs in e olu ionizing he landscape o MRI ea u e
ex ac ion o enhanced accu acy and e iciency in b ain umo
classi ica ion [61,172,193,206,210,217,270]. A majo i y o he e iewed pape s
(app oxima ely 60%) u ilized es ablished s a e-o - he-a CNN a chi ec u es o ob ain b ain
umo classi ica ion. Among hese, ResNe and VGGNe backbones we e he mos
p e alen choices, closely ollowed by AlexNe , GoogLeNe , and Incep ion. In con as , he
emaining 40% o he pape s concen a ed on enhancing b ain umo classi ica ion by
in oducing no el model a chi ec u es. The inhe en black-box na u e o CNNs highligh s
he impo ance o del ing in o he comp ehension o hei p edic ions, especially in a
medical con ex . Se e al s udies [176,195,222,243,253,263,266] ha e applied
pos -p ocessing explainabili y ools o alida e ha he ne wo k’s decision-making p ocess
aligns wi h he in ended diagnos ic c i e ia, he eby enhancing he eliabili y o CNN-based
medical applica ions.
Me ely 70% o he e iewed s udies disclosed he MRI modali ies u ilized o he analysis.
Among hese, close o 50% exclusi ely employed he T1ce sequence, while 26% used a
combina ion o T1ce, T1, T2, and FLAIR sequences, 12% used h ee sequences, and he es
chose one sequence. Va ious s a egies we e employed o in eg a e in o ma ion om
mul iple modali ies. The p e alen me hod in ol ed using hem as inpu channels,
compa able o he ea men o channels in RGB images. In hei s udy, Ge e al. [221]
e alua ed he sensi i i y o T1ce, T2, and Flai modali ies in glioma g ade classi ica ion.
Thei in es iga ion highligh ed he T1ce sequence as he mos in o ma i e among hese
modali ies. To u he enhance he classi ica ion pe o mance, hey inco po a ed
in o ma ion om each sou ce using an agg ega ion laye wi hin he ne wo k a chi ec u e.
Subsequen ly, simila ensemble lea ning app oaches we e adop ed by Gu a e al.
[61], Hussain e al. [250], Rui e al. [252]. No ably, Guo e al. [271] di ec ly compa ed he
pe o mance o a modali y- usion app oach, whe e he ou MRI modali ies we e
conca ena ed as a ou -channel inpu , wi h a decision- usion app oach, whe e inal
p edic ions we e de i ed h ough a linea weigh ed sum om he p obabili ies ob ained
h ough ou independen p e- ained unimodal models. This s udy ein o ced he no ion
o he T1ce modali y’s signi icance in glioma sub ype classi ica ion. Mo eo e , i e ealed
ha any mul imodal app oach consis en ly ou pe o med unimodal models.
While b ain MRIs inhe en ly cap u e 3D da a, a no able obse a ion is ha o e 80% o he
s udies conduc ed hei analyses wi hin a 2D domain, ocusing on 2D MRI slices. None he-
less, some in es iga ions [54,57,212,221,222,227,229,232,233,238–240,246,249,
250,253,260–264,272,273] ha e ac i ely explo ed he signi icance o inco po a ing 3D
olume ic in o ma ion in o he ealm o b ain umo classi ica ion. While 3D olumes
inhe en ly cap u e in o ma ion om he h ee ana omical planes, 2D slices a e es ic ed o
a speci ic iew. Among he s udies ha adop ed a 2D app oach, only 44% p o ided de ails
066
abou he chosen ana omical plane. Among his subse , mo e han 50% u ilized he h ee
ana omical iews, whileo e 40%exclusi elyemployedaxial iews. Decomposing3D olumes
in o indi idual 2D slices may in oduce he po en ial o da a leakage. Main aining he
eliabili y o he analysis is c ucial o ob aining obus and us wo hy indings. I is wo h
no ing, howe e , ha only a limi ed numbe o s udies ha use mul iple 2D slices [61,130,
135,146,149,158,182,207,210,212,221,223,227,228,234,235,237,252,254,266,267],
explici ly de ailed hei app oach o da a spli ing a he pa ien le el, add essing his c i ical
conce n. An insigh ul compa ison was ca ied ou in he wo k o Badˇ
za and Ba jak a o i´
c
[137] be ween da a spli ing s a egies a he pa ien and image le els. The indings elucida e
ha an image-wise app oach yields accu acy esul s as high as 96% o b ain umo ype
classi ica ion, while a pa ien -le el spli demons a es a highe deg ee o eliabili y wi h an
accu acy o 88%. These esul s unde sco e he c i ical impo ance o u ilizing a pa ien -wise
aining app oach o assess he model’s gene aliza ion capaci y. Simila ly, Ghassemi e al.
[139], Ismael e al. [140] also p o ided e idence o supe io pe o mance when using an
image-wise spli , u he ein o cing he impo ance o hough ul da a spli ing and eliable
and anspa en pe o mance epo ing. I is also impo an o no e ha 3D models ope a e
on comple e 3D olumes and a e inhe en ly s uc u ed a he pa ien le el. This app oach
subs an ially educes he likelihood o da a leakage, he eby enhancing he eliabili y o he
analysis and ensu ing ha he esul s ai h ully ep esen he model’s pe o mance. This
aspec may p o ide a aluable pe spec i e when in e p e ing di e ences in accu acy be ween
3D and 2D models.
The in eg a ion o in o ma ion om a ious da a sou ces has ga ne ed g owing in e es in
he medical ield. B ain umo s, due o hei dis inc ea u es bo h a he his opa hological
and adiological le el, ha e mo i a ed nume ous s udies o explo e he syne gy be ween WSI
and MRI da a [54,57,260,264,273]. These in es iga ions consis en ly highligh he iche
in o ma ion con en in WSI as compa ed o MRI. Howe e , hey also e eal ha combining
da a om bo h sou ces leads o imp o ed o e all pe o mance in b ain umo
cha ac e iza ion. Ensemble lea ning me hods ha e also shown p omise in combining
p edic ions om mul iple DL models on MRI o imp o e o e all
pe o mance [36,152,155,164,172,173,184,189,192,195,210,212,216,269] and he
ou pu s o adiomics and DL models [219,241,267].
3.2.2 TARGETING GLIOMA GRADING
In his subsec ion, we del e deepe in o s udies ha ha e con ibu ed o ad ancing ou unde -
s anding o he classi ica ion o glioma g ades. Ea lie men ioned, he majo i y o he a icles
ha e assessed his ques ion om he dis inc ion o LGG e sus HGG [36,144,145,176,206,
210,221–223,227–229,231–233,235,236,238–246,248–256,258,267–269,272,274]
while se e al s udies assessed he exac g ade acco ding o he WHO CNS c i e ia [61,129,
132–134,144,150,172,180,224–226,230,234,237,240,247,253]. Tables A.1 and A.2
p o ide a comp ehensi e summa y o he da a, me hods, and esul s ob ained in each s udy
067
o he di e en ia ion o lowe -g ade gliomas e sus glioblas oma and he exac WHO g ade,
espec i ely. The median accu acy o di e en ia ing LGG and HGG is 95% when using a 2D
model and91% wi ha 3Dmodel. In con as , when di e en ia ing WHO g ades, heaccu acy
is 92% and 71%, espec i ely. In e es ingly, 3D models gene ally esul in lowe accu acy,
especially o WHO g ade di e en ia ion. This could be due o se e al ac o s, including he
ac ha 3D models ensu e no slice leakage is p esen . Mo eo e , 3D models equi e mo e
da a o ain e ec i ely due o i s complexi y and mo e compu a ional powe , wha can lead
o limi a ions such as small ba ch size o ewe aining epochs. Due o hese compu a ional
equi emen s, i is also common o dec ease he esolu ion o he images, which can lead o a
loss o de ails and impac he model’s abili y o lea n use ul disc imina i e pa e ns.
Yang e al. [223] p oposed he usage o AlexNe and GoogLeNe o classi ying LGG and
HGG on a p i a e da abase composed o 113 diagnosed glioma pa ien s. In his s udy, he 2D
T1ce axial images ha con ain a leas 80% o he umo isible we e selec ed. Then, he image
was c opped a he umo bounding box. The da a we e spli in o he ain, alida ion, and
es subse s a he pa ien le el. The bes pe o mance (96.80 ROC-AUC) was achie ed wi h
a p e- ained GoogLeNe . Despi e he good o e all pe o mance, only accu acy and ROC-
AUC we e epo ed which, along wi h he small sample size, a e limi ing ac o s in his wo k.
Ge e al. [258] p oposed a 3D CNN combined wi h a saliency-awa e s a egy o enhance
umo egions by educing he pixel in ensi ies ou side he b ain mask o one- hi d hei
o iginal alues. This model achie ed a sensi i i y o 90.48% o HGG and 86.67% o LGG,
wi h an o e all accu acy o 89.47%, compa ed o 84.21% wi hou mask enhancemen . In a
ela ed s udy, Pe ei a e al. [222] examined he e ec o s anda dizing ei he he en i e image
o solely he b ain egion o he classi ica ion o 3D LGG and GBM images. Thei indings
indica ed ha s anda diza ion, applied speci ically o b ain masks, signi ican ly imp o ed
pe o mance when he model was ained on he umo egion, inc easing accu acy om
87.70% o 95.24%. In e es ingly, G adCAM maps e ealed ha when in ensi y
s anda diza ion was applied ac oss he en i e image, he CNN conside ed he bo de o he
b ain as disc imina i e. This emphasizes he impo ance o using explainabili y me hods o
alida ion pu poses.
Tandel e al. [36] conduc ed an e alua ion o a ious 2D DL-based models o classi ying
LGG and GBM u ilizing Flai , T1, and T2 sequences independen ly. Thei indings
unde sco ed Flai as he mos pi o al sequence, ollowed by T2, and T1 as he leas
signi ican . In he li e a u e, s a egies in ol ing agg ega ion laye s, o combining sequences
as RGB image channels ha e been used o le e age in o ma ion om mul iple modali ies. In
Ge e al. [221], a 2D classi ie p ocessing T1ce, T2, and Flai sequences independen ly was
de eloped, wi h subsequen ea u e combina ions h ough a linea agg ega ion laye . The
epo ed esul s indica ed T2 as he leas in o ma i e sequence o glioma g ading (69.84%
accu acy), ollowed by Flai (75.40%), and T1ce (83.73%), wi h he agg ega ed model
yielding he highes accu acy (90.87%). Simila ly, Gu a e al. [61] epo ed consis en
indings and in oduced T1, which eme ged as he leas ele an sequence o g ade gliomas.
068
In Rui e al. [252], a combina ion o T1, Flai , and T1ce was le e aged as inpu image RGB
channels, achie ing supe io pe o mance (86%) when compa ed o independen o
dual-sequences combina ions ac oss a ious classi ica ion asks (LGG e sus GBM, g ade 2
e sus highe g ades, and molecula sub ypes). This app oach was u he ein o ced in
Coupe e al. [212] by inco po a ing T2, and i was concluded ha he combina ion o
sequences yielded be e esul s (86.38%) han indi idual sequences in he p oblem o
dis inguishing heal hy om glioma images. The s udy epo ed in Guo e al. [271] compa ed
he modali y-ensemble (87.80%) and modali y- usion (84.60%) me hodologies o
classi ying glioma ypes in o As ocy oma, Oligodend oglioma, and Glioblas oma, bo h
ou pe o ming independen sequences wi h 71.90% accu acy using T1, 73.30% wi h T2,
74.20% wi h Flai , and 83.30% wi h T1ce. O e all, hese s udies p o ide e idence o he
in o ma ion p o ided by di e en MRI sequences o b ain umo classi ica ion,
highligh ing ha a join ep esen a ion signi ican ly enhances model pe o mance.
Bane jee e al. [227] classi ied LGG and HGG mul i-sequence b ain MRIs om TCGA and
B aTS2017 da a using mul iple slice-based app oaches. In hei wo k, hey p o ided a
compa ison o he pe o mance ob ained wi h CNNs ained om sc a ch on 2D image
pa ches (Pa chNe ), en i e 2D slices (SliceNe ), and mul i-plana slices h ough a inal
ensemble me hod ha a e ages he classi ica ion ob ained om each ana omical iew
(VolumeNe ). The classi ica ion ob ained wi h hese models is also compa ed wi h
p e- ained VGGNe and ResNe on ImageNe . The mul i-plana me hod ou pe o med
he es o he app oaches wi h an accu acy o 94.74%, and he lowes accu acy (68.07%) was
ob ained wi h p e- ained VGGNe . Un o una ely, TCGA and B aTS da a sha e some
pa ien da a, which could in ol e an o e lap be ween aining and es ing samples and hence
be p one o da a leakage. Ding e al. [241] combined adiomics and DL ea u es using 2D
p e- ained CNNs using single-plane images and pe o ming a subsequen mul i-plana
usion. VGG16, in combina ion wi h adiomics and RF, achie ed he highes accu acy o
80% when combining he in o ma ion ob ained om he h ee iews. E en hough he
mul i-plana app oach p ocesses he in o ma ion ga he ed om he axial, co onal, and
sagi al iews, i is s ill essen ially a 2.5D app oach, weak a ully cap u ing 3D con ex s.
Zhuge e al. [232] p esen ed a p ope ly na i e 3D CNN o umo segmen a ion and
subsequen bina y glioma g ade classi ica ion and compa ed i wi h a p e- ained 2D
ResNe 50 on ImageNe wi h p e ious umo de ec ion, employing a Mask R-CNN. The
esul s o he 3D app oach we e sligh ly highe han he 2D ones, epo ing 97.10% and
96.30% accu acy, espec i ely. In hei s udy, Cha e jee e al. [272] explo ed he ole o
(2+1)D, mixed 2D-3D, and na i e 3D con olu ions based on ResNe . This s udy highligh s
he e ec i eness o mixed 2D-3D con olu ions achie ing an accu acy o 96.98%, su passing
bo h he (2+1)D and he pu e 3D app oaches. Fu he mo e, he use o p e- ained ne wo ks
demons a ed enhanced pe o mance in he spa ial models, ye , in iguingly, he pu e 3D
model pe o med be e when ained om sc a ch.
Glioma diagnosis and p ognosis a e signi ican ly in luenced by molecula ac o s, p omp ing
nume ous s udies o in es iga e he po en ial o DL in ex ac ing meaning ul ea u es om
069
MRI scans o classi ying hese gene ic amewo ks [184,221,227,228,233,248,249,252,
253]. Ge e al. [221] and Bane jee e al. [227] u ilized CNNs o dis inguish low- and high-
g ade gliomas and u he di e en ia e low-g ade gliomas wi h and wi hou 1p/19q codele-
ion. T ipa hi and Bag [248] p oposed a wo-le el classi ica ion sys em, i s ca ego izing he
umo g ade and hen p edic ing 1p/19q s a us o low-g ade umo s. Building upon his,
T ipa hi and Bag [249] expanded hei amewo k o a mul i- ask lea ning app oach, simul-
aneously add essing low and high umo g ading, as well as umo cha ac e iza ion based
on bo h IDH and 1p/19q s a us. This app oach enables join ea u e lea ning, acili a ing
he ne wo k o disce n pa e ns such as he absence o 1p/19q codele ion in IDH-wild ype
umo s. Mul i- ask lea ning has ga ne ed a en ion [233,253,275]. an de Voo e al. [253]
u he ex ended he wo-class g ade classi ica ion o p edic he WHO umo g ade (2/3/4)
simul aneously wi h IDH mu a ion and 1p/19q codele ion. Thei esul s demons a ed a
90% ROC-AUC o IDH mu a ion s a us, albei wi h a sensi i i y o 40% o g ade 4 glioma
pa ien s and 73% o g ade 2-3 glioma pa ien s. Addi ionally, hey achie ed an 85% ROC-
AUC o 1p/19q codele ion p edic ion, al hough wi h a sensi i i y o 39% speci ically o
g ades 2-3.
A ecen s udy ca ied by Jin e al. [256] compa ed he pe o mance o 35 neu osu geons
wi h and wi hou AI assis ance on glioma g ading. The 3D AI-based wi h a VGG backbone
sys em was ained on B aTS2020 wi h an accu acy o 88%. The AI-based ool showed he
physicians he p edic ion along wi h a colo map on he MRI o highligh he impo an
egions o he AI decision. The a e age accu acy imp o ed om 82.50% up o 87.70% wi h
he AI assis ance, and he e was no addi ional boos wi h he assis ance o pos -hoc
explana ions. Ins ead, he ea u e maps help he physicians e i y AI decisions bu ailed o
gi e explici easons. This indica es ha he exis ing AI pos -hoc explana ion ailed o
indica e o physicians when o ely on AI ecommenda ions. Wi h AI assis ance physicians’
pe o mance signi ican ly inc eased bu did no exceed AI alone. Acco ding o he esul s o
his s udy, a physician has a p obabili y o 67.20% o ha ing highe accu acy when assis ed
by AI p edic ion and 71.00% when he p edic ion is accompanied by a pos -hoc explana ion
han pe o ming he ask alone. In he expe imen s, doc o s and AI decisions ag eed wi h
each o he in 81% decisions. The decision ag eemen inc eased o 86.80% when physicians
we e assis ed by AI p edic ion and u he inc eased o 87.20% when inco po a ed he
hea map explana ion. Doc o s a ed a s a is ically highe us and willingness o use he AI
sys em a e iewing he AI-d i en model pe o mance me ics, howe e , he e was no
signi ican di e ence a e using AI wi h pos -hoc explana ions. The s udy also shows ha
when he e was a case o disag eemen be ween he clinician and he AI decisions, physicians
we e mo e likely o check he ea u e a ibu ion AI explana ion.
070
KEY TAKEAWAYS
•Open da ase s ha e played a c ucial ole
in ad ancing AI-d i en echnologies
o b ain umo diagnosis. O e
85% o he s udies e iewed be ween
2018 and 2024 u ilize public da a
o b ain umo classi ica ion asks,
emphasizing he ole o accessible da a
in suppo ing he e olu ion o AI
echnologies in heal hca e and enabling
hei benchma king. Howe e , he
size o hese da ase s ypically anges
in he hund eds, which may limi
he obus ness and eliabili y o he
conclusions. A la ge olume o da a is
necessa y o enhance he gene alizabili y
and accu acy o AI models in his ield.
•Cu en publicly a ailable da abases
con aining glioma-g ade labels a e
labeled acco ding o he WHO CNS
c i e ia eleased in 2016. Howe e ,
hese c i e ia ha e since been upda ed in
2021, esul ing in e-de ined ca ego ies.
Consequen ly, he exis ing publicly
a ailablelabelsa eou da ed,highligh ing
he need o upda ed da a.
•A comple e disclosu e o he da a
u ilized, he me hodologies employed,
and he esul s ob ained a e no
consis en ly obse ed ac oss all pape s,
highligh ing a lack o s anda dized
epo ing p ocedu es. Such anspa ency
is c ucial as i enhances us in esea ch
ou comes and signi ican ly imp o es
he ep oducibili y o he indings in
esea ch.
•The as majo i y o pape s u ilize 2D
app oaches a he han 3D me hods
o analyzing b ain umo da a. The
mos used da abase, Figsha e, p o ides
2D slices ins ead o ull 3D olumes.
Howe e , e en among s udies ha use
o he da abases o e ing 3D olumes,
he e is a no able p e e ence o 2D
app oaches. This may be a ibu ed
o he compu a ional e iciency and
ela i e simplici y o in e p e ing 2D
da a compa ed o 3D.
•The classi ica ion pe o mance o 2D
app oaches is gene ally highe compa ed
o 3D me hods, despi e he addi ional
spa ial con ex p o ided by 3D olumes.
Howe e , he use o 2D slice da a
poses a isk o slice leakage du ing da a
spli ing a he image le el, po en ially
comp omising he alidi y o he esul s.
•Nume ous s udies ha e demons a ed
he po en ial o MRI da a p e-p ocessing
echniques, such as skull-s ipping,
in ensi y no maliza ion, and umo
de ec ion, in p oducing mo e accu a e
esul s. This unde sco es he signi ican
ole o da a-cen ic app oaches in
enhancing he quali y o he analysis.
Howe e , he li e a u e o en p io i izes
model-cen ic pe spec i es, ocusing on
a chi ec u al imp o emen s a he han
on da a enhancemen .
071
4
THE DATA
4.1 DESCRIPTION
To conduc his esea ch, we in eg a ed da a om h ee publicly a ailable da ase s:
B aTS [118] (n=369), EGD [123] (n=716), and a subse o he Remb and da ase [276]
p o ided wi h umo segmen a ion masks (n=59). Da a om B aTS included g ade umo
labels ca ego ized in a bina y manne as lowe -g ade gliomas (g ades 2 and 3) and high-g ade
gliomas (g ade 4 o glioblas oma), p o ided alongside age and su i al da a. The B aTS
da ase con ains samples om TCGA-LGG and TCGA-GBM collec ions om he
TCIA [119,120] om which he g ade label (2, 3, and 4) unde he WHO CNS c i e ia is
a ailable. Demog aphic in o ma ion o B aTS samples, including sex, age, and gene ic
labels, was collec ed om he da a published by Cecca elli e al. [277]. The EGD images also
included clinical labels o sex, age, gene ic in o ma ion, and de ailed scan cha ac e is ics.
The Remb and da abase includes sex, age, ace, and su i al da a. All da a labeling om he
h ee da ase s was conduc ed unde he WHO CNS c i e ia es ablished in 2016 [7].
All he scans om he h ee da ase s a e dis ibu ed a e some p e-p ocessing s eps,
in ol ing co- egis e ed o he same ana omical empla e (RSI24 in B aTS and Remb and
and MNI152 in EGD), in e pola ed o a consis en esolu ion o 1mm3, and addi ionally,
B aTS images a e p o ided a e skull emo al. We u he applied some p e-p ocessing s eps
o ha monize he MRI scans. Images we e spa ially aligned ollowing he s anda d o he
RSI24 empla e. The e o e, all he scans we e esampled o a common esolu ion o
[240,240,155] wi h an iso opic oxel size (1mm3). Skull-s ipping was pe o med using
he HD-BET algo i hm [73], which is an au oma ic b ain ex ac ion ool based on a i icial
neu al ne wo ks ha yielded median Dice coe icien s abo e 96% o he ou MRI
modali ies. BFC was conduc ed using N4BiasFieldCo ec ionImageFil e () om
SimpleITK.
083
B aTS EGD Remb and To al
LGG 76 214 40 330
HGG 293 502 19 814
To al 369 716 59 1144
G.2 28 135 24 187
G.3 37 79 13 129
G.4 293 502 19 814
To al 358 716 59 1130
Table 4.1. G ade dis ibu ion ac oss da ase s.
Figu e 4.1. Flow diag am o pa icipan numbe s ini ially and pos -quali y assessmen
o MRI modali ies, alongside he implemen ed da a spli s a egy.
084
The ini ial in eg a ed da ase was composed o 1144 samples. A ho ough da a quali y
analysis was unde aken and images wi h comp omised quali y – whe he due o lawed
sequences, unca ion, o inadequa e depic ion o he umo egion – we e me iculously
iden i ied and subsequen ly emo ed om he da ase . This cu a ion p ocess was employed
o ensu e he in eg i y and eliabili y o he emaining samples o subsequen analyses and
model aining. The esul an da ase comp ised a o al o 1125 samples (see Figu e 4.1). To
main ain an independen e alua ion s anda d, a dis inc es se was cu a ed om he ini ial
25% o he en i e da ase . This sepa a e es se was ca e ully se aside a he ou se o he
expe imen and emained un ouched du ing he model aining and alida ion phases,
he eby p o iding an unbiased measu e o he model’s pe o mance on en i ely new da a.
Gi en he cons ain s o a limi ed da ase size, a 3- old c oss- alida ion (CV) me hodology
was adop ed o enhance he obus ness o he esul s. Each i e a ion o he 3- old CV
u ilized a ensu ed 2/3 o aining o acili a e a comp ehensi e unde s anding o he
unde lying ea u es. The emaining 1/3 se ed as a alida ion se o assess gene aliza ion
pe o mance. S a i ied sampling p ese ed he o iginal dis ibu ion. The pa i ion was
pe o med by man aining he same g ade and da abase a io in each se .
Table 4.2 p o ides a summa y o he pa ien demog aphics and class dis ibu ion o he
comple e sample o s udy as well as o he di e en da a pa i ion se s, including he h ee
olds o CV and he independen es se . Age was segmen ed in o 5-yea anges o align
wi h he Remb and da ase age a iable o ma . The s a is ical di e ences in pa ien
demog aphics (sex and age) be ween da a spli s we e assessed using he χ2 es . No
signi ican s a is ical di e ences we e ound among he di e en se s wi h ega d o sex o
age, indica ing ha he da a pa i ioning p ocess e ec i ely main ained demog aphic balance
ac oss he expe imen al subse s. This ensu es ha any obse ed di e ences in model
pe o mance a e no con ounded by a ia ions in pa ien cha ac e is ics.
085
Pa ame e To al Fold 1 Fold 2 Fold 3 Tes
No. o Pa ien s 1125 (100.00) 281 (24.98) 281 (24.98) 281 (24.98) 282 (25.06)
Sex 1
Female 354 (31.46) 82 (29.18) 93 (33.10) 88 (31.32) 91 (32.27)
Male 552 (49.06) 140 (49.82) 139 (49.47) 137 (48.75) 136 (48.23)
Unknown 219 (19.48) 59 (21.00) 49 (17.43) 56 (19.93) 55 (19.50)
Age 2
[10–14] 1 (0.09) 1 (0.36) 0 (0.00) 0 (0.00) 0 (0.00)
[15–19] 4 (0.36) 0 (0.00) 1 (0.36) 2 (0.71) 1 (0.35)
[20–24] 19 (1.69) 5 (1.78) 4 (1.42) 4 (1.42) 6 (2.13)
[25–29] 35 (3.11) 10 (3.56) 11 (3.91) 10 (3.56) 4 (1.42)
[30–34] 39 (3.47) 8 (2.85) 6 (2.14) 16 (5.69) 9 (3.19)
[35–39] 57 (5.07) 19 (6.76) 11 (3.91) 12 (4.27) 15 (5.32)
[40–44] 56 (4.98) 17 (6.05) 12 (4.27) 15 (5.34) 12 (4.26)
[45–49] 100 (8.89) 32 (11.39) 28 (9.96) 14 (4.98) 26 (9.22)
[50–54] 129 (11.47) 28 (9.96) 38 (13.52) 35 (12.46) 28 (9.93)
[55–59] 116 (10.31) 27 (9.61) 29 (10.32) 29 (10.32) 31 (10.99)
[60–64] 145 (12.89) 32 (11.39) 39 (13.88) 37 (13.17) 37 (13.12)
[65–69] 128 (11.38) 40 (14.23) 29 (10.32) 28 (9.96) 31 (10.99)
[70–74] 125 (11.11) 26 (9.25) 36 (12.81) 29 (10.32) 34 (12.06)
[75–79] 70 (6.22) 13 (4.63) 18 (6.41) 18 (6.41) 21 (7.45)
[80–84] 23 (2.04) 8 (2.85) 1 (0.36) 8 (2.85) 6 (2.13)
[85–89] 8 (0.71) 0 (0.00) 3 (1.07) 4 (1.42) 1 (0.35)
Unknown 70 (6.22) 15 (5.34) 15 (5.34) 20 (7.12) 20 (7.09)
G ade Label 3
LGG 320 (28.44) 80 (28.47) 80 (28.47) 80 (28.47) 80 (28.37)
HGG 805 (71.56) 201 (71.53) 201 (71.53) 201 (71.53) 202 (71.63)
Da a is shown as coun (pe cen age)
1P- alue: .89
2P- alue: .40
3P- alue: 1.0
Table 4.2. Demog aphic cha ac e is ics o e he di e en da a pa i ions.
086
4.2 EXPLORATORY DATA ANALYSIS
4.2.1 MRI DATA
Absolu e MRI in ensi y alues lack p ecise physical meaning, as oxel in ensi ies can a y
signi ican ly based on he scanne se ings and acquisi ion pa ame e s. Ins ead, in ensi y
alues in MRI a e ela i e, meaning hei in e p e a ion elies on compa isons wi hin he
same image ac oss di e en egions wi hin he image. Hence, MRIs a e no compa able
ac oss scanne s, subjec s, o e en s udies. Wi hin he EGD da abase, in o ma ion ega ding
scanne endo s is a ailable. The numbe o scans acqui ed wi h sys ems om each endo is
p esen ed in Table 4.3. Figu e 4.2 illus a es he pixel dis ibu ions o Siemens, Philips, and
GE. Toshiba was emo ed om his analysis since only one image was acqui ed using his
sys em manu ac u e .
Sys em Manu ac u e Coun
Siemens 306
Philips 231
GE Medical Sys ems 163
Toshiba 1
Table 4.3. F equency o imaging acquisi ion scanne manu ac u e s in EGD da ase .
Figu e 4.2. Pixel dis ibu ion o EGD lai MRIs ac oss imaging acquisi ion scans.
087
This ela i e na u e emphasizes he impo ance o assessing in ensi y alues wi hin he
con ex o he speci ic scan o s udy, a he han ha ing a s anda dized, uni e sally applicable
scale ac oss di e en MRI machines o da ase s. Since hese in ensi ies lack s anda dized
physical meaning due o a ia ions in scanne se ings and image acquisi ion pa ame e s,
hei in e p e a ion elies hea ily on compa isons wi hin he same image and ac oss di e en
egions wi hin ha image. Unde s anding and assessing in ensi y alues wi hin hei
pa icula con ex becomes c ucial o accu a e analysis and in e p e a ion in MRI s udies.
To p o ide a comp ehensi e unde s anding o he MRI da a, his sec ion explo es he pixel
dis ibu ion o he maximum umo 2D slices in each an omical iew and MRI modali y.
Tables 4.4,4.5,4.6 p esen he pixel dis ibu ion s a is ics o he sagi al, co onal, and axial
planes, espec i ely, encompassing bo h he en i e image and he b ain egion ac oss ou
comple e se o images. These ables a e complemen ed by he his og ams o all he images
shown in Figu es 4.3,4.4, and 4.5. Backg ound pixels we e consis en ly se o 0.0. F om
Tables 4.4,4.5,4.6 we can obse e ha 50% o he image pixels a e backg ound pixels and,
consequen ly, omi ed om he image in ensi y his og ams o enhance he isualiza ion o
he b ain pixel dis ibu ion. As p e iously men ioned, MRI in ensi y alues a e ela i e and
he maximum alue is unbounded.
F om he s a is ics, we can obse e a conside able dis ance be ween he 99 h pe cen ile and
he maximum pixel alue, which indica es he exis ence o ou lie s. Fo example, 99% o he
sagi al Flai 2D slices ha e a maximum pixel alue inside he b ain mask lowe han 2 834.59,
while he emaining 1% ha e a maximum pixel alue unde 25 502.74. The esul s o he
co onal, axial planes, and he es o he sequences a e analogous. No ably, pixel alues in
T1ce and T2 sequences a e highe , yielding b igh e a eas. Fu he mo e, in ensi y alues a y
among MRI sequences and planes, which di icul ies he di ec image compa ison o analysis
wi hou p e ious pixel alue s anda diza ion.
In ensi y no maliza ion aligns his og ams wi hin a compa able in ensi y ange, which helps
he ne wo k o con e ge in a mo e s able manne and has become a s anda d p e-p ocessing
s ep o he de elopmen o image-based DL models. Iden i ying he op imal no maliza ion
echnique can signi ican ly enhance he compa abili y and consis ency o he da a. In his
chap e , we p o ide a comp ehensi e o e iew o he impac o di e en da a no maliza ion
on ou da a dis ibu ion applied in an image- and sequence-wise ashion, meaning ha each
image was no malized independen ly. Figu e 4.6 illus a es he his og ams o 2D axial slices
using h ee no maliza ion echniques: min-max no maliza ion, s anda diza ion using he
whole image, and s anda diza ion inside he b ain mask. We can obse e ha each
no maliza ion me hod p esen s a dis inc dis ibu ion pa e n. Min-max no maliza ion
scales he da a o a ixed ange be ween 0 and 1. S anda diza ion shi s he da a o ha e ze o
min and uni a iance, c ea ing a s anda d no mal dis ibu ion. When he backg ound
pixels hold no signi icance o he ask a hand, such as in ou scena io whe e backg ound
pixels a e consis en ly se o 0.0, s anda dizing solely on he b ain mask eme ges as an
e icien app oach, minimizing he impac o insigni ican ea u es.
088
S a is ic Min Mean S d p50 p75 p90 p95 p99 Max
Image
Flai 0.00 244.20 634.85 0.00 306.31 752.72 1094.77 1963.24 25502.74
T1ce 0.00 311.65 759.10 0.00 364.10 764.38 1431.91 3719.42 27866.77
T1 0.00 261.29 584.49 0.00 347.72 720.52 1109.04 2429.50 25287.51
T2 0.00 291.99 706.25 0.00 387.48 821.47 1259.92 2580.09 29774.06
B ain Mask
Flai 1.36×10-30 592.97 878.53 372.35 738.40 1210.33 1578.26 2834.59 25502.74
T1ce 5.29×10-16 763.93 1032.95 429.64 755.13 1749.07 2809.08 4988.68 27866.77
T1 8.21×10-16 630.49 769.13 400.35 703.92 1236.54 1866.97 3119.36 25287.51
T2 9.11×10-42 710.88 957.36 468.04 807.27 1397.13 1879.82 3719.51 29774.06
Table 4.4. S a is ics compu ed om sagi al 2D MRIs con aining he la ges a ea o umo .
Figu e 4.3. His og ams o pixel alues ex ac ed om sagi al 2D MRIs con aining he la ges a ea o umo .
089
KEY TAKEAWAYS
•MRI in ensi y alues a e inhe en ly
ela i e and no di ec ly compa able
ac oss images. Va ia ions in in ensi y
can occu due o di e ences in
scanning p o ocols o MRI sys ems.
Consequen ly, applying image in ensi y
no maliza ion o s anda diza ion is
c ucial o ensu e ha hese alues a e
compa able ac oss scans.
•G ade 4 gliomas domina e he da ase ,
ep esen ing o e 70% o he o al
cases, ollowed by nea ly 17% o g ade
2 cases, and 11% g ade 3. This
signi ican imbalance migh imply
po en ial challenges o model aining,
pa icula ly in classi ying mino i y
classes.
•The incidence o gliomas is highe in
males compa ed o emales; howe e ,
he e a e no signi ican sex-based
di e ences in umo g ade dis ibu ion.
Bo h male and emale classes exhibi
compa able p opo ions ac oss di e en
g ades.
•Gliomaincidenceand umo agg essi eness
bo h inc ease wi h ad ancing age.
Glioblas omas, in pa icula , become
mo e p e alen in olde age g oups, wi h
a signi ican ise obse ed in indi iduals
aged 60 o 69.
•The p esence o absence o IDH
mu a ions show a signi ican associa ion
wi h umo g ade. G ade 2 and g ade
3 gliomas p edominan ly exhibi IDH
mu a ions, while he absence o he
mu a ion cha ac e izes g ade 4 gliomas.
•The co-dele ion o ch omosome a ms 1p
and 19q is equen ly obse ed in IDH-
mu an lowe -g ade umo s, wi h he
highes p e alence in g ade 2 umo s.
•Molecula ea u es, such as IDH
mu a ions and 1p/19q co-dele ion, a e
s ongly associa ed wi h lowe -g ade
gliomas. This obse ed pa e n aligns
wi h he cu en es ablished molecula
cha ac e is ics o gliomas.
096
5
PRELUDE:
A CLASSIFICATION OF
LOWER AND HIGH GRADES
5.1 PREAMBLE
Nume ous s udies in he li e a u e aim o di e en ia e glioblas omas om lowe -g ade
gliomas. Se e al wo ks ha e add essed he bene i s o p e-p ocessing me hods be o e eeding
he b ain MRI da a in o he neu al ne wo k o glioma g ade classi ica ion [36,222,258].
O he s ha e e alua ed he impac o augmen ing he da ase ’s size [133,144,228,236],
using p e- ained models [223,224,235], adding mo e 2D slices [227], o combining MRI
modali ies [61,212,221,250,252]. O he wo ks ha e cen e ed on e alua ing he
e ec i eness o di e en ne wo k con igu a ions in he glioma g ading
ask [172,210,226,227,237,242,254,268].
In his chap e , we ocus on e alua ing di e en s a egies o enhance MRI da a and
imp o e he accu acy and obus ness o dis inguishing be ween LGG and HGG. We will
assess he in o ma i eness o each ana omical plane by compa ing he model’s pe o mance
using di e en iews as inpu da a. Nex , we will explo e di e en da a no maliza ion
p ocedu es and examine he signi icance o he mul iple MRI sequences, including hei
usion as a s a egy o combine he use ul in o ma ion each sequence p o ides. We will
ho oughly e alua e he c i ical ole o sample quan i y and di e si y in model
gene alizabili y h ough da a augmen a ion and ans e lea ning. Addi ionally, we will
examine he impac o using a single slice con aining he maximum umo a ea e sus
inco po a ing mul iple slices, as well as compa ing he use o he en i e b ain MRI image o
ocusing solely on he umo egion. Gi en ha nume ous s udies ha e achie ed high
classi ica ion pe o mance using SoA ne wo ks, we will compa e he pe o mance o ResNe
and VGGNe backbones, as hey we e among he mos popula choices.
103
5.2 DATA
Fo each pa ien , he 2D slices wi h he la ges umo a ea ha e been selec ed om he
sagi al, co onal, and axial planes. This is achie ed by iden i ying he slices ha con ain he
maximum numbe o pixels wi hin he umo segmen a ion mask, which is p o ided wi h
he MRI images o each o he h ee da ase s. Figu e 5.1 illus a es he MRI modali ies and
umo masks om he maximum umo 2D slices o a speci ic pa ien o he h ee
ana omical iews.
Figu e 5.1. 2D MRI slices con aining he la ges umo a ea om (a) sagi al, (b)
co onal, and (c) axial iews.
To assess i addi ional in o ma ion is p o ided o he model by inco po a ing mo e slices, we
implemen ed an o e sampling echnique inspi ed by he s a egy p oposed by Bane jee e al.
[227]. This echnique also helps mi iga e he challenges posed by imbalanced da a. This
app oach en ailed he selec ion o 10 con iguous slices bo h be o e and a e he slice
con aining he la ges umo a ea, wi h a skip o 5 slices o high-g ade umo s and a skip o
104
2 slices o lowe -g ade umo s. This esul ed in 5 slices o high-g ade gliomas and 11 slices
o lowe g ades. An example o he slice selec ion app oach o a LGG and an HGG pa ien
is illus a ed in Figu e 5.2. The a ionale behind his s a egy was o balance he da a
dis ibu ion ac oss di e en umo g ades while adding mo e iews o he umo . No ice
ha he ep esen a i i y o he lowe -g ade gliomas is enhanced. A e he aining p ocess,
he ul ima e pa ien p edic ions a e de e mined h ough a majo i y o ing mechanism. I is
wo h cla i ying ha he spli o he images in ain, alida ion, and es se s ollowing a
3- old CV app oach was done on pa ien ID, p e en ing slices om he same pa ien s om
alling in o ain and alida ion/ es subse s.
Figu e 5.2. Example o he selec ed 10 con iguous slices bo h be o e and a e he
slice con aining he maximum umo a ea wi h a skip o 2 o (a) LGG and a skip o 5
slices o (b) HGG pa ien s.
105
Figu e 5.8 illus a es he p obabili y dis ibu ions o models (which is a di ec quan i a i e
indica ion o he model’s ce ain y abou i s p edic ions) ained on images s anda dized
using he mean and s anda d de ia ion ocusing on he b ain egion. The dis ibu ion o
models u ilizing Flai , T1, and T2 sequences a e p edominan ly cen e ed a ound 0.5,
sugges ing a deg ee o unce ain y in he p edic i e ou comes. In e es ingly, he p obabili y
dis ibu ion de i ed om he model inco po a ing he T1ce sequence, as well as conside ing
he usion o all he sequences, p esen s a mo e nuanced and sp ead-ou dis ibu ion ac oss
he en i e ange o 0 o 1, which is indica i e o enhanced con idence in hei abili y o
disce n be ween classes accu a ely. This obse a ion aligns logically wi h ou nume ical
esul s, as he u iliza ion o he T1ce sequence has consis en ly demons a ed supe io
pe o mance when applied o independen sequences, ollowed by he combina ion o all
modali ies.
Figu e 5.9 shows he beha io o he dis ibu ion a e adding da a augmen a ion ans o ms
o he aining se as pa o he p e-p ocessing pipeline o he model using he usion o he
ou sequences. The esul s show a no able shi in he p obabili y dis ibu ion owa ds he
ex emi ies. This shi sugges s inc eased con idence in he model’s p edic ions, as he
augmen ed da ase in oduces a iche a ie y o scena ios o he model o lea n om. The
b oade and mo e p onounced dis ibu ion e lec s he enhanced p edic i e ce ain y.
Nume ical esul s a e p esen ed in Table A.9. O e all, hese indings unde sco e he
signi icance o sequence selec ion and da a augmen a ion in in luencing he pe o mance
and con idence le el o he model.
112
Figu e 5.7. G adCAM a en ion maps ac oss di e en p e-p ocessing and sequences.
113
Figu e 5.8. Model’s ou pu p obabili y dis ibu ion ac oss independen sequences and
hei usion, using mean-s d s anda dized images wi hin he b ain egion.
114
Figu e 5.9. Mul i-modal model’s ou pu p obabili y dis ibu ion inco po a ing da a
augmen a ion.
Addi ional p e-p ocessing s a egies such as c opping he image o he umo ROI, adding
mul iple slices om he same pa ien , and employing ine- uning om p e- ained models
on ImageNe ha e also been conside ed o enhance he dis inc ion be ween LGG and HGG
cases. Table 5.2 shows he esul s ob ained. As pe ou expe imen s, u ilizing only he umo
egion e sus he en i e b ain image enhances he ue HGG posi i e a e in 5% while no
signi ican di e ences a e obse ed in LGG sensi i i y. On he o he hand, adding mul iple
slices o using ImageNe knowledge do no seem o con ibu e subs an ial ele an ea u es
o he model o be e disce n be ween LGG and HGG pa ien s. These esul s imply ha
ocusing on he immedia e icini y o he la ges umo egion is su icien o he model o
make accu a e g ade classi ica ions. Including mo e slices may no yield disce nible
imp o emen s in he classi ica ion ask.
Accu acy ROC-AUC P ecision Recall F1
Tumo ROI 87.70 ±0.008 92.10 ±0.002
LGG - - 75.70 83.88 73.87
HGG - - 93.30 89.27 87.83
Mul i-slice 82.97 ±0.011 88.03 ±0.005
LGG - - 65.77 83.73 73.63
HGG - - 92.80 82.70 87.43
Fine- uning 82.27 ±0.013 86.77 ±0.005
LGG - - 66.43 77.07 69.50
HGG - - 90.33 84.33 85.30
Table 5.2. Mean 3- old CV pe o mance on he es se o he classi ica ion o LGG
s. HGG ob ained using he umo ROI, mul i-slice o e sampling, and ine- uning om
ImageNe .
115
5.4.3 EVALUATING NETWORK ARCHITECTURES
Pa allel expe imen s we e conduc ed employing a ious a chi ec u es, such as ResNe -34,
VGGNe -11, and VGGNe -16 o e alua e and compa e hei pe o mance in e ms o
accu acy me ics and compu a ional e iciency. Tables A.10 o A.12 p esen he pe o mance
ac oss hese a chi ec u es using he p e iously assessed p e-p ocessing me hods. In essence,
all he a chi ec u es pe o med be e in classi ying LGG e sus HGG when employing
mean-s d s anda diza ion compa ed o min-max no maliza ion. The e alua ion esul s using
he images wi h mean-s d s anda diza ion on he b ain mask, ex ac ed om he es
con usion ma ices and epo ed as 3- old CV a e ages, a e displayed in Table 5.3. The
accu acy and ROC-AUC alues a e 79.43 and 88.20, espec i ely, using he ResNe -18
backbone, which imp o es o 83.00 and 89.20 wi h ResNe -34. Simila ly, VGGNe -11 and
VGGNe -16 achie ed accu acy and ROC-AUC alues o 84.73 and 89.87, and 83.00 and
89.53, espec i ely. Al hough VGGNe -11 demons a es a sligh ly highe accu acy,
VGGNe -16 p o ides a mo e balanced classi ica ion o bo h lowe and high-g ade classes.
Speci ically, VGGNe -16 co ec ly classi ies 82.50% o LGG cases, showing a 6.27%
imp o emen o e VGGNe -11, and 84% o HGG cases, a dec ease o 3.97 compa ed o
VGGNe -11. In he subsequen analyses, we will u ilize VGGNe -16 as he backbone model.
Rega ding he compu a ional e iciency, Table 5.4 shows he numbe o pa ame e s and he
aining ime o each ne wo k a chi ec u e. Al hough ResNe -34 has ewe pa ame e s han
VGGNe s, i equi es mo e aining ime. This sugges s ha he a chi ec u al complexi y
and he p esence o esidual connec ions in ResNe models can impac he aining
du a ion.
Accu acy ROC-AUC P ecision Recall F1
ResNe -18 79.43 ±0.013 88.20 ±0.005
LGG 59.90 83.73 69.10
HGG 92.37 77.73 82.70
ResNe -34 83.00 ±0.006 89.20 ±0.010
LGG 67.17 78.33 72.43
HGG 90.93 84.80 86.73
VGGNe -11 84.73 ±0.008 89.87 ±0.003
LGG 71.87 76.23 74.03
HGG 90.40 88.10 88.30
VGGNe -16 83.70 ±0.013 89.93 ±0.004
LGG 67.57 82.50 72.10
HGG 92.40 84.13 86.00
Table 5.3. Mean 3- old CV pe o mance on he es o he classi ica ion o LGG s.
HGG using ResNe -18, ResNe -34, VGGNe -11, and VGGNe -16.
116
Pa ame e coun T aining ime
ResNe -18 11 177 602 1h 18 min
ResNe -34 21 285 762 1h 57 min
VGGNe -11 128 780 034 1h 27 min
VGGNe -16 134 277 186 1h 46 min
Table 5.4. Compa ison o model pa ame e s and aining ime o ResNe -18, ResNe -
34, VGGNe -11, and VGGNe -16.
5.4.4 EVALUATING THE SAMPLE SIZE
Table 5.5 shows he esul s on he es se ac oss di e en pe cen ages o he sample size. The
ROC-AUC imp o ed signi ican ly om 70% wi h 25% o he da a o 87% wi h he en i e
da a se . Simila ly, he sensi i i y o lowe -g ade gliomas showed an ini ial alue o 38%
expe iencing a subs an ial ise o 77%. As he da ase size dec eases, he e is an obse able
bias in he classi ica ion abili y owa ds he majo i y class. Conside ing he esul s ob ained
in he p e ious sec ion, e en enhanced wi h da a augmen a ion boos ing he ecall o
lowe -g ade class 82.50%. Gi en he imbalanced na u e o ou da ase (70%–30%), i is
essen ial o exe cise cau ion when in e p e ing me ics, as he minimum accu acy o 70% can
be achie ed when he model is en i ely biased owa ds he majo i y class and ails o classi y
he mino i y g oup.
Accu acy ROC-AUC P ecision Recall F1
25% 70.07 ±0.021 70.70 ±0.067
LGG 46.00 38.73 54.00
HGG 77.40 82.50 84.00
50% 80.13 ±0.003 83.83 ±0.004
LGG 67.33 58.70 71.57
HGG 84.47 88.60 88.43
75% 82.73 ±0.013 86.90 ±0.002
LGG 68.73 72.13 70.60
HGG 88.70 86.93 86.60
100% 83.00 ±0.012 87.87 ±0.001
LGG 67.43 77.90 72.67
HGG 90.70 84.97 86.40
Table 5.5. Mean 3- old CV pe o mance on he es se o he classi ica ion o LGG
s. HGG conside ing di e en sample sizes.
117
KEY TAKEAWAYS
•The axial plane p o ed o be mo e
e ec i e han he sagi al and co onal
planes in highligh ing ele an ea u es
o dis inguishing be ween LGG and
HGG cases in a 2D con ex .
•Image mean-s d s anda diza ion
ou pe o ms min-max no maliza ion o
in ensi y homogeniza ion ac oss scans.
This enhancemen is u he achie ed by
emo ing backg ound pixels om he
compu a ion and concen a ing solely
on he b ain egion. This imp o emen
is e lec ed no only in quan i a i e
me ics bu also in he ne wo k’s
inc eased a en ion o he umo a ea.
Addi ionally, con as enhancemen
does no o e subs an ial bene i s o he
classi ica ion.
•T1ce eme ged as he mos e ec i e
indi idual MRI modali y, ollowed
by Flai , T1, and, las ly T2, in
highligh ing ele an imaging ea u es
o he classi ica ion o LGG and
HGG cases. The mul i-sequence
app oach, signi ican ly enhances he
model’s pe o mance, pa icula ly
in dis inguishing LGG cases. This
imp o emen is e iden in he
quan i a i e esul s and he inc eased
con idence in he model’s p obabili y
ou pu s. Addi ionally, he ne wo k’s
a en ion appea s o be mo e
concen a ed on he umo a ea when
using T1ce and FLAIR sequences.
•Focusing on he 2D slice con aining
he la ges umo a ea is su icien o
accu a e di e en ia ion be ween lowe
and high-g ade gliomas. The addi ion o
addi ional slices does no show u he
classi ica ion imp o emen s.
•Using Kaiming weigh ini ializa ion
p o ed o be a mo e e ec i e s a egy
han ine- uning he weigh s om
p e- ained models on ImageNe
da a. This s a egy p o ided a be e
s a ing poin o aining, yielding
imp o ed con e gence and mo e
accu a e classi ica ion.
•Focusing on he umo egion in b ain
MRI, a he han using he en i e
image, enhanced he model’s abili y o
di e en ia e be ween lowe and high-
g ade gliomas by emphasizing he mos
ele an ea u es, he eby educing he
noise om non- umo ous a eas.
•Among he a iousne wo ka chi ec u es
e alua ed o classi ying LGG and HGG
g oups, ResNe -34 demons a ed be e
pe o mance han ResNe -18, which
was u he imp o ed using VGGNe -
11 and VGGNe -16, his achie ing he
bes classi ica ion o LGG cases.
•Inc easing he da ase ’s size signi ican ly
enhances he model’s classi ica ion
pe o mance, pa icula ly in enhancing
he sensi i i y o lowe -g ade gliomas,
which cons i u e he mino i y class.
Smalle da ase s may lead o a bias
owa ds he majo i y class, highligh ing
he impo ance o la ge and di e se
da ase s o obus classi ica ion.
118
Figu e A.4. Con usion ma ices using he mul i-plana model o he age 40−59 g oup.
Figu e A.5. Con usion ma ices using he mul i-plana model o he age ≥60 g oup.
Figu e A.6. Con usion ma ices using he mul i-plana model o he IDH wild- ype
g oup.
224
Figu e A.7. Con usion ma ices using he mul iplana model o he IDH mu a ed
g oup.
Figu e A.8. Con usion ma ices using he mul i-plana model o he 1p/19q in ac
g oup.
Figu e A.9. Con usion ma ices using he mul i-plana model o he 1p/19q co-
dele ed g oup.
225
C. MODELS’
HYPERPARAMETERS
The hype pa ame e s in ol ed in aining he models needed o be uned o achie e he bes
pe o mance. In his sec ion, we p esen he hype pa ame e s uned in each classi ica ion
p oblem and he alues ha lead o he bes pe o mance.
226
C.1 TESTED HYPERPARAMETERS
Pa ame e Values
Lea ning a e
ini ial lea ning a e 1×10−3,1×10−4,5×10−5,1×10−5
schedule pa ience 5,10,15
ac o 0.25,0.5
Ba ch size 8,16,32,64
Augmen a ion ac o 1,2,3
D opou a e 0.10,0.20,0.25,0.5
Op imize Adam, SGD
Weigh decay 1×10−3,1×10−4,5×10−5,1×10−5
Momen um 0.50,0.80,0.90
Weigh ini ializa ion Xa ie , Kaiming, ImageNe TL
Table A.21. Hype pa ame e s uned along wi h he es ed alues.
227
C.2 OPTIMAL HYPERPARAMETERS
SLICE-BASED MODELS
Pa ame e LGG s. HGG G.2 s. G.3 G.3 s. G.4 G.2 s. G.4 WHO G ade
Lea ning a e
ini ial alue 1×10−45×10−51×10−51×10−51×10−4
pa ience 5 5 5 5 5
ac o 0.5 0.5 0.5 0.5 0.5
Ba ch size 16 16 16 16 16
Augmen a ion ac o 2 2 2 2 2
D opou a e 0.5 0.5 0.5 0.5 0.5
Weigh decay 1×10−31×10−31×10−31×10−31×10−3
Momen um 0.9 0.9 0.9 0.9 0.9
MULTI-PLANAR MODELS
Pa ame e LGG s. HGG G.2 s. G.3 G.3 s. G.4 G.2 s. G.4 WHO G ade
Lea ning a e
ini ial alue 1×10−45×10−55×10−55×10−51×10−4
pa ience 5 5 5 5 5
ac o 0.5 0.5 0.5 0.5 0.5
Ba ch size 8 8 8 8 8
Augmen a ion ac o 2 2 2 2 2
D opou a e 0.1 0.1 0.1 0.1 0.1
Weigh decay 1×10−31×10−31×10−31×10−31×10−3
Momen um 0.9 0.9 0.9 0.9 0.9
Linea uni s 4096 1024 1024 1024 4096
228
VOLUME-BASED MODELS
Pa ame e LGG s. HGG G.2 s. G.3 G.3 s. G.4 G.2 s. G.4 WHO G ade
Lea ning a e
ini ial alue 5×10−45×10−55×10−55×10−55×10−4
pa ience 5 5 5 5 5
ac o 0.5 0.5 0.5 0.5 0.5
Ba ch size 4 4 4 4 4
Augmen a ion ac o 1 1 1 1 1
D opou a e 0.5 0.5 0.5 0.5 0.5
Weigh decay 1×10−31×10−31×10−31×10−31×10−3
Momen um 0.9 0.9 0.9 0.9 0.9
229
