Imp o ing Rec al Tumo Segmen a ion wi h Anomaly Fusion De i ed
om Ana omical Inpain ing: A Mul icen e S udy
Lishan Cai a, b, Mohamed A. Abdela y a, c, Luyi Han a, d, Doenja M. J. Lamb eg s a, Joos an
G ie huysen a, Edua do Pooch a, b, Regina G.H. Bee s-Tan a, b, Sean Benson a, e, Jo en B unek ee , h, †,
Jonas Teuwen , g, h, †, *
a Depa men o Radiology, Ne he lands Cance Ins i u e, The Ne he lands
b GROW School o Oncology and De elopmen al Biology, Maas ich Uni e si y Medical Cen e,
The Ne he lands
c Depa men o Diagnos ic and In e en ional Radiology, Kas Al-Ainy Hospi al, Egyp
d Depa men o Radiology and Nuclea Medicine, Radboud Uni e si y Medical Cen e, The
Ne he lands
e Depa men o Ca diology, Ams e dam Ca dio ascula Sciences, Ams e dam Uni e si y Medical,
Cen e, The Ne he lands
Depa men o Radia ion Oncology, Ne he lands Cance Ins i u e, The Ne he lands
g Radboud Uni e si y Medical Cen e , Depa men o Medical Imaging, The Ne he lands
h Uni e si y o Ams e dam, Facul y o Science, The Ne he lands
* Co esponding au ho
† Sha ed las au ho
Abs ac
Accu a e ec al umo segmen a ion using magne ic esonance imaging (MRI) is pa amoun o
e ec i e ea men planning. I allows o olume ic and o he quan i a i e umo assessmen s,
po en ially aiding in p ognos ica ion and ea men esponse e alua ion. Manual delinea ion o ec al
umo s and su ounding s uc u es is ime-consuming and ypically. O e he pas ew yea s, deep
lea ning has shown s ong esul s in au oma ed umo segmen a ion in MRI. Cu en s udies on
au oma ed ec al umo segmen a ion, howe e , ocus solely on umo al egions wi hou conside ing
he ec al ana omical en i ies and o en lack a solid mul icen e ex e nal alida ion. In his s udy, we
imp o ed ec al umo segmen a ion by inco po a ing anomaly maps de i ed om ana omical
inpain ing. This inpain ing was implemen ed using a U-Ne -based model ained o econs uc a
heal hy ec um and meso ec um om p os a e T2-weigh ed images (T2WI). The ec al anomaly maps
we e gene a ed om he di e ence be ween he o iginal ec al and econs uc ed pseudo-heal hy
slices du ing in e ence. The de i ed anomaly maps we e used in he downs eam umo segmen a ion
asks by using hem as an addi ional inpu channel (AAnnUNe ). Al e na i e me hods o in eg a ing
ec al ana omical knowledge we e e alua ed as baselines, including Mul i-Ta ge nnUNe
(MTnnUNe ), which added ec um and meso ec um segmen a ion as auxilia y asks, and Mul i-
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
NOTE: This p ep in epo s new esea ch ha has no been ce i ied by pee e iew and should no be used o guide clinical p ac ice.
Channel nnUNe (MCnnUNe ), which u ilized ec um and meso ec um masks as an addi ional inpu
channel. As pa o his s udy, we benchma ked nine models o ec al umo segmen a ion on a la ge
mul icen e da ase o p eope a i e T2WI as he baseline and nnUNe ou pe o med he o he eigh
models on he ex e nal da ase . The MTnnUNe demons a ed imp o emen s in bo h supe ised and
semi-supe ised se ings (AI-gene a ed ec um and meso e um we e used) compa ed o nnUNe ,
while he MCnnUNe showed bene i s only in he semi-supe ised se ing. Impo an ly, anomaly
maps we e s ongly associa ed wi h umo al egions, and hei in eg a ion wi hin AAnnUNe led o he
bes umo segmen a ion esul s ac oss bo h se ings. The e ec i eness o AAnnUNe demons a ed
he alue o he anomaly maps, indica ing a p omising di ec ion o imp o ing ec al umo
segmen a ion and model obus ness o mul icen e da a.
1. In oduc ion
Magne ic Resonance Imaging (MRI) plays a pi o al ole in s aging ec al cance and selec ing
ea men plans, p o iding aluable in o ma ion ega ding he ex en o umo in il a ion wi hin and
beyond he bowel wall, and in o c i ical ana omical s uc u es, including pe i ec al essels, he
meso ec al ascia (MRF), pe i oneum, and neighbo ing pel ic o gans (Bee s-Tan e al., 2018;
MERCURY S udy G oup, 2007). T2-weigh ed imaging (T2WI) o ms he mains ay o he MRI
p o ocol because o i s supe io so issue con as o disce n he di e en laye s o he ec al wall,
meso ec al a , adjacen essels, and MRF o allow o de ailed local s aging (Ho a e al., 2019;
Suzuki e al., 2008).
P ecisely segmen ing he umo is an impo an ask in ec al cance managemen . Tumo
segmen a ions a e u ilized o se e al pu poses including adia ion ea men planning, olume ic
analysis, and ex ac ion o imaging bioma ke s, which may se e as a basis o p ognos ica ion and
ea men esponse e alua ion. Manual ec al umo delinea ion by expe ienced adiologis s is
conside ed he cu en gold s anda d. Ne e heless, i is ime-consuming and subjec o subs an ial
in a- and in e -obse e a ia ion (Hea n e al., 2020; I ing e al., 2016; T ebeschi e al., 2017).
De eloping an accu a e, gene alizable, and obus ec al umo segmen a ion model can help educe
his a iabili y and assis in s anda dizing se e al s eps o diagnos ic and he apeu ic ec al cance
managemen .
Deep lea ning (DL) has seen a apid up ake in se e al ields, achie ing s a e-o - he-a esul s in
mul iple medical image analysis asks (Razzak e al., 2018). In pa icula , Con olu ional Neu al
Ne wo k-based (CNN) based DL app oaches excel in lea ning image ep esen a ions om anno a ed
da a by using lea nable ea u e ex ac ion il e s and sequen ial con olu ion, ac i a ion, and pooling
ope a ions. Among CNN a chi ec u es, he U-Ne (Ronnebe ge e al., 2015) and i s a ian s a e he
mos popula a chi ec u e in medical image segmen a ion. S udies (Jian e al., 2018; Knu h e al.,
2022; Wang e al., 2018) ha e explo ed he abili y o U-Ne and o he 2D CNNs on ec al umo
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
segmen a ion based on T2WI. These CNNs achie ed a Dice simila i y coe icien sco e (DSC)
anging om 0.59 o 0.84 o umo segmen a ion. Al hough 2D CNNs a e less compu a ionally
expensi e, 3D models can le e age iche con ex o imp o e p edic ions (Mlyna ski e al., 2019).
(Hamabe e al., 2022) implemen ed a 3D U-Ne , achie ing an a e aged DSC o 0.73 (0.60-0.80)
o e 10- old c oss- alida ion o ec al umo segmen a ion. Howe e , no ex e nal alida ion was
done in hei s udy. Besides CNN-based models, ans o me -based a chi ec u es a e also being
applied in medical image analysis due o hei abili y o access long- ange seman ic in o ma ion (Xiao
e al., 2023). (Li e al., 2023) p oposed RTAU-Ne , a no el 3D dual pa h usion ne wo k con aining a
ans o me encode o ex ac ing global con ou in o ma ion o he umo . RTAU-Ne achie ed an
a e aged DSC o 0.80 and 0.68 in da a om wo espec i e medical cen e s. Howe e , RTAU-Ne
equi es he manual emo al o umo - ee slices, which hinde s he ully au oma ed implemen a ion
o he model. Addi ionally, RTAU-Ne was no compa ed wi h he s a e-o - he-a medical
segmen a ion ne wo ks, such as nnUNe (Isensee e al., 2021), a sel -con igu ing implemen a ion o
he U-Ne a chi ec u e, o nnFo me (Zhou e al., 2023), which in oduces 3D ans o me blocks on
op o nnUNe .
Besides ec al umo delinea ion, some s udies also demons a ed ha CNNs can accu a ely
delinea e ana omical s uc u es such as ec um and meso ec um o pe i ec al a wi h DSC abo e 0.90
(DeSil io e al., 2023; Hamabe e al., 2022; Kim e al., 2019). Au oma ed ec um and meso ec um
delinea ion could po en ially imp o e adiological e alua ion. The p ognosis o ec al cance depends
on how a he umo has in il a ed he laye s o he ec al wall and he meso ec um, and he
success ul a ainmen o nega i e ci cum e en ial esec ion ma gins (CRMs) h ough su gical
in e en ion (Nag egaal e al., 2004). Addi ionally, Lee e al. (Lee e al., 2019) demons a ed ha a 2D
model’s a iance in umo egions can be educed by 90% by inco po a ing ec al segmen a ion on he
model’s objec i e. In eg a ing ec al ana omical knowledge can p o ide a mo e comp ehensi e
ep esen a ion o he T2WI, allowing he model o lea n iche and mo e nuanced pa e ns, leading o
be e pe o mance on unseen da a. Howe e , he impac o adding ec al ana omical s uc u es
including meso ec um o ec al umo segmen a ion has no been in es iga ed in a mul i-ins i u ional
se ing. Also, inco po a ing ec al ana omical s uc u e has so a been limi ed o adding addi ional
segmen a ion asks as p esen ed in (Lee e al., 2019).
Unlike o medical challenges wi h public da ase s like b ain umo segmen a ion (Menze e al., 2015)
o clinically signi ican p os a e lesion segmen a ion (PICAI) (Saha e al., 2023), he e is no la ge
mul icen e publicly a ailable MRI da ase o ec al cance s udies. This makes i di icul o
benchma k di e en models. An ex ensi e ex e nal alida ion s udy wi h mul icen e da a is highly
desi able o compa e di e en deep lea ning segmen a ion app oaches.
In his a icle, we had he ollowing con ibu ions:
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
• We de eloped and e alua ed a ec al umo segmen a ion model inco po a ing anomaly maps
om ana omical inpain ing, showing imp o ed pe o mance.
• To gene a e he anomaly maps, we p oposed and e alua ed a no el end- o-end ec al
ana omical inpain ing model, ained exclusi ely on p os a e T2WI, o de ec and highligh
anomalous a eas in ec al T2WI. The model ained on p os a e T2WI was applied o ec al
T2WI, challenging adi ional domain-speci ic p ac ices and demons a ing he po en ial o
ans e lea ning. The de i ed ec al anomaly maps can also be in eg a ed in o o he clinical
downs eam asks.
• As pa o ou s udy, we benchma ked nine 3D deep lea ning models o ec al umo
segmen a ion on a la ge mul icen e da ase o T2WI.
• We de eloped and e alua ed a 3D deep lea ning model speci ically o segmen ec al
ana omical s uc u es, including ec um and meso ec um.
• We explo ed di e en s a egies o inco po a ing ec al ana omical in o ma ion in o ec al
umo segmen a ion, including he in eg a ion o anomaly maps de i ed om he inpain ing
model, adding ec al s uc u es as addi ional asks, and u ilizing hem as p io knowledge.
• We eleased he ec um and meso ec um masks o 100 p os a e T2WIs om PICAI da ase ,
anno a ed by adiologis .
• We uploaded he model weigh s o ec um and meso ec um segmen a ion, as well as he
weigh s o MTnnUNe , MCnnUNe , and AAnnUNe .
2 Rela ed Wo k
2.1 Recons uc ion-based Medical Anomaly De ec ion
Supe ised lea ning equi es a subs an ial amoun o eliably labeled da a, which is o en di icul o
collec in medical imaging. The e o e, me hods equi ing pa ly labeled da a (semi-supe ised) o no
labeling (unsupe ised me hods) ha e a ac ed inc eased a en ion. Anomaly de ec ion is a me hod
ha can use semi-supe ised o unsupe ised echniques o add ess medical imaging asks such as
segmen a ion. Gene a i e models a e equen ly used in he ield o anomaly de ec ion due o hei
e ec i eness. The mode is o en ained o econs uc images om a speci ic da a dis ibu ion (e.g.,
no mal issue). These models, when con on ed wi h images ou side o his dis ibu ion (such as hose
con aining umo s), o en s uggle o accu a ely econs uc he anomalous egions, esul ing in highe
econs uc ion e o s in hose a eas Gene a i e ad e sa ial ne wo ks (GANs) (Good ellow e al.,
2014), Au o-Encode s (AE), and hei a ian s including Va ia ional AE (VAE) (Kingma and
Welling, 2022) and Vec o Quan ized VAE (VQ-VAE) ( an den Oo d e al., 2017) we e p o en o be
p omising (Bau e al., 2021; Chen e al., 2020; Pinaya e al., 2022) in his ield. Howe e , hese
me hods ha e p esen ed se e al challenges. They s uggle o lea n and ep esen heal hy ana omical
s uc u es when using ull image esolu ion. Addi ionally, when econs uc ing he en i e image, he
highligh ed egions migh no exclusi ely co espond o diseased a eas. As a esul , he inal anomaly
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
maps may become mo e sensi i e o o he a i ac s. Finally, i i is in he sel -supe ised se ing whe e
heal hy o no mal samples a e no a ailable o aining, hese me hods can some imes can
success ully econs uc anomalies due o a high gene aliza ion capaci y (Gong e al., 2019).
2.2.1 Pa ial Image Recons uc ion ia Inpain ing o Medical Anomaly De ec ion
Ins ead o image- o-image econs uc ion me hods, inpain ing ocuses on illing in missing o
occluded pa s o an image by le e aging he su ounding con ex . The masks used o inpain ing a e
gene ally independen o he da ase . (Nguyen e al., 2021) implemen ed an inpain ing-based b ain
umo segmen a ion pipeline o T1-weigh ed MRI, whe e anomalous egions we e de e mined by
iden i ying a eas o highes econs uc ion loss. Wi h p io ana omical knowledge, inpain ing can
ocus solely on high- isk egions, educing he impac om he backg ound. (Yeganeh e al., 2022)
de eloped an ana omy-awa e masking s a egy o inpain ing o e ec i ely aid he model in lea ning
he shape ep esen a ion o he o gans o in e es . (Woo e al., 2024) ha e p oposed a UNe -based
model o de ec ing bone lesions in knee MR images h ough econs uc ion ia inpain ing and
demons a ed ha de ec ed anomalies can be u he u ilized o segmen a ion. The co e idea o
anomaly de ec ion h ough ana omical inpain ing in ol es masking he egion o in e es (ROI),
ypically co e ing he ele an ana omical s uc u es. The inpain ing model is ained o ill in he
masked a ea wi hou po en ial anomalies. The disc epancy be ween he o iginal and econs uc ed
images is hen u ilized o iden i y anomalies.
In his s udy, he ana omical s uc u es s ongly associa ed wi h ec al cance , including he ec um
and meso ec um ( a y issue su ounding he ec um), we e masked, and he inpain ing model was
ained using p os a e T2WI images om he PICAI da ase (Saha e al., 2023). Despi e he di e en
ields o iew be ween p os a e T2WI and ec al T2WI, hey o e lap in ana omical s uc u es. Mos
p os a e T2WI con ains a heal hy ec um and meso ec um, ensu ing he inpain ing model was ained
o lea n he dis ibu ion o he heal hy issues. The in e ed econs uc ed ec al T2WI slices can hen
be used o gene a e anomaly maps, highligh ing po en ially umo al a eas.
3. Me hods
3.1 Rec al Anomaly De ec ion wi h Masked T2WI using Ana omical Inpain ing
The o e all pipeline, inspi ed by (Han e al., 2024, 2023), o de ec ing ec al anomalies is shown in
Figu e 2. P os a e T2WI wi h masked ec um and meso ec um we e used o ain he model o hei
econs uc ion. The inpain ing model was adap ed om (Han e al., 2024), an end- o-end MRI
sequence gene a ion amewo k. The amewo k consis s o wo s ages. In he i s s age, only he
econs uc ion loss is op imized o he encode and gene a o . In he second s age, bo h ad e sa ial
loss and cycle-consis en loss a e inco po a ed in addi ion o he econs uc ion loss, wi h he
op imiza ion applied o he encode , gene a o , and disc imina o . The aining o he ana omical
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
inpain ing was based on 2D slices. The inpain ing model con ains an encode 𝑬 and a decode 𝑮. A
masked ( ec um and meso ec um) 2D T2WI slice 𝑋, can be comp essed by 𝑬 in o a la en space 𝑧 =
𝑬(𝑋) and 𝑮 can econs uc he o iginal slice om he la en ep esen a ion 𝑧. The skip connec ions
we e added o eco e ine-g ained de ails. To en o ce simila i y be ween he gene a ed slices and he
ac ual slices, a supe ised econs uc ion loss is used:
𝐿𝑟𝑒𝑐 = λ𝑟‖𝑋′− 𝑋‖1+ λ𝑝𝐿𝑝(𝑋′− 𝑋) (1)
whe e 𝑋 is he o iginal slice, and 𝑋′= 𝑮(𝑬(𝑋)) is he es o ed image. ‖⋅‖1 is he 𝐿1 loss and 𝐿𝑝 is
he pe cep ual loss om p e- ained VGG19, which in ol es compa ing high-le el ea u es (no jus
pixel alues) om bo h he gene a ed and e e ence images (Johnson e al., 2016). Ins ead o
measu ing aw pixel di e ences, i e alua es how simila he images a e in e ms o hei con en and
s yle, based on ea u es ex ac ed om di e en laye s o he VGG19 ne wo k. λ𝑟 and λ𝑝 a e weigh
ac o s, o which he espec i e alues 10 and 0.01 we e chosen empi ically.
Fo he second s age o he aining, he ad e sa ial loss and cycle-consis en loss (Zhu e al., 2017)
we e added on op o he econs uc ion loss o ensu e ha he inpain ed images we e bo h ealis ic
and consis en wi h he o iginal images. The ad e sa ial loss helps o ensu e ha he comple ed
egions look ealis ic and blend seamlessly wi h he su ounding a eas and he cycle-consis en loss
ocuses on p ese ing he o iginal s uc u e by ensu ing he inpain ed image can be accu a ely
econs uc ed back o he o iginal.
𝑚𝑖𝑛𝐷𝑚𝑎𝑥𝐺 𝐿𝑎𝑑𝑣 =‖𝑫(𝑋)− 1‖2+‖𝑫(𝑋′)‖2 (2)
𝐿𝑐𝑦𝑐 =‖𝑋′′ − 𝑋‖1 (3)
whe e 𝑋′′ = 𝑮(𝑬(𝑋′)) and ‖⋅‖2 is he 𝐿2 loss and 𝑫 is he disc imina o . The anomaly maps a e
hen de ined as he absolu e di e ences be ween he econs uc ed slice and he o iginal slice,
𝑀 = |𝑋 − 𝑋′| (4)
Le 𝐼 be an image wi h in ensi y alues. The no maliza ion scheme in ol ed he ollowing s eps:
𝑙 = 𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑖𝑙𝑒0.5(𝐼)
ℎ = 𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑖𝑙𝑒99.5(𝐼)
𝐼𝑛𝑜𝑟𝑚 =𝑚𝑎𝑥(𝐼, 𝑙)− 𝑙
ℎ − 𝑙
Fi s Compu e he 0.5 h pe cen ile (𝑙) and he 99.5 h pe cen ile (ℎ) o he in ensi y alues in he image
and no malize he in ensi y alues in he image using he compu ed pe cen iles. The ec um was
masked wi h a alue o 0, while he meso ec um was masked wi h a alue o 0.5. The model inpu
pa ch size was (384, 384), wi h a ba ch size o 1. We used he AdamW op imize o ain he ne wo k
(bo h s ages) wi h β 0.9 and 0.95, an ini ial lea ning a e o 0.0001, a weigh decay ac o o 0.05, and
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
ollowing a polynomial decay. The model was implemen ed in PyTo ch (To ch e sion 2.1.2) and he
aining was conduc ed on an NVIDIA RTX A6000 GPU.
3.2 S udy Design
To ain he inpain ing model, 100 p os a e T2WI wi h manually segmen ed ec um and meso ec um
masks we e spli in o 80 o aining and 20 o in e nal alida ion. The model was addi ionally es ed
on 200 p os a e T2WIs and he en i e ec al da ase , comp ising 705 T2WI. Howe e , because
anno a ing ec al s uc u es is labo -in ensi e, only 180 ec al andomly selec ed samples ha e
adiologis -anno a ed masks o he ec um and meso ec um. To add ess his, a nnUNe , de ined as
ana omy nnUNe , was ained speci ically o segmen he ec um and meso ec um using a da ase o
39 samples ( aining coho 1) om a single cen e , which had he highes numbe o manually
anno a ed ec um and meso ec um masks among nine cen e s. Some s udies ha e demons a ed ha
CNNs can accu a ely delinea e ana omical s uc u es such as ec um and meso ec um o pe i ec al a
wi h DSC abo e 0.90 (DeSil io e al., 2023; Hamabe e al., 2022; Kim e al., 2019). The model was
hen e alua ed on 141 ex e nal samples, see Figu e 3. This nnUNe was hen used o in e all he
ec um and meso ec um masks ac oss he whole ec al da ase . The p edic ed ec um and meso ec um
masks we e de ined as AI-gene a ed pseudo ec um and meso ec um masks.
We inco po a ed anomaly maps gene a ed by he inpain ing model in o downs eam ec al umo
segmen a ion asks by adding hem as an addi ional inpu o nnUNe , called Anomaly-Awa e nnUNe
(AAnnUNe ). We compa ed his app oach wi h o he s a egies o in eg a ing ana omical knowledge
in o umo segmen a ion, including Mul i-Ta ge nnUNe (MTnnUNe ) and Mul i-Channel nnUNe
(MCnnUNe ). MTnnUNe added ec um and meso ec um segmen a ion as auxilia y asks, and Mul i-
Channel nnUNe (MCnnUNe ) u ilized ec um and meso ec um masks as addi ional inpu channels.
Fi s , we es ablished a baseline o ec al umo segmen a ion by compa ing he pe o mance o nine
3D deep lea ning models, see Figu e 1, including UNe (Çiçek e al., 2016), ResUNe (Diakogiannis
e al., 2020), UNe R (Ha amizadeh e al., 2022b), SwinUNe R (Ha amizadeh e al., 2022a),
A en ionUNe (A en-UNe ) (Ok ay e al., 2018), MedFo me (Gao e al., 2023), nnFo me (Zhou e
al., 2023), U-Mamba (bo ) (Ma e al., 2024) and nnUNe (Isensee e al., 2021). All he models
unde wen aining using aining coho 1 wi h a 5- old c oss- alida ion. Subsequen ly, models we e
ex e nally es ed on he emaining 666 samples om nine cen e s. MTnnUNe , MCnnUNe , and
AAnnUNe we e ained on aining coho 1 using 5- old c oss- alida ion and es ed on 666 samples
om 9 cen e s. Fo he in e ence o MCnnUNe and AAnnUNe , AI-gene a ed pseudo ec um and
meso ec um we e used.
Wi h AI-gene a ed pseudo ana omical s uc u es, MTnnUNe , MCnnUNe , and AAnnUNe we e also
ained on a la ge coho comp ising 141 samples ( aining coho 2) using 5- old c oss- alida ion
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
and es ed on 564 samples om eigh cen e s. Ins ead o elying on g ound u h ec um and
meso ec um masks, AI-gene a ed anno a ions we e employed in aining. This me hod me ged AI-
labeled ana omical s uc u es wi h manually labeled umo s, indica ing a semi-supe ised lea ning
app oach.
3.3 Segmen a ion Models
1. UNe (Çiçek e al., 2016), ex ends he p e ious u-ne a chi ec u e om Ronnebe ge e al., 2015
(Ronnebe ge e al., 2015) by eplacing all 2D ope a ions wi h hei 3D coun e pa s.
2. ResUNe (Diakogiannis e al., 2020), is a modi ied e sion o UNe . I eplaces double
con olu ion laye s o UNe wi h esidual blocks om ResNe (He e al., 2016), inco po a ing sho cu
connec ions o as e con e gence. This adap a ion wo ks in bo h 2D and 3D se ings, enhancing
pe o mance in cap u ing complex pa e ns.
3. UNETR (Ha amizadeh e al., 2022b), adop s a ViT-inspi ed encode and employs a CNN decode
o 3D image segmen a ion. The images a e ini ially di ided in o pa ches, which a e linea ly
ans o med in o oken embeddings. These okens unde go p ocessing h ough a sel -a en ion block,
akin o ViT. To manage he quad a ic complexi y o sel -a en ion, he pa ch size is se o be ela i ely
la ge (16) o p e en o e ly long sequence leng hs.
4. SwinUNETR (Ha amizadeh e al., 2022a), e o mula es he segmen a ion ask as a sequence- o-
sequence p edic ion using a Swin T ans o me as he encode . The encode is hen connec ed o a
Fully Con olu ional Neu al Ne wo k (FCNN)-based decode h ough skip connec ions.
5. A en ion UNe (Ok ay e al., 2018), in oduces an a en ion-ga ing module o UNe o enhance i s
abili y o supp ess i ele an egions and highligh salien ea u es c ucial o a gi en ask.
6. nnFo me (Zhou e al., 2023), is a 3D ans o me o olume ic medical image segmen a ion
nnFo me combines in e lea ed con olu ion and sel -a en ion ope a ions. I in oduces a special sel -
a en ion mechanism o unde s and bo h local and global aspec s o he image olume. To imp o e
e iciency, i also uses skip a en ion ins ead o he usual conca ena ion o summa ion ope a ions.
7. MedFo me (Gao e al., 2023), is a ans o me -based designed o handle scalable 3D medical
image segmen a ion, including h ee c ucial componen s: a bene icial induc i e bias, hie a chical
modeling using linea -complexi y a en ion, and mul i-scale ea u e usion ha combines spa ial and
seman ic in o ma ion globally. MedFo me can lea n om bo h small and la ge scale da a wi hou
p e- aining.
8. U-Mamba (Ma e al., 2024), is inspi ed by he S a e Space Sequence Models (SSMs) (Gu e al.,
2021), which a e known o hei abili y o handle long sequences. The model is designed speci ically
o biomedical image segmen a ion wi h he hyb id CNN-SSM block ha in eg a es he local ea u e
ex ac ion powe o con olu ional laye s wi h he abili ies o SSMs o cap u e he long- ange
dependency.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
9. nnUNe (Isensee e al., 2021), is a sel -con igu ing amewo k o medical image segmen a ion. I
u ilizes UNe as i s a chi ec u e bu o e s a specialized p ep ocessing, aining echnique, and hype -
pa ame e con igu a ion. nnUNe achie es s a e-o - he-a pe o mance on se e al medical image
segmen a ion challenges wi h a ela i ely simple a chi ec u al design.
10. Ana omy nnUNe , is he nnUNe ained o segmen ec al- ela ed ana omical s uc u es
including he ec um and meso ec um.
11. MTnnUNe , is he nnUNe ained o segmen ec um, meso ec um, and ec al umo s.
12. MCnnUNe , is he nnUNe ained o segmen ec al umo s wi h ec um and meso ec um masks
as addi ional inpu channels.
13. AAnnUNe , is he nnUNe ained o segmen ec al umo s wi h anomaly maps 𝑀 de i ed om
ana omical inpain ing.
Fo ec al umo segmen a ion, he imaging p ep ocessing app oach was adop ed om nnUNe , which
included ZSco eNo maliza ion o s anda dizing in ensi ies, uni o m esampling o all images, and a
c opping p ocess. All he segmen a ion models we e implemen ed in PyTo ch (To ch e sion 2.1.2)
and ained on an NVIDIA A6000 GPU wi h andomly ini ialized weigh s wi hou ans e lea ning.
The ba ch size was se o 2 and he models we e ained o 1000 epochs wi h he SGD op imize . The
loss unc ion is he sum o he c oss-en opy and Dice loss. Du ing in e ence, p edic ions we e
ob ained by a e aging he ou pu s o each model esul ing om he 5- old c oss- alida ion p ocedu e.
3.4 E alua ion Me ics and S a is ical Analysis
S a is ical analysis was conduc ed in Py hon ( e sion 3.9) wi h he SciPy package ( e sion 1.13.1). To
measu e he pe o mance o image econs uc ion, S uc u al Simila i y Index Measu emen (SSIM),
Peak Signal- o-NoiseRa io (PSNR) we e used. To measu e he segmen a ion pe o mance, he Dice
Simila i y Coe icien sco e (DSC) and 95% Hausdo Dis ance (HD) we e u ilized on bo h c oss-
alida ion and ex e nal es s. The cha ac e is ic di e ences o coho s we e compa ed by he K uskal-
Wallis es . The Mann–Whi ney U- es was used o compa e he di e ence o indica o s among
di e en me hods. The model pe o mance di e ences we e calcula ed using he pai ed sample - es .
All s a is ical analyses we e wo-sided and p- alues below 0.05 we e ega ded as s a is ically
signi ican . 95% con idence in e als we e gene a ed using he boo s ap me hod wi h 10,000
eplica ions.
4. Resul s
4.1 Da ase and Pa ien Cha ac e is ics
As a pa o a p e ious ins i u ional e iew boa d app o ed mul icen e s udy p ojec (Bog e adze e
al., 2022; Cai e al., 2024; El Khababi e al., 2023; “ESGAR 2020 Book o Abs ac s,” 2020; Schu ink
e al., 2023, 2022) he clinical and imaging da a om 1426 pa ien s wi h biopsy-p o en ec al cance
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Acknowledgmen
The s udy was suppo ed by he Resea ch High Pe o mance Compu ing (RHPC) acili y o he
Ne he lands Cance Ins i u e.
Funding sou ces
This s udy has ecei ed unding om he Eu opean Union’s Ho izon 2020 Resea ch and Inno a ion
P og amme unde he Ma ie Skłodowska-Cu ie g an ag eemen No 857894.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Re e ence
Bau , C., Denne , S., Wies le , B., Na ab, N., Alba qouni, S., 2021. Au oencode s o unsupe ised
anomaly segmen a ion in b ain MR images: A compa a i e s udy. Med Image Anal 69,
101952. h ps://doi.o g/10.1016/j.media.2020.101952
Bee s-Tan, R.G., Lamb eg s, D.M., Maas, M., Bipa , S., Ba ba o, B., Cu o-Semedo, L., Fenlon,
H.M., Gollub, M.J., Gou soyianni, S., Halligan, S., 2018. Magne ic esonance imaging o
clinical managemen o ec al cance : upda ed ecommenda ions om he 2016 Eu opean
Socie y o Gas oin es inal and Abdominal Radiology (ESGAR) consensus mee ing.
Eu opean adiology 28, 1465–1475.
Bog e adze, N., el Khababi, N., Schu ink, N.W., an G ie huysen, J.J.M., de Bie, S., Bosma, G.,
Cappendijk, V.C., Geenen, R.W.F., Neijenhuis, P., Pe e son, G., Veeken, C.J., Vliegen,
R.F.A., Maas, M., Lahaye, M.J., Bee s, G.L., Bee s-Tan, R.G.H., Lamb eg s, D.M.J., 2022.
E olu ions in ec al cance MRI s aging and isk s a i ica ion in The Ne he lands. Abdom
Radiol 47, 38–47. h ps://doi.o g/10.1007/s00261-021-03281-8
Bog e adze, N., Snaebjo nsson, P., G o enhuis, B.A., an T ies , B., Lahaye, M.J., Maas, M., Bee s,
G.L., Bee s-Tan, R.G.H., Lamb eg s, D.M.J., 2023. MRI ana omy o he ec um: key concep s
impo an o ec al cance s aging and ea men planning. Insigh s in o Imaging 14, 13.
h ps://doi.o g/10.1186/s13244-022-01348-8
Cai, L., Lamb eg s, D.M.J., Bee s, G.L., Mass, M., Pooch, E.H.P., Gué endel, C., Bee s-Tan, R.G.H.,
Benson, S., 2024. An au oma ed deep lea ning pipeline o EMVI classi ica ion and esponse
p edic ion o ec al cance using baseline MRI: a mul i-cen e s udy. npj P ecis. Onc. 8, 1–11.
h ps://doi.o g/10.1038/s41698-024-00516-x
Chen, X., You, S., Tezcan, K.C., Konukoglu, E., 2020. Unsupe ised Lesion De ec ion ia Image
Res o a ion wi h a No ma i e P io .
Çiçek, Ö., Abdulkadi , A., Lienkamp, S.S., B ox, T., Ronnebe ge , O., 2016. 3D U-Ne : Lea ning
Dense Volume ic Segmen a ion om Spa se Anno a ion, in: Ou selin, S., Joskowicz, L.,
Sabuncu, M.R., Unal, G., Wells, W. (Eds.), Medical Image Compu ing and Compu e -
Assis ed In e en ion – MICCAI 2016. Sp inge In e na ional Publishing, Cham, pp. 424–
432. h ps://doi.o g/10.1007/978-3-319-46723-8_49
De eudis, A., Mazze i, S., Panic, J., Micilo a, M., Vassallo, L., Gianne o, G., Ga i, M., Fale i, R.,
Ci illo, S., Regge, D., 2022. MRI-based adiomics o p edic esponse in locally ad anced
ec al cance : Compa ison o manual and au oma ic segmen a ion on ex e nal alida ion in a
mul icen e s udy. Eu opean Radiology Expe imen al 6, 19.
DeSil io, T., An unes, J.T., Be a, K., Chi a, P., Le, H., Liska, D., S ein, S.L., Ma de s ein, E., Hall,
W., Paspula i, R., Gollamudi, J., Pu ysko, A.S., Viswana h, S.E., 2023. Region-speci ic deep
lea ning models o accu a e segmen a ion o ec al s uc u es on pos -chemo adia ion T2w
MRI: a mul i-ins i u ional, mul i- eade s udy. F on . Med. 10.
h ps://doi.o g/10.3389/ med.2023.1149056
Diakogiannis, F.I., Waldne , F., Cacce a, P., Wu, C., 2020. ResUNe -a: A deep lea ning amewo k
o seman ic segmen a ion o emo ely sensed da a. ISPRS Jou nal o Pho og amme y and
Remo e Sensing 162, 94–114. h ps://doi.o g/10.1016/j.isp sjp s.2020.01.013
Dou, M., Chen, Z., Tang, Y., Sheng, L., Zhou, J., Wang, X., Yao, Y., 2023. Segmen a ion o ec al
umo om mul i-pa ame ic MRI images using an a en ion-based usion ne wo k. Med Biol
Eng Compu 61, 2379–2389. h ps://doi.o g/10.1007/s11517-023-02828-9
El Khababi, N., Bee s-Tan, R.G.H., Tissie , R., Lahaye, M.J., Maas, M., Cu o-Semedo, L., D esen,
R.C., Nouga e , S., Bee s, G.L., Lamb eg s, D.M.J., Bake s, F.C.H., Ba os, P., Baue , F., de
Bie, S.H., Ballan yne, S., Du a, J.B., Busko , L., Bog e adze, N., Bosma, G.P.T.,
Cappendijk, V.C., Cas agnoli, F., Cha alampos, S., Delli Pizzi, A., Digby, M., Geenen,
R.W.F., an G ie huysen, J.J.M., La ance, J., Mahajan, V., Malekzadeh, S., Neijenhuis, P.A.,
Pe e son, G.M., Pie e s, I., Schu ink, N.W., Smi , R., Veeken, C.J., Vliegen, R.F.A., W ay,
A., Zeina, A.-R., on behal o he ec al MRI s udy g oup, 2023. P edic ing esponse o
chemo adio he apy in ec al cance ia isual mo phologic assessmen and s aging on
baseline MRI: a mul icen e and mul i eade s udy. Abdom Radiol 48, 3039–3049.
h ps://doi.o g/10.1007/s00261-023-03961-7
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
ESGAR 2020 Book o Abs ac s, 2020. . Insigh s in o Imaging 11, 64.
h ps://doi.o g/10.1186/s13244-020-00873-8
FACG, M.A.M., MD, FACR, 2006. Dynamic Radiology o he Abdomen: No mal and Pa hologic
Ana omy. Sp inge Science & Business Media.
Gao, Y., Zhou, M., Liu, D., Yan, Z., Zhang, S., Me axas, D.N., 2023. A Da a-scalable T ans o me
o Medical Image Segmen a ion: A chi ec u e, Model E iciency, and Benchma k.
h ps://doi.o g/10.48550/a Xi .2203.00131
Gong, D., Liu, L., Le, V., Saha, B., Mansou , M.R., Venka esh, S., Van Den Hengel, A., 2019.
Memo izing No mali y o De ec Anomaly: Memo y-Augmen ed Deep Au oencode o
Unsupe ised Anomaly De ec ion, in: 2019 IEEE/CVF In e na ional Con e ence on
Compu e Vision (ICCV). P esen ed a he 2019 IEEE/CVF In e na ional Con e ence on
Compu e Vision (ICCV), pp. 1705–1714. h ps://doi.o g/10.1109/ICCV.2019.00179
Good ellow, I., Pouge -Abadie, J., Mi za, M., Xu, B., Wa de-Fa ley, D., Ozai , S., Cou ille, A.,
Bengio, Y., 2014. Gene a i e Ad e sa ial Ne s, in: Ad ances in Neu al In o ma ion
P ocessing Sys ems. Cu an Associa es, Inc.
Gu, A., Johnson, I., Goel, K., Saab, K., Dao, T., Rud a, A., Ré, C., 2021. Combining Recu en ,
Con olu ional, and Con inuous- ime Models wi h Linea S a e Space Laye s, in: Ad ances in
Neu al In o ma ion P ocessing Sys ems. Cu an Associa es, Inc., pp. 572–585.
Hamabe, A., Ishii, M., Kamoda, R., Sasuga, S., Okuya, K., Oki a, K., Akizuki, E., Sa o, Y., Miu a, R.,
Onode a, K., Ha akenaka, M., Takemasa, I., 2022. A i icial in elligence–based echnology
o semi-au oma ed segmen a ion o ec al cance using high- esolu ion MRI. PLoS One 17,
e0269931. h ps://doi.o g/10.1371/jou nal.pone.0269931
Han, L., Tan, T., Zhang, T., Huang, Y., Wang, X., Gao, Y., Teuwen, J., Mann, R., 2024. Syn hesis-
based imaging-di e en ia ion ep esen a ion lea ning o mul i-sequence 3D/4D MRI. Med
Image Anal 92, 103044. h ps://doi.o g/10.1016/j.media.2023.103044
Han, L., Zhang, T., Huang, Y., Dou, H., Wang, X., Gao, Y., Lu, C., Tan, T., Mann, R., 2023. An
Explainable Deep F amewo k: Towa ds Task-Speci ic Fusion o Mul i- o-One MRI
Syn hesis, in: G eenspan, H., Madabhushi, A., Mousa i, P., Salcudean, S., Duncan, J., Syeda-
Mahmood, T., Taylo , R. (Eds.), Medical Image Compu ing and Compu e Assis ed
In e en ion – MICCAI 2023. Sp inge Na u e Swi ze land, Cham, pp. 45–55.
h ps://doi.o g/10.1007/978-3-031-43999-5_5
Ha amizadeh, A., Na h, V., Tang, Y., Yang, D., Ro h, H.R., Xu, D., 2022a. Swin UNETR: Swin
T ans o me s o Seman ic Segmen a ion o B ain Tumo s in MRI Images, in: C imi, A.,
Bakas, S. (Eds.), B ainlesion: Glioma, Mul iple Scle osis, S oke and T auma ic B ain
Inju ies. Sp inge In e na ional Publishing, Cham, pp. 272–284. h ps://doi.o g/10.1007/978-
3-031-08999-2_22
Ha amizadeh, A., Tang, Y., Na h, V., Yang, D., My onenko, A., Landman, B., Ro h, H.R., Xu, D.,
2022b. UNETR: T ans o me s o 3D Medical Image Segmen a ion. P esen ed a he
P oceedings o he IEEE/CVF Win e Con e ence on Applica ions o Compu e Vision, pp.
574–584.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Iden i y Mappings in Deep Residual Ne wo ks, in: Leibe,
B., Ma as, J., Sebe, N., Welling, M. (Eds.), Compu e Vision – ECCV 2016. Sp inge
In e na ional Publishing, Cham, pp. 630–645. h ps://doi.o g/10.1007/978-3-319-46493-0_38
Hea n, N., Bugg, W., Chan, A., Vigna ajah, D., Cahill, K., A well, D., Lagopoulos, J., Min, M., 2020.
Manual and semi-au oma ed delinea ion o locally ad anced ec al cance sub olumes wi h
di usion-weigh ed MRI. B J Radiol 93, 20200543. h ps://doi.o g/10.1259/bj .20200543
Ho a , N., Ca los Ta a es Rocha, C., Clemen e Oli ei a, B., Pe ko ska, I., Gollub, M.J., 2019. MRI
o ec al cance : umo s aging, imaging echniques, and managemen . Radiog aphics 39,
367–387.
I ing, B., F anklin, J.M., Papież, B.W., Ande son, E.M., Sha ma, R.A., Gleeson, F.V., B ady, S.M.,
Schnabel, J.A., 2016. Pieces-o -pa s o supe oxel segmen a ion wi h global con ex :
Applica ion o DCE-MRI umou delinea ion. Medical Image Analysis 32, 69–83.
h ps://doi.o g/10.1016/j.media.2016.03.002
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Isensee, F., Jaege , P.F., Kohl, S.A., Pe e sen, J., Maie -Hein, K.H., 2021. nnU-Ne : a sel -con igu ing
me hod o deep lea ning-based biomedical image segmen a ion. Na u e me hods 18, 203–
211.
Isensee, F., Wald, T., Ul ich, C., Baumga ne , M., Roy, S., Maie -Hein, K., Jaege , P.F., 2024. nnU-
Ne Re isi ed: A Call o Rigo ous Valida ion in 3D Medical Image Segmen a ion.
Jayap akasam, V.S., Pa ode , V., Gibbs, P., Bajwa, R., Gangai, N., Sosa, R.E., Pe ko ska, I., Golia
Pe nicka, J.S., Fuqua, J.L., Ba es, D.D., 2022. MRI adiomics ea u es o meso ec al a can
p edic esponse o neoadju an chemo adia ion he apy and umo ecu ence in pa ien s wi h
locally ad anced ec al cance . Eu opean adiology 1–10.
Jian, J., Xiong, F., Xia, W., Zhang, R., Gu, J., Wu, X., Meng, X., Gao, X., 2018. Fully con olu ional
ne wo ks (FCNs)-based segmen a ion me hod o colo ec al umo s on T2-weigh ed magne ic
esonance images. Aus alas Phys Eng Sci Med 41, 393–401. h ps://doi.o g/10.1007/s13246-
018-0636-9
Johnson, J., Alahi, A., Fei-Fei, L., 2016. Pe cep ual Losses o Real-Time S yle T ans e and Supe -
Resolu ion, in: Leibe, B., Ma as, J., Sebe, N., Welling, M. (Eds.), Compu e Vision – ECCV
2016. Sp inge In e na ional Publishing, Cham, pp. 694–711. h ps://doi.o g/10.1007/978-3-
319-46475-6_43
Kim, J., Oh, J.E., Lee, J., Kim, M.J., Hu , B.Y., Sohn, D.K., Lee, B., 2019. Rec al cance : Towa d
ully au oma ic disc imina ion o T2 and T3 ec al cance s using deep con olu ional neu al
ne wo k. In e na ional Jou nal o Imaging Sys ems and Technology 29, 247–259.
h ps://doi.o g/10.1002/ima.22311
Kingma, D.P., Welling, M., 2022. Au o-Encoding Va ia ional Bayes.
Knu h, F., Adde, I.A., Huynh, B.N., G oendahl, A.R., Win e , R.M., Negå d, A., Holmedal, S.H.,
Mel ze , S., Ree, A.H., Fla ma k, K., Dueland, S., Hole, K.H., Seie s ad, T., Redalen, K.R.,
Fu sae he , C.M., 2022. MRI-based au oma ic segmen a ion o ec al cance using 2D U-Ne
on wo independen coho s. Ac a Oncologica 61, 255–263.
h ps://doi.o g/10.1080/0284186X.2021.2013530
Lamb eg s, D.M.J., Maas, M., Bee s-Tan, R.G.H., 2010. MRI o he Rec um, in: S oke , J. (Ed.), MRI
o he Gas oin es inal T ac . Sp inge , Be lin, Heidelbe g, pp. 205–227.
h ps://doi.o g/10.1007/978-3-540-85532-3_13
Lee, J., Oh, J.E., Kim, M.J., Hu , B.Y., Sohn, D.K., 2019. Reducing he Model Va iance o a Rec al
Cance Segmen a ion Ne wo k. IEEE Access 7, 182725–182733.
h ps://doi.o g/10.1109/ACCESS.2019.2960371
Li, D., Wang, J., Yang, J., Zhao, J., Yang, X., Cui, Y., Zhang, K., 2023. RTAU-Ne : A no el 3D
ec al umo segmen a ion model based on dual pa h usion and a en ional guidance.
Compu e Me hods and P og ams in Biomedicine 242, 107842.
h ps://doi.o g/10.1016/j.cmpb.2023.107842
Ma, J., Li, F., Wang, B., 2024. U-Mamba: Enhancing Long- ange Dependency o Biomedical Image
Segmen a ion. h ps://doi.o g/10.48550/a Xi .2401.04722
Menze, B.H., Jakab, A., Baue , S., Kalpa hy-C ame , J., Fa ahani, K., Ki by, J., Bu en, Y., Po z, N.,
Slo boom, J., Wies , R., Lanczi, L., Ge s ne , E., Webe , M.-A., A bel, T., A an s, B.B.,
Ayache, N., Buendia, P., Collins, D.L., Co die , N., Co so, J.J., C iminisi, A., Das, T.,
Delinge e, H., Demi alp, Ç., Du s , C.R., Doja , M., Doyle, S., Fes a, J., Fo bes, F., Ge emia,
E., Glocke , B., Golland, P., Guo, X., Hamamci, A., I ekha uddin, K.M., Jena, R., John,
N.M., Konukoglu, E., Lashka i, D., Ma iz, J.A., Meie , R., Pe ei a, S., P ecup, D., P ice, S.J.,
Ra i , T.R., Reza, S.M.S., Ryan, M., Sa ikaya, D., Schwa z, L., Shin, H.-C., Sho on, J.,
Sil a, C.A., Sousa, N., Subbanna, N.K., Szekely, G., Taylo , T.J., Thomas, O.M., Tus ison,
N.J., Unal, G., Vasseu , F., Win e ma k, M., Ye, D.H., Zhao, L., Zhao, B., Zikic, D.,
P as awa, M., Reyes, M., Van Leempu , K., 2015. The Mul imodal B ain Tumo Image
Segmen a ion Benchma k (BRATS). IEEE T ansac ions on Medical Imaging 34, 1993–2024.
h ps://doi.o g/10.1109/TMI.2014.2377694
MERCURY S udy G oup, 2007. Ex amu al dep h o umo in asion a hin-sec ion MR in pa ien s
wi h ec al cance : esul s o he MERCURY s udy. Radiology 243, 132–139.
h ps://doi.o g/10.1148/ adiol.2431051825
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Mlyna ski, P., Delinge e, H., C iminisi, A., Ayache, N., 2019. 3D con olu ional neu al ne wo ks o
umo segmen a ion using long- ange 2D con ex . Compu e ized Medical Imaging and
G aphics 73, 60–72. h ps://doi.o g/10.1016/j.compmedimag.2019.02.001
Nag egaal, I., Gaspa , C., Ma ijnen, C., an de Velde, C., Fodde, R., an K ieken, H., 2004.
Mo phological changes in umou ype a e adio he apy a e accompanied by changes in
gene exp ession p o ile bu no in clinical beha iou . The Jou nal o Pa hology 204, 183–192.
h ps://doi.o g/10.1002/pa h.1621
Nguyen, B., Feldman, A., Be hapudi, S., Jennings, A., Willcocks, C.G., 2021. Unsupe ised Region-
Based Anomaly De ec ion In B ain MRI Wi h Ad e sa ial Image Inpain ing, in: 2021 IEEE
18 h In e na ional Symposium on Biomedical Imaging (ISBI). P esen ed a he 2021 IEEE
18 h In e na ional Symposium on Biomedical Imaging (ISBI), pp. 1127–1131.
h ps://doi.o g/10.1109/ISBI48211.2021.9434115
Ok ay, O., Schlempe , J., Folgoc, L.L., Lee, M., Hein ich, M., Misawa, K., Mo i, K., McDonagh, S.,
Hamme la, N.Y., Kainz, B., Glocke , B., Ruecke , D., 2018. A en ion U-Ne : Lea ning
Whe e o Look o he Panc eas. h ps://doi.o g/10.48550/a Xi .1804.03999
Pinaya, W.H.L., Tudosiu, P.-D., G ay, R., Rees, G., Nache , P., Ou selin, S., Ca doso, M.J., 2022.
Unsupe ised b ain imaging 3D anomaly de ec ion and segmen a ion wi h ans o me s. Med
Image Anal 79, 102475. h ps://doi.o g/10.1016/j.media.2022.102475
Razzak, M.I., Naz, S., Zaib, A., 2018. Deep Lea ning o Medical Image P ocessing: O e iew,
Challenges and he Fu u e, in: Dey, N., Ashou , A.S., Bo a, S. (Eds.), Classi ica ion in
BioApps: Au oma ion o Decision Making. Sp inge In e na ional Publishing, Cham, pp.
323–350. h ps://doi.o g/10.1007/978-3-319-65981-7_12
Ronnebe ge , O., Fische , P., B ox, T., 2015. U-Ne : Con olu ional Ne wo ks o Biomedical Image
Segmen a ion, in: Na ab, N., Ho negge , J., Wells, W.M., F angi, A.F. (Eds.), Medical Image
Compu ing and Compu e -Assis ed In e en ion – MICCAI 2015, Lec u e No es in Compu e
Science. Sp inge In e na ional Publishing, Cham, pp. 234–241. h ps://doi.o g/10.1007/978-
3-319-24574-4_28
Saha, A., Bosma, J., Twil , J., Ginneken, B. an, Yaka , D., Elscho , M., Vel man, J., Fü e e , J.,
Rooij, M. de, Huisman, H., 2023. A i icial In elligence and Radiologis s a P os a e Cance
De ec ion in MRI — The PI-CAI Challenge. P esen ed a he Medical Imaging wi h Deep
Lea ning, sho pape ack.
Schu ink, N.W., an K anen, S.R., Robe i, S., an G ie huysen, J.J.M., Bog e adze, N., Cas agnoli,
F., el Khababi, N., Bake s, F.C.H., de Bie, S.H., Bosma, G.P.T., Cappendijk, V.C., Geenen,
R.W.F., Neijenhuis, P.A., Pe e son, G.M., Veeken, C.J., Vliegen, R.F.A., Bee s-Tan, R.G.H.,
Lamb eg s, D.M.J., 2022. Sou ces o a ia ion in mul icen e ec al MRI da a and hei e ec
on adiomics ea u e ep oducibili y. Eu Radiol 32, 1506–1516.
h ps://doi.o g/10.1007/s00330-021-08251-8
Schu ink, N.W., an K anen, S.R., an G ie huysen, J.J., Robe i, S., Snaebjo nsson, P., Bake s, F.C.,
de Bie, S.H., Bosma, G.P., Cappendijk, V.C., Geenen, R.W., 2023. De elopmen and
mul icen e alida ion o a mul ipa ame ic imaging model o p edic ea men esponse in
ec al cance . Eu opean Radiology 1–10.
Suzuki, C., To kzad, M.R., Tanaka, S., Palme , G., Lindholm, J., Holm, T., Blomq is , L., 2008. The
impo ance o ec al cance MRI p o ocols on iIn e p e a ion accu acy. Wo ld J Su g Onc 6,
89. h ps://doi.o g/10.1186/1477-7819-6-89
Teney, D., Abbasnejad, E., Ka le, K., Sh es ha, R., Kanan, C., an den Hengel, A., 2020. On he
Value o Ou -o -Dis ibu ion Tes ing: An Example o Goodha ’ s Law, in: Ad ances in
Neu al In o ma ion P ocessing Sys ems. Cu an Associa es, Inc., pp. 407–417.
T ebeschi, S., an G ie huysen, J.J.M., Lamb eg s, D.M.J., Lahaye, M.J., Pa ma , C., Bake s, F.C.H.,
Pe e s, N.H.G.M., Bee s-Tan, R.G.H., Ae s, H.J.W.L., 2017. Deep Lea ning o Fully-
Au oma ed Localiza ion and Segmen a ion o Rec al Cance on Mul ipa ame ic MR. Sci Rep
7, 5301. h ps://doi.o g/10.1038/s41598-017-05728-9
an den Oo d, A., Vinyals, O., ka ukcuoglu, ko ay, 2017. Neu al Disc e e Rep esen a ion Lea ning,
in: Ad ances in Neu al In o ma ion P ocessing Sys ems. Cu an Associa es, Inc.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Va oquaux, G., Cheplygina, V., 2022. Machine lea ning o medical imaging: me hodological ailu es
and ecommenda ions o he u u e. npj Digi . Med. 5, 1–8. h ps://doi.o g/10.1038/s41746-
022-00592-y
Wang, J., Lu, J., Qin, G., Shen, L., Sun, Y., Ying, H., Zhang, Z., Hu, W., 2018. Technical No e: A
deep lea ning-based au osegmen a ion o ec al umo s in MR images. Medical Physics 45,
2560–2564. h ps://doi.o g/10.1002/mp.12918
Willemink, M.J., Koszek, W.A., Ha dell, C., Wu, J., Fleischmann, D., Ha ey, H., Folio, L.R.,
Summe s, R.M., Rubin, D.L., Lung en, M.P., 2020. P epa ing Medical Imaging Da a o
Machine Lea ning. Radiology 295, 4–15. h ps://doi.o g/10.1148/ adiol.2020192224
Woo, B., Engs om, C., Ba esic, W., F ipp, J., C ozie , S., Chand a, S.S., 2024. Au oma ed anomaly-
awa e 3D segmen a ion o bones and ca ilages in knee MR images om he Os eoa h i is
Ini ia i e. Med Image Anal 93, 103089. h ps://doi.o g/10.1016/j.media.2024.103089
Xiao, H., Li, L., Liu, Q., Zhu, X., Zhang, Q., 2023. T ans o me s in medical image segmen a ion: A
e iew. Biomedical Signal P ocessing and Con ol 84, 104791.
h ps://doi.o g/10.1016/j.bspc.2023.104791
Yeganeh, Y., Fa shad, A., Na ab, N., 2022. Shape-Awa e Masking o Inpain ing in Medical
Imaging.
Zhou, H.-Y., Guo, J., Zhang, Y., Han, X., Yu, L., Wang, L., Yu, Y., 2023. nnFo me : Volume ic
Medical Image Segmen a ion ia a 3D T ans o me . IEEE T ansac ions on Image P ocessing
32, 4036–4045. h ps://doi.o g/10.1109/TIP.2023.3293771
Zhu, J.-Y., Pa k, T., Isola, P., E os, A.A., 2017. Unpai ed Image- o-Image T ansla ion Using Cycle-
Consis en Ad e sa ial Ne wo ks, in: 2017 IEEE In e na ional Con e ence on Compu e
Vision (ICCV). P esen ed a he 2017 IEEE In e na ional Con e ence on Compu e Vision
(ICCV), pp. 2242–2251. h ps://doi.o g/10.1109/ICCV.2017.244
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Table 1
Summa y o pa ien demog aphic and clinical cha ac e is ics o he mul icen e da ase
Values in age pa en heses a e he minimum and maximum. Values in pa en heses o o he i ems a e he
pe cen ages. cT, baseline clinical T s aging. cN, baseline clinical N s aging. Loca ion, umo loca ion. EMVI+:
EMVI posi i e. EMVI-, EMVI nega i e.
∗, p alues we e calcula ed using he K uskal-Wallis es be ween he aining coho , aining coho 2 and he
ex e nal da ase .
∗∗, p alues we e calcula ed using he Chi-squa e es be ween he aining coho 1, aining coho 2 and he
ex e nal da ase .
All
T aining Coho 1
T aining Coho 2
p- alue
Age (median, ange)
65 (26-88)
65 (39-82)
66 (44-87)
0.75∗, 0.17∗
Sex
Female
247 (35%)
12 (31%)
40 (28%)
0.69∗∗, 0.08∗∗
Male
458 (65%)
27 (69%)
101 (72%)
1-2
87 (12%)
3 (8%)
19 (13%)
0.58∗, 0.24∗
cT
3
519 (74%)
34 (87%)
107 (76%)
4
99 (13%)
2 (5%)
15 (11%)
0
268 (38%)
14 (36%)
67 (48%)
0.52∗, 0.001∗
cN
1
264 (37 %)
13 (33%)
54 (38%)
2
173 (25%)
12 (31%)
20 (14%)
Low
432 (61%)
25 (64%)
93 (66%)
0.45∗, 0.65∗
Loca ion
Middle
236(34%)
14 (36%)
43 (30%)
High
37 (5%)
0 (0%)
5 (4%)
EMVI
EMVI +
274 (39%)
18 (46%)
38 (27%)
0.43∗∗, 0.002∗∗
EMVI -
431 (61%)
21 (54%)
103 (73%)
To al
705
39
141
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Table 2
Rec um and Meso ec um inpain ing pe o mance in he es da a
aSSIM ± STD
aPSNR ± STD
P os a e T2WI (Num = 200)
86.72 ± 4.45
25.87 ± 1.51
Rec al T2WI (Num = 705)
83.38 ± 5.33
23.87 ± 2.56
Rec al T2WI Tumo Regions
39.32 ± 13.98
17.50 ± 3.80
Rec al T2WI Tumo -F ee Regions
83.62 ± 4.22
24.76 ± 2.41
aSSIM, a e aged SSIM. aPSNR, a e aged PSNR. STD: S anda d De ia ion
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Table 3
Compa ison o a ious models on ec al umo segmen a ion in he ex e nal es (Num = 666,
9 cen e s)
Ne wo k
UNe
ResUNe
UNe R
SwinUNe R
A en-UNe
MedFo me
nnFo me
U-Mamba
nnUNe
aDSC
(%)
55.5
(53.7, 57.2)
55.8
(54.0, 0.57.7)
42.2
(40.4, 44.1)
41.3
(39.4, 43.3)
55.0
(53.0, 57.0)
57.9
(56.0, 59.8)
44.8
(42.4, 47.3)
57.6
(55.4, 60.0)
62.8
(60.7, 64.8)
mDSC
(%)
63.2
(61.8, 64.7)
64.8
(63.3, 66.3)
47.8
(44.8, 50.7)
46.2
(42.3, 50.1)
65.2
(63.1, 67.2)
67.0
(65.3, 68.7)
56.7
(51.6, 61.8)
70.9
(69.2, 72.6)
73.2
(71.8, 74.7)
aHD
(mm)
26.04
(22.85, 29.23)
24.31
(21.46, 27.16)
43.8
(39.5, 48.2)
37.65
(34.23, 41.07)
24.73
(21.51, 27.95)
22.74
(19.77, 25.70)
26.90
(23.39, 30.42)
21.52
(17.60, 25.45)
17.28
(14.63,19.94)
mHD
(mm)
10.40
(9.32,11.49)
9.39
(8.39, 10.40)
22.0
(18.4, 25.7)
20.66
(18.04, 23.27)
8.69
(7.48, 9.91)
8.15
(6.93, 9.32)
16.13
(13.41, 18.84)
7.45
(6.54, 8.36)
6.36
(5.49, 7.22)
aDSC: a e age Dice Coe icien Simila i y. mDSC: median Dice Coe icien Simila i y. aHD: a e age o 95%
Hausdo Dis ance. mHD: median o 95% Hausdo Dis ance. The 95% con iden ial in e als a e p esen ed.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Table 4
Compa ison o nnUNe , MTnnUNe , MCnnUNe , AannUNe and Ensemble on ec al umo
segmen a ion in he ex e nal es , ully supe ised se ing (Num = 666, 9 cen e s)
Ne wo k
nnUNe
MTnnUNe
MCnnUNe
AA-nnUNe
Ensemble
aDSC
(%)
62.8
(60.7, 64.8)
65.4
(63.7, 67.2)
60.5
(58.6,62.4)
65.6
(63.9, 67.3)
69.1
(67.6, 70.6)
mDSC
(%)
73.2
(71.8, 74.7)
74.4
(73.4, 75.5)
70.2
(69.0, 71.5)
72.8
(72.0, 73.7)
75.5
(74.5, 76.5)
aHD
(mm)
17.28
(14.63,19.94)
13.63
(11.87, 15.38)
21.06
(19.51,22.62)
16.69
(14.57,18.82)
14.74
(12.72, 16.75)
mHD
(mm)
6.36
(5.45, 7.22)
5.61
(5.04, 6.18)
7.99
(7.00, 9.02)
5.61
(4.96, 6.25)
5.09
(4.64, 5.55)
aDSC: a e age Dice Coe icien Simila i y. mDSC: median Dice Coe icien Simila i y. aHD: a e age o 95%
Hausdo Dis ance. mHD: median o 95% Hausdo Dis ance. The 95% con iden ial in e als a e p esen ed.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
(a)
(b)
Figu e 7: (a) The DSC and 95% HD boxplo s o nnUNe , MTnnUNe , and MCnnUNe in a ully
supe ised manne . (b) The DSC 95% HD boxplo s o nnUNe , MTnnUNe , and MCnnUNe in a
semi-supe ised manne .
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
Figu e 8: The isualiza ion o he segmen a ion pe o mance o nnUNe , MTnnUNe , MCnnUNe ,
AAnnUNe , and Ensemble using T2WI, supe ised se ing. Each ow is a di e en sample om he
ex e nal es se . The columns om le o igh a e o iginal T2WI, g ound u h, umo p edic ion
masks om nnUNe , MTnnUNe , MCnnUNe , AAnnUNe , and Ensemble.
Figu e 9: The example whe e all algo i hms ailed o loca e and segmen he ec al umo , bu he
anomaly map highligh ed he umo al egion.
All igh s ese ed. No euse allowed wi hou pe mission.
pe pe ui y.
p ep in (which was no ce i ied by pee e iew) is he au ho / unde , who has g an ed medRxi a license o display he p ep in in
The copy igh holde o his his e sion pos ed Oc obe 16, 2024. ; h ps://doi.o g/10.1101/2024.10.15.24315517doi: medRxi p ep in
