applied
sciences
A icle
A Hyb id End- o-End App oach In eg a ing
Condi ional Random Fields in o CNNs o P os a e
Cance De ec ion on MRI
Paulo Lapa 1, Mau o Cas elli 1,∗, I o Gonçal es 2, E is Sala 3,4 and Leona do Rundo 3,4
1
No a In o ma ion Managemen School (NOVA IMS), Campus de Campolide, Uni e sidade No a de Lisboa,
1070-332 Lisboa, Po ugal; [email p o ec ed]
2INESC Coimb a, DEEC, Uni e si y o Coimb a, Pólo 2, 3030-290 Coimb a, Po ugal;
[email p o ec ed]
3Depa men o Radiology, Uni e si y o Camb idge, Camb idge CB2 0QQ, UK; [email p o ec ed] (E.S.);
[email p o ec ed] (L.R.)
4Cance Resea ch UK Camb idge Cen e, Camb idge CB2 0RE, UK
*Co espondence: [email p o ec ed]; Tel.: +351-2138-2861-0208
Recei ed: 6 Oc obe 2019; Accep ed: 24 Decembe 2019; Published: 2 Janua y 2020


Fea u ed Applica ion: In eg a ion o Condi ional Random Fields in o Con olu ional Neu al
Ne wo ks as a hyb id end- o-end app oach o p os a e cance de ec ion on non-con as -enhanced
Magne ic Resonance Imaging.
Abs ac :
P os a e Cance (PCa) is he mos common oncological disease in Wes e n men. E en hough
a g owing e o has been ca ied ou by he scien i ic communi y in ecen yea s, accu a e and eliable
au oma ed PCa de ec ion me hods on mul ipa ame ic Magne ic Resonance Imaging (mpMRI) a e
s ill a compelling issue. In his wo k, a Deep Neu al Ne wo k a chi ec u e is de eloped o he ask o
classi ying clinically signi ican PCa on non-con as -enhanced MR images. In pa icula , we p opose
he use o Condi ional Random Fields as a Recu en Neu al Ne wo k (CRF-RNN) o enhance
he classi ica ion pe o mance o XmasNe , a Con olu ional Neu al Ne wo k (CNN) a chi ec u e
speci ically ailo ed o he PROSTATEx17 Challenge. The de ised app oach builds a hyb id
end- o-end ainable ne wo k, CRF-XmasNe , composed o an ini ial CNN componen pe o ming
ea u e ex ac ion and a CRF-based p obabilis ic g aphical model componen o s uc u ed p edic ion,
wi hou he need o wo sepa a e aining p ocedu es. Expe imen al esul s show he sui abili y o
his me hod in e ms o classi ica ion accu acy and aining ime, e en hough he high- a iabili y
o he obse ed esul s mus be educed be o e ans e ing he esul ing a chi ec u e o a clinical
en i onmen . In e es ingly, he use o CRFs as a sepa a e pos p ocessing me hod achie es signi ican ly
lowe pe o mance wi h espec o he p oposed hyb id end- o-end app oach. The p oposed hyb id
end- o-end CRF-RNN app oach yields excellen peak pe o mance o all he CNN a chi ec u es aken
in o accoun , bu i shows a high- a iabili y, hus equi ing u u e in es iga ion on he in eg a ion o
CRFs in o a CNN.
Keywo ds:
p os a e cance de ec ion; magne ic esonance imaging; con olu ional neu al ne wo ks;
condi ional andom ields; ecu en neu al ne wo ks
1. In oduc ion
Acco ding o he Ame ican Cance Socie y, P os a e Cance (PCa) is he mos common ype o
cance in Wes e n men [
1
]; in 2018, app oxima ely 1.3 million new cases we e diagnosed and 359,000
ela ed dea hs occu ed wo ldwide [
2
]. Despi e i s incidence and socie al impac , he cu en diagnos ic
Appl. Sci. 2020,10, 338; doi:10.3390/app10010338 www.mdpi.com/jou nal/applsci
Appl. Sci. 2020,10, 338 2 o 19
echniques—i.e., Digi al Rec al Exam, P os a e-Speci ic An igen (PSA) [
3
]—may be subjec i e and
e o -p one [
4
]. Fu he mo e, in a- umo he e ogenei y is obse ed in PCa, con ibu ing o disease
p og ession [5].
Cu en ly, high- esolu ion mul ipa ame ic Magne ic Resonance Imaging (mpMRI) o
Compu e -Aided Diagnosis (CAD) is gaining clinical and scien i ic in e es [
6
] by enabling quan i a i e
measu emen s o in a- and in e - umo al he e ogenei y based on adiomics s udies [
7
]. Addi ional
and o en complemen a y in o ma ion can be acqui ed by means o di e en MRI sequences:
ana omical in o ma ion can be ob ained using T2-weigh ed (T2w), T1-weigh ed (T1w) and P o on
Densi y (PDw) p o ocols [
4
,
8
,
9
]. Fu he in o ma ion is con eyed by unc ional imaging [
10
],
allowing o be e depic ion o mul iple aspec s o he umo s uc u e: i s mic o-en i onmen
by es ima ing he wa e molecule mo emen using Di usion-Weigh ed Imaging (DWI) and he
de i ed Appa en Di usion Coe icien (ADC) maps [
11
], as well as he ascula s uc u e o he
umo wi h Dynamic Con as -Enhanced (DCE) MRI [
12
]. Un o una ely, mul i- ocal umo s in he
p os a e occu commonly, posing addi ional challenges o accu a e p ognoses on MRI [
13
]; hus,
de ising and exploi ing ad anced Machine Lea ning me hods [
14
,
15
] o p os a e cance de ec ion
and di e en ia ion is clinically ele an [16].
The e o e, he asks o PCa classi ica ion can bene i om he combina ion o se e al modali ies,
each con eying clinically use ul in o ma ion [
8
,
17
]. Clinical consensus o PCa diagnosis ypically
conside s mpMRI by combining T2w wi h a leas wo unc ional imaging p o ocols [
18
]. In his wo k,
T2w, PDw and ADC MRI sequences we e chosen as inpu s o he models. T2w con eys ele an
in o ma ion abou he p os a e zonal ana omy [
15
] as well as umo loca ion and ex en [
19
]: PCa has
low signal in ensi y, which can be sui ably de ec ed om he heal hy hype -in ense pe iphe al zone
issue (ha bo ing app oxima ely 70% o PCa cases [
20
]), bu i is mo e di icul o di e en ia e in he
cen al and ansi ional zones due o hei no mal low signal in ensi y [4,8].
PDw, by quan i ying he amoun o wa e p o ons con ibu ing o each oxel [
9
], p o ides a
good dis inc ion be ween a and luid [
21
]. ADC yields a quan i a i e map o he wa e di usion
cha ac e is ics o he p os a ic issue: PCa ypically has packed and dense egions wi h in a- and
in e -cellula memb anes ha in luence wa e mo ion [
4
,
8
]. Las ly, DCE sequences depic he pa ien ’s
ascula sys em in de ail, since umo s exhibi a highly ascula ized mic o-en i onmen , by exploi ing
a Gadolinium-based con as medium [22].
In Deep Lea ning applica ions o medical image analysis, some challenges a e s ill p esen [
23
];
namely, (i) he lack o la ge aining da a se s, (ii) he absence o eliable g ound u h da a, and (iii) he
di icul y in aining la ge models [
24
]. None heless, some ac o s a e consis en ly p esen in success ul
models [
25
]: expe knowledge, no el da a p ep ocessing o augmen a ion echniques, and he
applica ion o ask-speci ic a chi ec u es.
Despi e he g owing in e es in de eloping no el models o he ask o PCa, li le e o has been
de o ed o he addi ion o new ypes o laye s. Recen ly, he Seman ic Lea ning Machine (SLM) [
26
–
28
]
neu oe olu ion algo i hm was success ully employed o eplace he backp opaga ion algo i hm
commonly used in he Fully-Connec ed (FC) laye s o Con olu ional Neu al Ne wo ks (CNNs) [
29
,
30
].
When compa ed wi h backp opaga ion, SLM achie ed highe classi ica ion accu acy in PCa de ec ion
as well as a aining speed-up o one o de o magni ude. A CNN a chi ec u e de eloped o he
classi ica ion ask is ypically composed o egula laye s, which pe o m con olu ions, do p oduc s,
ba ch no maliza ion, o pooling ope a ions. This wo k in eg a es a Condi ional Random Field (CRF)
model as a Recu en Neu al Ne wo k (RNN) [
31
], gene ally exploi ed in segmen a ion, o he PCa
classi ica ion p oblem. In acco d wi h he la es clinical ends aiming a dec easing con as medium
usage [
4
], we analyzed only he non-con as -enhanced mpMRI sequences o assess also he easibili y
o ou me hodology om a pa ien sa e y and heal h economics pe spec i e [32].
Appl. Sci. 2020,10, 338 3 o 19
Resea ch Ques ions.
We speci ically add ess wo ques ions:
•Can he CRF-CNN be in eg a ed in o a s a e-o - he-a CNN as an end- o-end app oach?
•
Can he smoo hing e ec o CRFs inc ease he classi ica ion pe o mance o CNNs in
PCa de ec ion?
Con ibu ions.
Ou main con ibu ions a e he ollowing:
•
A hyb id end- o-end ainable ne wo k ha combines CRF-RNN [
31
] and a s a e-o - he-a CNN,
namely XmasNe [33], wi hou equi ing a wo-phase aining p ocedu e.
•
The p oposed CRF-XmasNe a chi ec u e gene ally ou pe o ms he baseline a chi ec u e
XmasNe [33] in e ms o PCa classi ica ion on mpMRI.
•
The p oposed end- o-end in eg a ion o CRF-RNN p o ides be e gene aliza ion abili y when
compa ed o a wo-phase implemen a ion, using a CRF as a pos p ocessing s ep.
In pa icula , he p oposed app oach aimed a ou pe o ming XmasNe , a ne wo k speci ically
c ea ed o dealing wi h p os a e cance MRI da a. This ne wo k is he mos s a e-o - he-a o his
kind o applica ion. Thus, ou pe o ming i would be a aluable con ibu ion in he medical ield, as i
shows how he in eg a ion o CRFs in XmasNe could imp o e he pe o mance o he commonly used
XmasNe a chi ec u e.
The manusc ip is o ganized as ollows. Sec ion 2in oduces he heo e ical ounda ions o CRFs
and he Deep Neu al Ne wo k (DNN) a chi ec u es unde lying he de ised me hod. Sec ion 3p esen s
he cha ac e is ics o he analyzed p os a e mpMRI da ase , as well as he p oposed me hod. Sec ion 4
shows and c i ically analyzes he achie ed expe imen al esul s. Finally, Sec ion 5concludes he pape
and sugges s u u e esea ch a enues.
2. Theo e ical Backg ound
This sec ion in oduces he basic concep s necessa y o ully unde s and he a ionale and he
unc ioning o he de ised DNN o PCa de ec ion.
2.1. Con olu ional Neu al Ne wo ks
CNNs ha e become one o he mos common supe ised lea ning echniques [
34
]. They can
lea n complex pa e ns om uns uc u ed da a (i.e., ex o images) wi h limi ed domain knowledge.
By le e aging he con olu ion ope a ion, CNNs can pe o m hei ask on wo- o highe -dimensional
inpu s. They can conside he neighbo ing egion a ound a pixel, making hem well-sui ed o image
applica ions [35].
They ha e ound success in Medical Image Analysis (MIA) applica ions, namely in cance - ela ed
p oblems [25]. In he p os a e egion, deep CNNs achie ed be e pe o mance when compa ed wi h
non-deep CNNs [
36
] o PCa classi ica ion asks. Fo his ask, CNNs ha e also been used o ex ac
disc imina i e ea u es om T1w and DCE sequences [
37
], om 3D ea u es ex ac ed ei he om
MRI sequences [
38
] o Gleason Sco e (GS) p edic ion based on T ans ec al Ul asound (TRUS)-guided
biopsy esul s [
39
,
40
]. CNNs ha e also been used wi h U-Ne inspi ed a chi ec u es [
41
,
42
] o
PCa segmen a ion.
A he mos gene al le el, CNNs ha e been used in almos e e y ana omic a ea o he human
body (e.g., b ain, eyes, o so, knees) o a ious asks (disease loca ion, issue segmen a ion o su i al
p obabili y calcula ion) wi h a high deg ee o success [25].
Fo ins ance, XmasNe was de eloped by Liu e al. [
33
] speci ically o he PROSTATEx Challenge
2017 [
43
], inspi ed by he Visual Geome y G oup (VGG) ne [
44
]. Despi e i s ela i e simplici y,
i achie ed s a e-o - he-a esul s, ou pe o ming 69 me hods o 33 g oups and ha ing he second
highes A ea Unde he Recei e Ope a ing Cha ac e is ics Cu e (AUROC) on he unseen es
Appl. Sci. 2020,10, 338 4 o 19
se . XmasNe is a ela i ely adi ional a chi ec u e: ou con olu ional laye s and wo FC laye s.
The a chi ec u e also makes use o Ba ch No maliza ion, Rec i ied Linea Uni (ReLU) ac i a ion
unc ions, and Max Pooling.
O he a chi ec u es we e compa ed in his wo k, namely AlexNe [
34
], VGG16 [
44
] and
ResNe [
45
]. AlexNe was he winne o he ImageNe La ge Scale Visual Recogni ion Challenge 2012
(ILSVRC2012) [
46
], and i s success e i ed he in e es in CNNs in Compu e Vision and b ough abou
se e al no el implemen a ions: ReLU non-linea ac i a ion unc ions, mul iple GPU aining, Local
Response No maliza ion, and O e lapping Pooling, ha a e s ill used in cu en a chi ec u es. VGG16
is based on AlexNe and was he i s uly deep CNN wi h 16 con olu ional laye s, made possible by
he use o small con olu ional il e s and ReLU ac i a ion unc ions whene e possible [
34
]. I achie ed
i s and second place in ILSVRC 2014 and se es as he inspi a ion o XmasNe . Las ly, ResNe , can be
conside ed he deepes ne wo k among he a chi ec u es in es iga ed in his s udy, wi h 50 laye s.
I s complexi y is enabled by he use o esidual connec ions be ween laye s, which lea n a e e ence
esidual mapping, making he aining o a bi a ily deep CNNs heo e ically possible.
2.2. Condi ional Random Fields as Recu en Neu al Ne wo ks
CRFs achie ed s a e-o - he-a esul s in he image segmen a ion asks, bo h in he adi ional
benchma k [
47
], as well as in applica ion o medical image analysis, such as in PCa segmen a ion [
48
],
weakly supe ised segmen a ion o PCa [49], GS g ading [50] o PCa de ec ion [51].
CRF unc ioning is based on he no ion o ene gy
E(·)
, i.e., he cos o assigning a label o a gi en
pixel. A CRF is composed o wo ypes o ene gy—namely, una y and pai wise—which mus ag ee
and can be desc ibed as:
E(x) = ∑
i
Ψu(xi) + ∑
p
Ψp(xi,xj). (1)
The una y ene gy
Ψu(xi)
is he p obabili y o a pixel
i
belonging o a gi en label
xi
in his case
ex ac ed by a CNN. Con e sely,
Ψp(xi
,
xj)
co esponds o he pai wise ene gy. I measu es he cos
o assigning he labels
xi
and
xj
o pixels
i
and
j
simul aneously. I ensu es image smoo hness and
consis ency; pixels wi h simila p ope ies should ha e simila labels. Thus, CRFs p omo e egions o
homogeneous p edic ions. The pai wise ene gy is de ined acco ding o [52]:
Ψp(xi,xj) = µ(xi,xj)K( i, j), (2)
whe e
µ(xi
,
xj)
is a label compa ibili y unc ion and he Gaussian ke nel
K(·
,
·)
applied on he ea u e
ec o s iand j, wi h w(m)being linea combina ion weigh s:
K( i, j) =
k
∑
m=1
w(m)K(m)( i, j). (3)
In ou case, he ea u es conside he posi ions
pi
and
pj
as well as he in ensi y alues
Ii
and
Ij
o
he pixels in he image:
k( i, j) = w(1)exp −|pi−pj|2
2θ2
α
−|Ii−Ij|2
2θ2
β!
| {z }
appea ance ke nel
+w(2)exp −|pi−pj|2
2θ2
γ!
| {z }
smoo hness ke nel
, (4)
wi h
θα
,
θβand θγ
being hype -pa ame e s con olling he impo ance o he hype - oxel dis ance
in he ea u e space (i.e., he appea ance ke nel p omo es pixels wi h simila in ensi y alues o
be in he same class, while he smoo hness ke nel emo es small isola ed egions [
52
])—a s onge
penal y is gi en i nea by pixels ha e di e en labels. In his e sion, he Po s model is used,
i.e., µ(xi,xj) = [xi6=xj][31,52].
Appl. Sci. 2020,10, 338 5 o 19
The aining ime o a CRF g ows exponen ially wi h espec o he numbe o inpu pixels
N
,
e en when conside ing app oxima e aining me hods, like Ma ko Chain Mon e Ca lo (MCMC),
pseudo-likelihood o junc ion ees [
31
,
53
]. To sol e his sho coming, he Mean Field app oxima ion
(MFa) can be used [
52
]. MFa consis s in app oxima ing he dis ibu ion
P(X)
,—whe e
X
is he ec o
o he andom a iables
X1
,
X2
,
. . .
,
XN
deno ing he
N
pixels composing he image—by a simple
dis ibu ion
Q(X)
, which can be w i en as he p oduc o independen ma ginal dis ibu ions:
Q(X) =
∏iQi(Xi), subjec o: ∑xiQi(Xi) = 1.
The unc ion
Qi(xi)
can be de ined such ha is upda ed i e a i ely [
52
]. Bea ing his in mind,
an MFa- ained CRF s ill canno be ained by means o he backp opaga ion, making i s end- o-end
in eg a ion wi h CNN in easible, as he CNN and CRF need o be ained sepa a ely. Fu he
e ining MFa, he au ho s o [
31
] o malized he CRFs as Recu en Neu al Ne wo ks (CRF-RNN).
This wo k ede ines he MFa as a se ies o con olu ional and ecu en laye s. The con olu ional
laye s pe o m he Gaussian ope a ions on he ea u es ia lea nable il e s, while he ecu en laye s
beha e as se e al i e a ions o he MFa me hod. This join app oach, by un olling he CRF MFa
in e ence s eps, builds an end- o-end ainable eed- o wa d ne wo k composed o an ini ial CNN
componen pe o ming ea u e ex ac ion and a CRF-based p obabilis ic g aphical model componen
o s uc u ed p edic ion, wi hou he need o wo sepa a e aining p ocedu es. Since he wo
componen s can lea n in he same en i onmen , hey can coope a e o achie e he bes pe o mance [
31
].
The eby, wi h his implemen a ion, a CRF-RNN ne wo k is limi ed o a ba ch size o 1 due o GPU
memo y cons ain s [31].
FC CRFs achie ed ou s anding pe o mance in seman ic segmen a ion [
52
,
54
] and emo e
sensing [
55
] when used as a pos p ocessing s a egy. In e es ingly, he esul s in [
31
] showed an e iden
compe i i e ad an age o he join end- o-end amewo k wi h espec o he o line applica ion o
CRFs as pos p ocessing me hod (disconnec ed om he CNN aining). This migh be a ibu ed
o he ac ha du ing he backp opaga ion-enabled join aining, he CNN and CRF componen s
coope a e o yield an op imized ou pu by i e a i ely inco po a ing he CRF con ibu ion. Relying on
hese expe imen al indings, we de ised a hyb id end- o-end ainable model by in oducing he
CRF-RNN in o XmasNe , along wi h h ee skip connec ions o me ge he in o ma ion om mul iple
laye s. I is wo h no ing ha we chose XmasNe as baseline since i was speci ically designed o
he PROSTATEx17 Challenge and is no a pa icula ly deep a chi ec u e; he e o e, i can se e as a
sui able case s udy o e alua ing he e ec i e bene i s achie ed by he in eg a ion o he CRF-RNN
module in o CNNs.
3. Ma e ials and Me hods
This sec ion p esen s he analyzed p os a e MRI da ase s, as well as he p oposed end- o-end deep
lea ning amewo k combining CRF-RNN and XmasNe .
3.1. Expe imen al Da ase : The PROSTATEx17 Da ase
This wo k conside s he MRI da ase p o ided by he PROSTATEx Challenge 2017 [
43
] as
pa o he 2017 SPIE Medical Imaging Symposium [
56
], o ganized by he Socie y o Pho og aphic
Ins umen a ion Enginee s (SPIE) and suppo ed by he Ame ican Associa ion o Physicis s in Medicine
(AAPM) and he Na ional Cance Ins i u e (NCI). The aim o he PROSTATEx Challenge 2017 is o
de elop a quan i a i e diagnos ic classi ica ion me hod o p os a e lesions. This da ase was p e iously
collec ed and cu a ed by he Radboud Uni e si y Medical Cen e (Nijmegen, he Ne he lands) in he
P os a e MR Re e ence Cen e unde he supe ision o P o . Jelle Ba en sz [18].
The da ase con ains mpMRI s udies o 344 subjec s. All s udies include T2w, PDw, DCE, and DWI
MRI sequences, acqui ed using he MAGNETOM T io and Sky a 3T MRI scanne models om Siemens
(Siemens Heal hinee s, E langen, Ge many), wi hou employing an endo ec al coil. Fo mo e de ails on
image acquisi ion, please e e o [
43
]. Al hough DCE imaging con eys ele an unc ional in o ma ion,
i ca ies he d awback o needing an ex e nal agen , ypically ia he injec ion o a Gadolinium-based

Appl. Sci. 2020,10, 338 6 o 19
con as [
32
,
57
]. The con as may cause discom o o he pa ien , wi h an inc ease in he isk o
esidual deposi ion in he human body [
58
] and wi hou p oo o an imp o emen in cance de ec ion
quali y [
32
]. In his s udy, we used non-con as -enhanced MRI sequences only, since DWI—and
especially ADC maps [
59
]—showed p omising applica ions in he clinic. In a e y ecen s udy [
60
],
simila cance de ec ion a es om bipa ame ic MRI (bpMRI)— ocusing on T2w and DWI—and
con as -enhanced mpMRI, pa icula ly o Clinically Signi ican (CS) cases o PCa, we e achie ed.
Examples o inpu T2w, PDw and ADC MR images a e shown in Figu es 1a, 1b and 1c, espec i ely.
Each MRI s udy was e alua ed unde he supe ision o an expe adiologis ha iden i ied a eas
o suspicion. I an a ea was ma ked as likely o cance , a biopsy was pe o med and hen g aded by a
pa hologis . I he biopsy esul s had a GS highe han 7, i was conside ed CS [
18
]. The ul ima e goal
o he PROSTATEx Challenge 2017 is o p edic he clinical signi icance o a pa ien ’s lesion based on
his MRI s udies.
The whole coho was di ided in o wo sub-se s; each lesion’s CS in o ma ion was a ailable in
he aining se (204 subjec s) bu no in he es se (140 subjec s). The e o e, only he aining se was
conside ed in his s udy.
3.2. The P oposed End- o-End Solu ion In eg a ing CRF-RNN wi h CNNs o PCa De ec ion
The p ocessing phases o he p oposed end- o-end me hod a e desc ibed in his sec ion.
3.2.1. Da a P ep ocessing
Conside ing ha he images we e collec ed in di e en condi ions (e.g., di e en scanne s and
acquisi ion con igu a ions), an in e media e da a p ep ocessing s ep was deemed necessa y o ensu e
eliable da a p ope ies. The a ailable images we e cha ac e ized by ha ing a di e en esolu ion in
he 3D space (i.e., aniso opic oxels). The e o e, iso opic cubic in e pola ion was used on e e y image
o achie e a esolu ion o 1.0 mm3.
An image egis a ion p ocedu e was also necessa y because se e al ac o s can con ibu e
o making he indi idual s udies no compa able (i.e, pa ien mo emen , di e en machine
con igu a ions) [
61
]. In image egis a ion, a se o ans o ma ions
τ
is applied on one image
(i.e., he mo ing image) so ha i s landma ks/ ea u es o pixel in ensi ies o e lap on o ano he
e e ence (i.e., he ixed image), maximizing he ma ching o he pai . Only a ine ans o ma ions
(i.e., o a ion, ansla ion, and shea ing) we e conside ed. Mu ual In o ma ion C i e ia [
62
] we e used
o measu e he quali y o he egis a ion. T2w was se as he ixed image, while PDw and ADC
we e used as mo ing images. The image in e pola ion and egis a ion we e pe o med by using he
Di usion Imaging Py hon (DIPy) package [63].
To ex ac he lesion in o ma ion, a 64
×
64-pixel Region o In e es (RoI) was cen e ed on he PCa
coo dina es. Z-sco e no maliza ion was hen employed o ans o m all he pixel alues o each MRI
slice o a common scale wi h ze o mean and uni a y s anda d de ia ion. The no malized RoI pa ches
ex ac ed om he T2w, ADC, and PDw MRI sequences we e conca ena ed as a 64
×
64
×
3 image o
se e as he model inpu s, as illus a ed in Figu e 1.
A he end o his p ocedu e, conside ing he 204 subjec s wi h CS anno a ions, 335 lesions we e
collec ed, co esponding o 20.5% o he lesions analyzed.
Appl. Sci. 2020,10, 338 7 o 19
(a) (b) (c)
(d) (e) ( )
Figu e 1.
Example MR images om Pa ien #005: (
a
) T2w, (
b
) PDw, and (
c
) ADC sequences. The PCa
lesions a e highligh ed (wi h ed c osses) in he coo dina es
(
95, 92
)
,
(
103, 113
)
, and
(
78, 109
)
, wi h slice
index
z=
30, a e he egis a ion agains he T2w MRI sequence. The h ee lesions a e sepa a ely
displayed on he T2w slice in (d– ). In pa icula , hese RoIs a e cen e ed and c opped on each lesion.
3.2.2. The CRF-XmasNe A chi ec u e
We s a ed om he XmasNe [
33
] a chi ec u e as baseline in ou case s udy on he PROSTATEx17
da ase [
43
], since i achie ed he second highes AUC in he PROSTATEx17 challenge despi e
i s simplici y. Indeed, XmasNe can be seen as a pa ame e -e icien e sion o he VGG ne [
44
]
(i.e., XmasNe is less deep han VGG16) ailo ed o PCa de ec ion in o de o show he po en ial o deep
lea ning in oncological imaging [
33
]. I is wo h no ing ha he XmasNe model in he o iginal pape
exploi ed he ensemble o 20 indi idual XmasNe ins ances (which maximize he alida ion AUC on
di e en combina ions o he inpu MRI sequences) by using he weigh ed a e age o he p edic ions
ia a g eedy bagging algo i hm [
33
]. In ou wo k, o keep ull con ol o he CRF-RNN [
31
] in oduc ion
and o a oid he s ochas ic e ec due o he ne wo k ensemble, we used only one XmasNe model o
ob ain he lesion’s classi ica ion. In his manne , we can e ec i ely assess he bene i s p o ided by he
end- o-end aining o he p oposed hyb id CRF-CNN app oach.
Figu e 2schema izes he p oposed CRF-XmasNe a chi ec u e. The ne wo k can be di ided in o
h ee main componen s: (i) downsampling, based on con olu ion, Ba ch No maliza ion (BN), ReLU
and Max Pooling ope a o s; (ii) upsampling, using one-dimensional con olu ion and decon olu ion
ope a o s; (iii) classi ica ion in ol ing la en and FC laye s, along wi h ReLU and sigmoid ac i a ion
ac ions. The de ails abou hese h ee componen s a e p o ided in Tables 1–3, espec i ely. Impo an ly,
a CRF-RNN [
31
] laye was in eg a ed in o he XmasNe a chi ec u e, u ilizing and me ging he ea u es
ex ac ed by he con olu ional po ions as inpu s (see Figu e 2). The me ging ope a ion is pe o med
ia skip connec ions om mul iple laye s and summing-up he ea u e maps.
Appl. Sci. 2020,10, 338 8 o 19
Addi ional modi ica ions we e p oposed, aimed a imp o ing he ne wo k e ec i eness:
• h ee skip connec ions and wo con olu ional laye s we e added, as shown in Figu e 2;
•
d opou [
64
] was in oduced be ween he FC laye s and he sigmoid (wi h a d opou a e o 0.5);
•
he numbe o pa ame e s in he i s and second FC laye s was changed, om 1024 o 128 and
1024 o 256, espec i ely, because o pe o mance cons ain s o he a ailable compu ing powe .
Skip connec ions we e added as hey allow o : (i) a simple me hod o me ging in o ma ion
coming om se e al laye s in o he CRF; (ii) p ocessing he inpu o highe -le el ea u es no p esen
in he CRF laye ou pu in o he classi ie componen o he ne wo k; (iii) imp o ing he aining
p ocess [
45
], pa icula ly du ing he ea ly epochs, since in he backp opaga ion p ocedu e some e o s
migh be di ec ly p esen ed o he downsampling componen .
CRF
Fla en
Fully Connec ed + ReLU
Vec o
Sigmoid
Downsampling
Upsampling Me ging
Classifica ion
32 32
16 16 16 16
4096
128 256
1
1 1
1
1
1
1
1
1
Fully Connec ed + ReLU + D opou
Legend
Con + BN + ReLU
Max Pooling Skip Connec ion
1x1 con olu ion
Decon olu ion
Addi ion + BN
Addi ion
Figu e 2.
The p oposed CRF-XmasNe a chi ec u e in eg a ing CRFs [
31
] in o he baseline XmasNe [
33
]
as an end- o-end app oach. This hyb id ne wo k, allowing o join aining ia backp opaga ion,
analyzes h ee non-con as -enhanced mpMRI sequences (namely, T2w, T1w and ADC) and yields a
p edic ion p obabili y o each CS PCa case.Mo e speci ically, he whole a chi ec u e can be di ided
in o h ee componen s: (i) downsampling, (ii) upsampling, and (iii) classi ica ion. In o de o e ec i ely
me ge he in o ma ion om mul iple laye s in o he CRF, h ee skip connec ions we e added. The legend
box shows he symbol no a ion and colo seman ics. The digi s abo e he laye s ou pu s ep esen
he dep h.
Table 1. Downsampling componen ne wo k pa ame e s.
Laye Con 1 Con 2 MaxPool1 Con 3 Con 4 MaxPool2
Pa ch size/s ide 3 ×3/1 3 ×3/1 2 ×2/2 3 ×3/1 3 ×3/1 2 ×2/2
Ou pu size 64 ×64 ×32 64 ×64 ×32 32 ×32 ×32 32 ×32 ×32 32 ×32 ×32 16 ×16 ×32
Table 2. Upsampling componen ne wo k pa ame e s.
Laye 1d_Con 1 1d_Con 2 1d_Con 3 Decon 1 Decon 2
Inpu Laye Con 2 Con 4 MaxPool2 Ou _1d_Con 2 Ou _1d_Con 3
Pa ch size/s ide 1 ×1/1 1 ×1/1 1 ×1/1 2 ×2×2 2 ×2/4
Ou pu size 64 ×64 ×64 32 ×32 ×1 16 ×16 ×1 64 ×64 ×1 64 ×64 ×1
Table 3. Classi ica ion componen ne wo k pa ame e s.
Laye FC1 FC2 FC3
Ou pu size 128 ×1 256 ×1 1 ×1
To e alua e he pe o mance o he p oposed CRF-XmasNe a chi ec u e, we also compa ed i s
pe o mance agains he XmasNe a chi ec u e in which a CRF is used as a pos p ocessing phase.
Appl. Sci. 2020,10, 338 9 o 19
Mo e speci ically, he adi ional XmasNe a chi ec u e is ained and, subsequen ly, CRFs a e used
o possibly imp o e he classi ica ion pe o mance o he XmasNe a chi ec u e. While he wo k o
Zheng e al. [
31
] showed ha he use o CRFs as a pos p ocessing me hod (i.e., independen om he
CNN aining) esul s in poo pe o mance when compa ed wi h he join end- o-end amewo k,
we belie e ha his analysis is impo an o s eng hen he sui abili y o he p oposed CRF-XmasNe
a chi ec u e. In he emainde o he pape , we deno e he XmasNe a chi ec u e, which uses CRFs as a
pos p ocessing s ep, as XmasNe -CRF-pos p ocessing (XmasNe -CRFpp).
Along wi h he in eg a ion o CRFs in o XmasNe , he p oposed hyb id end- o-end app oach was
applied also o VGG16 and AlexNe wi h he aim o showing i s sui abili y. In p ac ice, CRF-RNNs can
be in eg a ed in o any CNN: he downsampling componen o he CRF-XmasNe , illus a ed in Figu e 2,
can be subs i u ed by he ea u e ex ac ion sub-ne wo k o any a chi ec u e (i.e., he laye s, be o e he
FC laye s ha pe o m he classi ica ion, esponsible o ex ac ing ea u es). In pa icula , he VGG16
and AlexNe a chi ec u es we e ans o med in o hei espec i e CRF-VGG16 and CRF-AlexNe
e sions by de ining he downsampling componen as he laye s p eceding he la ening ope a ion o
hei o iginal a chi ec u es. Indeed, VGG16 and AlexNe a e o e all mo e complex han he baseline
XmasNe , which emains ou benchma k since i was speci ically designed o he PROSTATEx17
challenge [43].
3.2.3. Expe imen al Se up and Implemen a ion De ails
CNN pe o mance migh be pa icula ly suscep ible o wo p oblems: (i) hype -pa ame e se ings
and (ii) sensi i i y o ini ializa ion alues. Aiming a ensu ing eliable and epea able esul s, we es ed
se e al con igu a ions and hen ained mul iple imes.
Fi s , h ee pa i ions we e c ea ed: aining, alida ion, and es ing, wi h 60%, 20% and 20% o
he o iginal da ase , espec i ely. Fo each a chi ec u e, wen y hype -pa ame e con igu a ions we e
andomly c ea ed; e e y con igu a ion was epea edly ained 30 imes o 350 epochs o un il he
loss sco e (Bina y C oss En opy, BCE) did no imp o e o e 1
×
10
−4
in he las 15 epochs. A e he
aining, he model was e alua ed wi h he es se . The con igu a ion wi h he highes a e age AUROC
alue was conside ed he bes o each gi en a chi ec u e. Mo e speci ically, he XmasNe -CRFpp
a chi ec u e was ained h ough a wo-s ep p ocedu e: i s , he XmasNe was ained; subsequen ly,
using he weigh s o he bes model, he mpMRI ea u es we e ex ac ed o o m an in e media y
da ase . The CRF-RNN model was ained on his da ase , hus employing CRFs o pos p ocessing.
The aining was pe o med wi h a ba ch size o 6 o he s a e-o - he-a CNN a chi ec u es
and o 1 o he hyb id CRF-RNN/CNN app oach. In mo e de ail, o he CNN a chi ec u es ha
do no in eg a e CRFs, a ba ch size o 6 was used, as i is he ba ch size alue ha showed a sui able
ade-o be ween aining ime and pe o mance. On he o he hand, he hyb id CRF-XmasNe and
XmasNe -CRFpp a chi ec u es, whe e he CRF was in eg a ed wi hin he CNN (as an end- o-end and
as a pos p ocessing, espec i ely), used a ba ch size o 1, which was selec ed o a oid eaching he
memo y limi s o he GPU (as also sugges ed in [31]).
We employed a andom sample algo i hm, selec ing one o se e al possible alues o each
hype -pa ame e , as desc ibed in Table 4. Only high alues o momen um
m
we e conside ed in he
g id sea ch, based on good empi ical esul s du ing p o o yping when compa ed o lowe alues (e.g.,
m<0.9), as well as wi h he suppo o he li e a u e [65].
Appl. Sci. 2020,10, 338 16 o 19
16.
Wibme , A.; H icak, H.; Gondo, T.; Ma sumo o, K.; Vee a agha an, H.; Feh , D.; Zheng, J.; Goldman, D.;
Moskowi z, C.; Fine, S.W.; e al. Ha alick Tex u e Analysis o p os a e MRI: U ili y o di e en ia ing
non-cance ous p os a e om p os a e cance and di e en ia ing p os a e cance s wi h di e en Gleason
Sco es. Eu . Radiol. 2015,25, 2840–2850. [C ossRe ]
17.
Pa k, J.; Rho, M.J.; Pa k. Y.H.; Jung, C.K.; Chong, Y.; Kim, C.S.; Go, H.; Jeon, S.S.; Kang, M.; Lee, H.J.; e al.
PROMISE CLIP P ojec : A Re ospec i e, Mul icen e S udy o P os a e Cance ha In eg a es Clinical,
Imaging and Pa hology Da a. Appl. Sci. 2019,9, 2982. [C ossRe ]
18.
Li jens, G.; Deba s, O.; Ba en sz, J.; Ka ssemeije , N.; Huisman, H. Compu e -aided de ec ion o p os a e
cance in MRI. IEEE T ans. Med. Imaging 2014,33, 1083–1092. [C ossRe ]
19.
Villei s, G.M.; De Mee lee , G.O. Magne ic esonance imaging (MRI) ana omy o he p os a e and applica ion
o MRI in adio he apy planning. Eu . J. Radiol. 2007,63, 361–368. [C ossRe ]
20.
Quon, J.S.; Moosa i, B.; Khanna, M.; Flood, T.A.; Lim, C.S.; Schieda, N. False posi i e and alse nega i e
diagnoses o p os a e cance a mul i-pa ame ic p os a e MRI in ac i e su eillance. Insigh s Imaging
2015
,
6, 449–463. [C ossRe ]
21.
Mang um, W.; Ch is ianson, K.; Duncan, S.; Hoang, P.; Song, A.; Me kle, E. Duke Re iew o MRI P inciples,
1s ed.; Case Re iew Se ies; Else ie : Ma yland Heigh s, MI, USA, 2012.
22. To s, P.S. T1-weigh ed DCE imaging concep s: Modelling, acquisi ion and analysis. Signal 2010,500, 400.
23.
Shen, D.; Wu, G.; Suk, H.I. Deep lea ning in medical image analysis. Annu. Re . Biomed. Engin.
2017
,
19, 221–248. [C ossRe ] [PubMed]
24.
Ra ì, D.; Wong, C.; Deligianni, F.; Be helo , M.; And eu-Pe ez, J.; Lo, B.; Yang, G.-Z. Deep Lea n. Heal h
In o ma ics. IEEE J. Biomed. Heal h In o m. 2017,21, 4–21. [C ossRe ] [PubMed]
25.
Li jens, G.; Kooi, T.; Bejno di, B.E.; Se io, A.A.A.; Ciompi, F.; Gha oo ian, M.; an de Laak, J.A.;
an Ginneken, B.; Sánchez, C.I. A su ey on deep lea ning in medical image analysis. Med. Image Anal.
2017
,
42, 60–88. [C ossRe ] [PubMed]
26.
Gonçal es, I.; Sil a, S.; Fonseca, C.M. Seman ic Lea ning Machine: A Feed o wa d Neu al Ne wo k
Cons uc ion Algo i hm Inspi ed by Geome ic Seman ic Gene ic P og amming. In P og ess in A i icial
In elligence; Lec u e No es in Compu e Science; Sp inge : Be lin/Heidelbe g, Ge many, 2015; Volume 9273,
pp. 280–285. [C ossRe ]
27.
Gonçal es, I. An Explo a ion o Gene aliza ion and O e i ing in Gene ic P og amming: S anda d and
Geome ic Seman ic App oaches. Ph.D. Thesis, Depa men o In o ma ics Enginee ing, Uni e si y o
Coimb a, Coimb a, Po ugal, 2017.
28.
Jagusch, J.B.; Gonçal es, I.; Cas elli, M. Neu oe olu ion unde Unimodal E o Landscapes: An Explo a ion
o he Seman ic Lea ning Machine Algo i hm. In P oceedings o he Gene ic and E olu iona y Compu a ion
Con e ence Companion (GECCO ’18), Kyo o, Japan, 15–19 July 2018; ACM: New Yo k, NY, USA, 2018.
[C ossRe ]
29.
Lapa, P.; Gonçal es, I.; Rundo, L.; Cas elli, M. Seman ic lea ning machine imp o es he CNN-based
de ec ion o p os a e cance in non-con as -enhanced MRI. In P oceedings o he Gene ic and E olu iona y
Compu a ion Con e ence (GECCO) Companion, P ague, Czech Republic, 13–17 July 2019; ACM: Hoboken,
NJ, USA, 2019; pp. 1837–1845. [C ossRe ]
30.
Lapa, P.; Gonçal es, I.; Rundo, L.; Cas elli, M. Enhancing classi ica ion pe o mance o con olu ional neu al
ne wo ks o p os a e cance de ec ion on magne ic esonance images: A s udy wi h he seman ic lea ning
machine. In P oceedings o he Gene ic and E olu iona y Compu a ion Con e ence Companion (GECCO
’19), P ague, Czech Republic, 13–17 July 2019; ACM: New Yo k, NY, USA, 2019; pp. 381–382. [C ossRe ]
31.
Zheng, S.; Jayasumana, S.; Rome a-Pa edes, B.; Vinee , V.; Su, Z.; Du, D.; Huang, C.; To , P.H.S. Condi ional
andom ields as ecu en neu al ne wo ks. In P oceedings o he IEEE In e na ional Con e ence on
Compu e Vision (ICCV), San iago, Chile, 13–16 Decembe 2015; pp. 1529–1537. [C ossRe ]
32.
Junke , D.; S einkohl, F.; F i z, V.; Bek ic, J.; Tokas, T.; Aigne , F.; He mann, T.R.; Riege , M.; Nagele, U.
Compa ison o mul ipa ame ic and bipa ame ic MRI o he p os a e: A e gadolinium-based con as agen s
needed o ou ine examina ions? Wo ld J. U ol. 2018, 1–9. [C ossRe ] [PubMed]
33.
Liu, S.; Zheng, H.; Feng, Y.; Li, W. P os a e cance diagnosis using deep lea ning wi h 3D mul ipa ame ic
MRI. P oc. SPIE 2017,10134, 1013428. [C ossRe ]
34.
K izhe sky, A.; Su ske e , I.; Hin on, G.E. ImageNe Classi ica ion wi h Deep Con olu ional Neu al
Ne wo ks. Ad . Neu al In . P ocess. Sys . 2012. [C ossRe ]

Appl. Sci. 2020,10, 338 17 o 19
35.
Good ellow, I.; Bengio, Y.; Cou ille, A. Deep Lea ning; MIT P ess: Camb idge, MA, USA, 2016. A ailable
online: h p://www.deeplea ningbook.o g (accessed on 16 Decembe 2019).
36.
Wang, X.; Yang, W.; Wein eb, J.; Han, J.; Li, Q.; Kong, X.; Yan, Y.; Ke, Z.; Luo, B.; Liu, T.; e al. Sea ching
o p os a e cance by ully au oma ed magne ic esonance imaging classi ica ion: Deep lea ning e sus
non-deep lea ning. Sci. Rep. 2017,7, 15415. [C ossRe ]
37. Ampelio is, D.; An onakoudi, A.; Be be idis, K.; Psa akis, E.; Kounoudes, A. A compu e -aided sys em o
he de ec ion o p os a e cance based on magne ic esonance image analysis. In P oceedings o he 3 d
In e na ional Symposium on Communica ions, Con ol and Signal P ocessing (ISCCSP), S Julians, Mal a,
12–14 Ma ch 2008; pp. 1372–1377. [C ossRe ]
38.
Guo, Y.; Gao, Y.; Shen, D. De o mable MR p os a e segmen a ion ia deep ea u e lea ning and spa se pa ch
ma ching. IEEE T ans. Med. Imaging 2016,35, 1077–1089. [C ossRe ]
39.
Feh , D.; Vee a agha an, H.; Wibme , A.; Gondo, T.; Ma sumo o, K.; Va gas, H.A.; Sala, E.; H icak, H.;
Deasy, J.O. Au oma ic classi ica ion o p os a e cance Gleason sco es om mul ipa ame ic magne ic
esonance images. P oc. Na l. Acad. Sci. USA 2015,112, E6265–E6273. [C ossRe ]
40.
Bha acha jee, S.; Pa k, H.G.; Kim, C.H.; P akash, D.; Madusanka, N.; So, J.H.; Cho, N.H.; Choi, H.K.
Quan i a i e analysis o benign and malignan umo s in his opa hology: P edic ing p os a e cance g ading
using SVM. Appl. Sci. 2019,9, 2969. [C ossRe ]
41.
Jung, W.; Pa k, S.; Jung, K.H.; Hwang, S. P os a e cance segmen a ion using mani old mixup U-Ne .
In P oceedings o he Medical Imaging wi h Deep Lea ning (MIDL), London, UK, 8–10 July 2019.
42.
Ing, N.; Ma, Z.; Li, J.; Salemi, H.; A nold, C.; Knudsen, B.; Ge ych, A. Seman ic segmen a ion o p os a e
cance g ading by con olu ional neu al ne wo ks. Medical Imaging 2018: Digi al Pa hology. P oc. SPIE
2018,10581, 105811B. [C ossRe ]
43.
A ma o, S.G.; Huisman, H.; D uk, K.; Hadjiiski, L.; Ki by, J.S.; Pe ick, N.; Redmond, G.; Gige , M.L.;
Cha, K.; Mamono , A.; e al. PROSTATEx Challenges o compu e ized classi ica ion o p os a e lesions om
mul ipa ame ic magne ic esonance images. J. Med. Imaging 2018,5, 044501. [C ossRe ] [PubMed]
44.
Simonyan, K.; Zisse man, A. Ve y deep con olu ional ne wo ks o la ge-scale image ecogni ion. a Xi
2015, a Xi :1409.1556.
45.
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep esidual lea ning o image ecogni ion. In P oceedings o he
Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), Las Vegas, NV, USA, 26 June–1 July 2016;
pp. 770–778. [C ossRe ]
46.
Russako sky, O.; Deng, J.; Su, H.; K ause, J.; Sa heesh, S.; Ma, S.; Huang, Z.; Ka pa hy, A.; Khosla, A.;
Be ns ein, M.; e al. ImageNe La ge Scale Visual Recogni ion Challenge. In . J. Compu . Vis.
2015
,
115, 211–252. [C ossRe ]
47.
A nab, A.; Zheng, S.; Jayasumana, S.; Rome a-Pa edes, B.; La sson, M.; Ki illo , A.; Sa chynskyy, B.;
Ro he , C.; Kahl, F.; To , P.H. Condi ional andom ields mee deep neu al ne wo ks o seman ic
segmen a ion: Combining p obabilis ic g aphical models wi h deep lea ning o s uc u ed p edic ion.
IEEE Signal P ocess. Mag. 2018,35, 37–52. [C ossRe ]
48.
A an, Y.; Lange , D.L.; Haide , M.A.; Van de Kwas , T.H.; E ans, A.J.; We nick, M.N.; Ye ik, I.S. P os a e
cance segmen a ion wi h mul ispec al MRI using cos -sensi i e condi ional andom ields. In P oceedings
o he IEEE In e na ional Symposium on Biomedical Imaging (ISBI): F om Nano o Mac o, Bos on, MA, USA,
28 June–1 July 2009; pp. 278–281. [C ossRe ]
49.
Ke adec, H.; Dolz, J.; Tang, M.; G ange , E.; Boyko , Y.; Ayed, I.B. Cons ained-CNN losses o weakly
supe ised segmen a ion. Med. Image Anal. 2019,54, 88–99. [C ossRe ]
50.
Jacobs, J.G.; Panagio aki, E.; Alexande , D.C. Gleason G ading o P os a e Tumou s wi h Max-Ma gin
Condi ional Random Fields; In Machine Lea ning in Medical Imaging (MLMI), P oceedings o he 5 h In e na ional
Wo kshop, MLMI 2014, Held in Conjunc ion wi h MICCAI 2014, Bos on, MA, USA, 14 Sep embe 2014; Wu, G.,
Zhang, D., Zhou, L., Eds.; LNCS; Sp inge In e na ional Publishing: Cham, Swi ze land, 2014; Volume 8679,
pp. 85–92. [C ossRe ]
51.
Monaco, J.P.; Tomaszewski, J.E.; Feldman, M.D.; Hagemann, I.; Mo adi, M.; Mousa i, P.; Boag, A.; Da idson,
C.; Abolmaesumi, P.; Madabhushi, A. High- h oughpu de ec ion o p os a e cance in his ological sec ions
using p obabilis ic pai wise Ma ko models. Med. Image Anal. 2010,14, 617–629. [C ossRe ]
Appl. Sci. 2020,10, 338 18 o 19
52.
K ähenbühl, P.; Kol un, V. E icien in e ence in ully connec ed CRFs wi h Gaussian edge po en ials.
In P oceedings o he Ad ances in Neu al In o ma ion P ocessing Sys ems (NIPS), G anada, Spain,
12–14 Decembe 2011; pp. 109–117.
53.
Su on, C.; McCallum, A. An In oduc ion o Condi ional Random Fields. Found. T ends
R

Mach. Lea n.
2012
,
4, 267–373. [C ossRe ]
54.
Chen, L.C.; Papand eou, G.; Kokkinos, I.; Mu phy, K.; Yuille, A.L. DeepLab: Seman ic image segmen a ion
wi h deep con olu ional ne s, a ous con olu ion, and ully connec ed CRFs. IEEE T ans. Pa e n Anal.
Mach. In ell. 2017,40, 834–848. [C ossRe ]
55.
Sh es ha, S.; Vanneschi, L. Imp o ed ully con olu ional ne wo k wi h condi ional andom ields o building
ex ac ion. Remo e Sens. 2018,10, 1135. [C ossRe ]
56.
Cla k, K.; Vend , B.; Smi h, K.; F eymann, J.; Ki by, J.; Koppel, P.; Moo e, S.; Phillips, S.; Ma i , D.;
P ingle, M.; e al. The Cance Imaging A ch. (TCIA): Main aining and ope a ing a public in o ma ion
eposi o y. J. Digi . Imaging 2013,26, 1045–1057. [C ossRe ] [PubMed]
57.
Rundo, L.; Mili ello, C.; Vi abile, S.; Casa ino, C.; Russo, G.; Midi i, M.; Gila di, M.C. Combining
spli -and-me ge and mul i-seed egion g owing algo i hms o u e ine ib oid segmen a ion in MRgFUS
ea men s. Med. Biol. Eng. Compu . 2016,54, 1071–1084. [C ossRe ] [PubMed]
58.
Gulani, V.; Calaman e, F.; Shellock, F.G.; Kanal, E.; Reede , S.B.; In e na ional Socie y o Magne ic Resonance
in Medicine. Gadolinium deposi ion in he b ain: Summa y o e idence and ecommenda ions.
Lance Neu ol.
2017,16, 564–570. [C ossRe ]
59.
Ba e , T.; Law ence, E.M.; P ies , A.N.; Wa en, A.Y.; Gnanap agasam, V.J.; Gallaghe , F.A.; Sala, E.
Repea abili y o di usion-weigh ed MRI o he p os a e using whole lesion ADC alues, skew and his og am
analysis. Eu . J. Radiol. 2019,110, 22–29. [C ossRe ]
60.
She e , R.L.; Glase , Z.A.; Go de sky, J.B.; Nix, J.W.; Po e , K.K.; Rais-Bah ami, S. Compa ison o
bipa ame ic MRI o ull mul ipa ame ic MRI o de ec ion o clinically signi ican p os a e cance .
P os a e Cance P os a ic Dis. 2019,22, 331. [C ossRe ]
61.
Rundo, L.; Tanghe loni, A.; Mili ello, C.; Gila di, M.C.; Mau i, G. Mul imodal medical image egis a ion
using pa icle swa m op imiza ion: A e iew. In P oceedings o he Symposium Se ies on Compu a ional
In elligence (SSCI), A hens, G eece, 6–9 Decembe 2016; pp. 1–8. [C ossRe ]
62.
Maes, F.; Collignon, A.; Vande meulen, D.; Ma chal, G.; Sue ens, P. Mul imodali y image egis a ion by
maximiza ion o mu ual in o ma ion. IEEE T ans. Med. Imaging 1997,16, 187–198. [C ossRe ]
63.
Ga y allidis, E.; B e , M.; Ami bekian, B.; Rokem, A.; Van De Wal , S.; Desco eaux, M.; Nimmo-Smi h, I.
DIPy, a lib a y o he analysis o di usion MRI da a. F on . Neu oin o m. 2014,8, 8. [C ossRe ]
64.
S i as a a, N.; Hin on, G.; K izhe sky, A.; Su ske e , I.; Salakhu dino , R. D opou : A simple way o p e en
neu al ne wo ks om o e i ing. J. Mach. Lea n. Res. 2014,15, 1929–1958.
65. Rude , S. An o e iew o g adien descen op imiza ion algo i hms. a Xi 2016, a Xi :1609.04747.
66. Cholle , F.Ke as. A ailable online: h ps://ke as.io (accessed on 16 Decembe 2019).
67.
Abadi, M.; Aga wal, A.; Ba ham, P.; B e do, E.; Chen, Z.; Ci o, C.; Co ado G.S.; Da is A.; Dean J.;
De in, M.; e al. Tenso Flow: La ge-Scale Machine Lea ning on He e ogeneous Sys ems. 2015. A ailable
online: enso low.o g (accessed on 16 Decembe 2019).
68. Smi h, S.L.; Kinde mans, P.J.; Ying, C.; Le, Q.V. Don’ decay he lea ning a e, inc ease he ba ch size. a Xi
2018, a Xi :1711.00489.
69.
Zhang, C.; Bengio, S.; Ha d , M.; Rech , B.; Vinyals, O. Unde s anding deep lea ning equi es e hinking
gene aliza ion. a Xi 2017, a Xi :1611.03530.
70.
Mon ei o, M.; Figuei edo, M.A.T.; Oli ei a, A.L. Condi ional andom ields as ecu en neu al ne wo ks o
3D medical imaging segmen a ion. a Xi 2018, a Xi :1807.07464.
71.
Cheplygina, V.; de B uijne, M.; Pluim, J.P. No -so-supe ised: A su ey o semi-supe ised, mul i-ins ance,
and ans e lea ning in medical image analysis. Med. Image Anal. 2019,54, 280–296. [C ossRe ] [PubMed]
72.
Han, C.; Rundo, L.; A aki, R.; Nagano, Y.; Fu ukawa, Y.; Mau i, G.; Nakayama, H.; Hayashi, H.
Combining Noise- o-Image and Image- o-Image GANs: B ain MR Image Augmen a ion o Tumo De ec ion.
IEEE Access 2019,7, 156966–156977. [C ossRe ]
Appl. Sci. 2020,10, 338 19 o 19
73.
Han, C.; Ki amu a, Y.; Kudo, A.; Ichinose, A.; Rundo, L.; Fu ukawa, Y.; Umemo o, K.; Li, Y.; Nakayama, H.
Syn hesizing di e se lung nodules whe e e massi ely: 3D mul i-condi ional GAN-based CT image
augmen a ion o objec de ec ion. In P oceedings o he In e na ional Con e ence on 3D Vision (3DV),
Québec Ci y, QC, Canada, 16–19 Sep embe 2019; pp. 729–737. [C ossRe ]
74.
Janke, J.; Cas elli, M.; Popo iˇc, A. Analysis o he p o iciency o ully connec ed neu al ne wo ks in he
p ocess o classi ying digi al images benchma k o di e en classi ica ion algo i hms on high-le el image
ea u es om con olu ional laye s. Expe Sys . Appl. 2019. [C ossRe ]
75.
B indle, K. New app oaches o imaging umou esponses o ea men . Na . Re . Cance
2008
,8, 94.
[C ossRe ]
76.
Lindenbe g, L.; Choyke, P.; Dahu , W. P os a e cance imaging wi h no el PET ace s. Cu . U ol. Rep.
2016
,
17, 18. [C ossRe ]
77.
Ba e , T.; Rieme , F.; McLean, M.A.; Kaggie, J.D.; Robb, F.; Wa en, A.Y.; G a es, M.J.; Gallaghe , F.A.
Molecula imaging o he p os a e: Compa ing o al sodium concen a ion quan i ica ion in p os a e cance
and no mal issue using dedica ed 13C and 23Na endo ec al coils. J. Magn. Reson. Imaging
2019
. [C ossRe ]
78.
G anlund, K.L.; Tee, S.S.; Va gas, H.A.; Lyashchenko, S.K.; Reznik, E.; Fine, S.; Laudone, V.; Eas ham, J.A.;
Touije , K.A.; Reu e , V.E.; e al. Hype pola ized MRI o human p os a e cance e eals inc eased lac a e
wi h umo g ade d i en by Monoca boxyla e T anspo e 1. Cell Me ab. 2019. [C ossRe ]
79.
Rundo, L.; S e ano, A.; Mili ello, C.; Russo, G.; Sabini, M.G.; D’A igo, C.; Ma le a, F.; Ippoli o, M.; Mau i, G.;
Vi abile, S.; e al. A ully au oma ic app oach o mul imodal PET and MR image segmen a ion in Gamma
Kni e ea men planning. Compu . Me hods P og ams Biomed. 2017,144, 77–96. [C ossRe ] [PubMed]
80.
Veliˇcko i´c, P.; Wang, D.; Lane, N.D.; Liò, P. X-CNN: C oss-modal con olu ional neu al ne wo ks o spa se
da ase s. In P oceedings o he Symposium Se ies on Compu a ional In elligence (SSCI), A hens, G eece,
6–9 Decembe 2016; pp. 1–8. [C ossRe ]
c
2020 by he au ho s. Licensee MDPI, Basel, Swi ze land. This a icle is an open access
a icle dis ibu ed unde he e ms and condi ions o he C ea i e Commons A ibu ion
(CC BY) license (h p://c ea i ecommons.o g/licenses/by/4.0/).