Swin transformer for fast MRI

Swin ans o me o as MRI
Jiahao Huang
a,b,
⇑
, Yingying Fang
a
, Yinzhe Wu
a,c
, Huanjun Wu
a,c
, Zhi an Gao
d
, Yang Li
e
,
Ja ie Del Se
,g
, Jun Xia
h
, Guang Yang
a,b,
⇑
a
Na ional Hea and Lung Ins i u e, Impe ial College London, London SW7 2AZ, Uni ed Kingdom
b
Ca dio ascula Resea ch Cen e, Royal B omp on Hospi al, London SW3 6NP, Uni ed Kingdom
c
Depa men o Bioenginee ing, Impe ial College London, London SW7 2AZ, Uni ed Kingdom
d
School o Biomedical Enginee ing, Sun Ya -sen Uni e si y, Guangzhou 510275, China
e
School o Au oma ion Sciences and Elec ical Enginee ing, Beihang Uni e si y, Beijing 100190, China
Depa men o Communica ions Enginee ing, Uni e si y o he Basque Coun y UPV/EHU, Bilbao 48013, Spain
g
TECNALIA, Basque Resea ch and Technology Alliance (BRTA), De io 48160, Spain
h
Depa men o Radiology, Shenzhen Second People’s Hospi al, The Fi s A ilia ed Hospi al o Shenzhen Uni e si y Heal h Science Cen e , Shenzhen 518037, China
a icle in o
A icle his o y:
Recei ed 10 Janua y 2022
Re ised 15 Ma ch 2022
Accep ed 3 Ap il 2022
A ailable online 12 Ap il 2022
Keywo ds:
MRI econs uc ion
T ans o me
Comp essed sensing
Pa allel imaging
abs ac
Magne ic esonance imaging (MRI) is an impo an non-in asi e clinical ool ha can p oduce high-
esolu ion and ep oducible images. Howe e , a long scanning ime is equi ed o high-quali y MR
images, which leads o exhaus ion and discom o o pa ien s, inducing mo e a e ac s due o olun a y
mo emen s o he pa ien s and in olun a y physiological mo emen s. To accele a e he scanning p ocess,
me hods by k-space unde sampling and deep lea ning based econs uc ion ha e been popula ised. This
wo k in oduced SwinMR, a no el Swin ans o me based me hod o as MRI econs uc ion. The whole
ne wo k consis ed o an inpu module (IM), a ea u e ex ac ion module (FEM) and an ou pu module
(OM). The IM and OM we e 2D con olu ional laye s and he FEM was composed o a cascaded o esidual
Swin ans o me blocks (RSTBs) and 2D con olu ional laye s. The RSTB consis ed o a se ies o Swin
ans o me laye s (STLs). The shi ed windows mul i-head sel -a en ion (W-MSA/SW-MSA) o STL
was pe o med in shi ed windows a he han he mul i-head sel -a en ion (MSA) o he o iginal ans-
o me in he whole image space. A no el mul i-channel loss was p oposed by using he sensi i i y maps,
which was p o ed o ese e mo e ex u es and de ails. We pe o med a se ies o compa a i e s udies
and abla ion s udies in he Calga y-Campinas public b ain MR da ase and conduc ed a downs eam seg-
men a ion expe imen in he Mul i-modal B ain Tumou Segmen a ion Challenge 2017 da ase . The
esul s demons a e ou SwinMR achie ed high-quali y econs uc ion compa ed wi h o he benchma k
me hods, and i shows g ea obus ness wi h di e en unde sampling masks, unde noise in e up ion
and on di e en da ase s. The code is publicly a ailable a h ps://gi hub.com/ayanglab/SwinMR.
Ó2022 The Au ho (s). Published by Else ie B.V. This is an open access a icle unde he CC BY license
(h p://c ea i ecommons.o g/licenses/by/4.0/).
1. In oduc ion
Magne ic esonance imaging (MRI) is an impo an non-
in asi e imaging echnique, which enables excellen assessmen s
o s uc u al and unc ional condi ions wi h no adia ion in a
ep oducible manne . Basically, MRI is aimed o econs uc he
images om he obse ed signals whose deg ada ion p ocess can
be o mula ed as ollows:
y¼Fxþn;ð1Þ
whe e x;y2C
N
a e he ec o s deno ing he la en image o econ-
s uc in he image domain and he obse ed measu emen s in k-
space, F2C
NN
is he wo-dimensional (2D) disc e e Fou ie ans-
o m (DFT) and nis he noise ine i ably appea ing in he signal
acquisi ion p ocess.
Howe e , acqui ing he ull measu emen s o y o cons uc a
high-quali y MR image xis highly ime-consuming. Mo eo e ,
he long scanning ime will b ing abou he a e ac s a ising om
he olun a y mo emen s o he pa ien s and in olun a y physio-
logical mo emen s [1]. In o de o mi iga e he long acquisi ion
ime o MRI as well as alle ia e he aliasing a e ac s, a ange o
me hods has been de eloped o accele a ing MRI o ob ain accu-
a e econs uc ions. T adi ionally, g adien e ocusing [2] and
mul iple- adio equency media ed [3] app oaches we e p oposed.
h ps://doi.o g/10.1016/j.neucom.2022.04.051
0925-2312/Ó2022 The Au ho (s). Published by Else ie B.V.
This is an open access a icle unde he CC BY license (h p://c ea i ecommons.o g/licenses/by/4.0/).
⇑
Co esponding au ho a : Na ional Hea and Lung Ins i u e, Impe ial College
London, London SW7 2AZ, Uni ed Kingdom.
E-mail add esses: [email p o ec ed] (J. Huang), [email p o ec ed]
(G. Yang).
Neu ocompu ing 493 (2022) 281–304
Con en s lis s a ailable a ScienceDi ec
Neu ocompu ing
jou nal homepage: www.else ie .com/loca e/neucom
Unde cons ain s o he Nyquis -Shannon sampling heo em, hey
did educe he scanning ime al hough by only a limi ed ac o .
Wi h he de elopmen o he pa allel imaging (PI) and he com-
p essed sensing (CS), he as MRI based on hese wo heo ies
a ac ed much esea ch and ad ancemen s.
Pa allel imaging was in oduced o ake ad an age o spa ial
sensi i i y dis ibu ion de i ed om an a ay o ca e ully dis-
ibu ed ecei e su ace coils, o educe he measu emen om
each coil, alle ia ing he need o enhancing g adien pe o mance
and hence educing he acquisi ion ime [4]. The unde sampled
k-space signal using PI-MRI can be ep esen ed by a gene al model
as:
y
q
¼F
u
ðS
q
xÞþn
q
;q¼1;...S;ð2Þ
whe e S
q
and n
q
a e he sensi i i y map and ine i able noise o he
q
h
coil (Scoils in o al). deno es he pixel-wise mul iplica ion.
F
u
2C
MN
is he unde sampled 2D DFT ma ix wi h MN o
educe he measu emen s o each y
q
. Wi h Scoils applied pa allelly,
one can ob ain y
1
;...;y
S
simul aneously o econs uc he la en
image x. To econs uc hese PI acqui ed images, g ea p og ess
in de eloping PI econs uc ion echniques has aken place, p opos-
ing popula me hods such as he simul aneous acquisi ion o spa ial
ha monic (SMASH) [5], sensi i i y encoding (SENSE) [6] and gene -
alized au o-calib a ing pa ially pa allel acquisi ion (GRAPPA) [7].
The in en ion o CS heo y [8] u he ad anced he sampling
e iciency o MRI. The CS-MRI u ilises he non-linea me hodology
and spa se ans o ma ion o econs uc he la en image om
only a small po ion o k-space measu emen unde a much smal-
le downsampling a e han he Nyquis a e. The gene al p oblem
o MRI using he CS-MRI is o ind he minimise image o he ol-
lowing p oblem:
a gmin
x
jjUxjj
1
;s: :y¼F
u
x;ð3Þ
whe e Uis he spa si ying ans o ma ion, F
u
2C
MN
is unde sam-
pled 2D DFT wi h MN, and y2C
M
is he obse ed unde sampled
measu emen s in k-space. A ange o non-linea econs uc ion
me hods has demons a ed success in esol ing his, including
some ixed spa si ying me hods such as o al a ia ion [9], cu ele s
[10] and double-densi y complex wa ele [11], and a ew adap i e
spa si ying models aking he ad an age o dic iona y lea ning
[12]. While bo h CS-MRI and PI-MRI can signi ican ly educe he
equi ed numbe o measu emen s in k-space, he i e a i e algo-
i hms a e equi ed o de i e he image howe e p olong he econ-
Fig. 1. O e iew o he p oposed SwinMR. (A) and (B) a e he schema ic diag ams o he ecep i e ield o 2D con olu ion (Con 2D), mul i-head sel -a en ion (MSA) and
shi ed windows based mul i-head sel -a en ion (W-MSA/SW-MSA). Con 2D is locally sensi i e and lacks long- ange dependency. Compa ed wi h Con 2D, MSA and (S)W-
MSA ha e la ge ecep i e ields. MSA is pe o med in he whole image space, while W-MSA and SW-MSA a e al e na ingly used in Swin ans o me [36], and pe o med in
shi ed windows. (Red box: he ecep i e ield o he ope a ion; g een box: he pixel; blue box: he pa ch in sel -a en ion.) (C) is he o e iew o SwinMR. (D) shows he
esul s o he p oposed SwinMR compa ed wi h GT, ZF and ano he me hod DAGAN [37]. (IM: he inpu module; FEM: he ea u e ex ac ion module; OM: he ou pu module.
ZF: unde sampled ze o- illed MR images; Recon: econs uc ed MR images; Mul i-Channel GT: mul i-channel g ound u h MR images; GT: single-channel g ound u h MR
images; Mask: he unde sampling mask; SM: sensi i i y maps.).
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
282
s uc ion ime and hence cause conce ns when ans e ed o
ac ual clinical uses.
As a mode n popula me hod o gene al image analysis, deep
lea ning has been e y success ul by exploi ing he non-linea
and complex na u es o he ne wo k wi h supe ised o unsupe -
ised lea ning, and widely applied in medical image esea ch
[13–17]. Con olu ional neu al ne wo ks (CNNs) as a special ype
o deep lea ning ne wo ks enable enhanced la en ea u e ex ac-
ion by hei e y deep hie a chical s uc u e. CNN has demon-
s a ed i s supe io i y in mul iple asks, including de ec ion [18],
classi ica ion [19], segmen a ion [20] and supe - esolu ion [21].
Wang e al. [22] became he pionee o ake ad an age o CNNs
by ex ac ing la en co ela ions be ween unde sampled and ully
sampled k-space da a o MRI econs uc ion. Yang e al. [23] u -
he imp o ed he ne wo k s uc u es by e-applying he al e na -
ing di ec ion me hod o mul iplie s (ADMM), which was o iginally
used o CS-MRI econs uc ion me hods. A cascaded s uc u e was
de eloped by Schlempe e al. [24] o he mo e a ge ed econ-
s uc ion o dynamic sequences in ca diac MRI. To enable u he
la en mapping in he econs uc ion model, Zhu e al. [25] de el-
oped a no el amewo k o p o ide mo e dense mapping h ough
domains ia i s p oposed au oma ed ans o m by mani old
app oxima ion.
Fo a long ime, CNNs ha e had a dominan posi ion in he ield
o compu e ision (CV) since con olu ions a e e ec i e ea u e
ex ac o s. Mos deep lea ning-based MRI econs uc ion me hods
a e based on CNNs, including he GAN-based model. As Fig. 1(A)
shows, he ea u e ex ac ion o CNNs is based on con olu ion,
which is locally sensi i e and lacks long- ange dependency. The
ecep i e ield o CNNs is limi ed by he con olu ional ke nel and
he ne wo k dep h. O e sized con olu ional ke nel b ings huge
compu a ional cos , and o e ly-deep ne wo k dep h can cause g a-
dien anishing.
A no el s uc u e, ans o me , aking ad an age o e en deepe
mapping, sequence- o-sequence model design [26] and adap i e
sel -a en ion se ing [27–30] wi h expanding ecep i e ields
(Fig. 1(A)) [31,32] has been p oposed ecen ly and been popu-
la ised in na u al language p ocessing (NLP) ini ially [33]. Then i
has been applied o objec de ec ion [34], image ecogni ion [35]
and ex ended o supe - esolu ion [31] o gene al image analysis.
Wi h i s supe io abili y in image econs uc ion and syn hesis
as demons a ed in na u al images, we could see ans o me s
applied in MRI in many di e en ways. Fo syn hesis, i has g ea ly
enhanced c oss-modali y image syn hesis (PET- o-MR by di ec-
ional encode [38], T1- o-T2 by a py amid s uc u e [39], and
MR- o-CT and T1/T2/PD by a no el agg ega ed esidual ans-
o me block [40]). Va ian s o he ans o me also enabled
imp o ed pe o mance in econs uc ion and supe - esolu ion
asks. I was i s applied on he econs uc ion o b ain MR imag-
ing [41]. Ko kmaz e al. [42] de eloped an unsupe ised ad e sa -
ial me hod o alle ia e he sca ce aining sample popula ions. To
u he imp o e he quali y o imaging, Feng e al. [43] enabled
an end- o-end join econs uc ion and supe - esolu ion. Feng
e al. [44] u he ad anced he model o hese dual asks by inco -
po a ing he model wi h ask-speci ic no el c oss-a en ion
modules.
Howe e , he shi om NLP asks o CV asks leads o chal-
lenges: (1) Di e ence in scale: isual elemen s (e.g., pixels) in CV
asks end o a y subs an ially in scale unlike language elemen s
(e.g., wo d okens) in NLP asks. (2) Highe esolu ion: he esolu-
ion o pixels in images (o ames) end o be much highe han
wo ds in sen ences. [36] The e o e, i is a ade-o o less compu-
a ional complexi y o limi he scale o sel -a en ion in a local
window, as Fig. 1(A) and (B) shows. Shi ed windows (Swin) ans-
o me [36] eplaced he adi ional mul i-head sel -a en ion
(MSA) by he shi ed windows based mul i-head sel -a en ion
(W-MSA/SW-MSA). W-MSA and SW-MSA we e al e na ingly used
in consecu i e ans o me laye s, since i all a en ion ope a ions
a e conduc ed in ixed windows, he c oss-window ela ionship
may be igno ed. Based on he Swin ans o me module, Liang
e al. [45] p oposed SwinIR o image es o a ion asks.
In his wo k, we in oduced he SwinMR, a no el pa allel imag-
ing coupled Swin ans o me based model o as CS-MRI econ-
s uc ion, as Fig. 1(C) shows. The main con ibu ions can be
summa ised as ollows:
Fig. 2. The da a low o p oposed SwinMR. Roo sum squa e (RSS) is applied o combine he mul i-channel g ound u h MR images (Mu i-Channel GT) o single-channel
g ound u h MR images (GT). Unde sampling and noise in e up ion a e pe o med in k-space using as Fou ie ans o m (FFT) and in e se as Fou ie ans o m (iFFT) o
con e he GT o unde sampled ze o- illed MR images (ZF) as he inpu o ou p oposed SwinMR. Mul i-channel econs uc ed MR images (Mu i-Channel Recon) a e
calcula ed by he pixel-wise mul iplica ion o single-channel econs uc ed MR images (Recon), which a e he ou pu o he p oposed SwinMR, and sensi i i y maps, which
a e es ima ed by ESPIRiT om he Mul i-Channel GT.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
283
A no el pa allel imaging coupled Swin ans o me -based
model o as MRI econs uc ion was p oposed, as Fig. 1(C)
shows.
A no el mul i-channel loss was p oposed by using he sensi i -
i y maps, which was p o ed o p ese e mo e ex u es and
de ails in he econs uc ion esul s.
A se ies o abla ion s udies and compa ison expe imen s we e
conduc ed. Expe imen al s udies using di e en unde sampling
ajec o ies wi h a ious noises we e pe o med o alida e he
obus ness o ou p oposed SwinMR.
A downs eam ask expe imen using a segmen a ion ne wo k
was conduc ed. A p e- ained segmen a ion ne wo k was
applied o es he segmen a ion sco e o econs uc ed images.
2. Me hod
2.1. Classic model-based CS-MRI econs uc ion
To eco e be e spa ial in o ma ion wi h less a e ac s om
he unde sampled k-space da a, adi ional CS-MRI me hods usu-
ally conside sol ing he ollowing op imisa ion p oblem:
min
x
1
2jjF
u
xyjj
2
2
þkRðUxÞ;ð4Þ
whe e Uis he spa si ying ans o m, e.g., disc e e wa ele ans-
o m [46], g adien ope a o [9,47] and dic iona y-based ans o m
[48].RðÞ is he egula isa ion unc ion imposed on he spa si y, e.g,
l
1
-no m and l
0
-no m, and kis he weigh pa ame e o balance he
Fig. 3. The s uc u e o p oposed SwinMR. (A) shows he o e all s uc u e o SwinMR. In SwinMR a chi ec u e, wo Con 2Ds a e placed a he beginning and he ending. A
cascade o RSTBs and a Con 2D wi h a esidual connec ion a e placed be ween he wo Con 2Ds. (B) shows he s uc u e o RSTB. The RSTB consis s o a pa ch embedding
ope a o , Qcascaded STLs, a pa ch unembedding ope a o , a Con 2D, and a esidual connec ion be ween he inpu and ou pu o RSTB. An STL consis s o an LN, an (S)W-MSA,
an LN and an MLP, wi h wo esidual connec ions. (RSTB: he esidual Swin ans o me block; STL: he Swin ans o me laye ; Con 2D: he 2D con olu ional laye ; LN: he
laye no malisa ion laye ; MLP: he mul i-laye pe cep on; (S)W-MSA: he (shi ed) windows mul i-head sel -a en ion. W-MSA and SW-MSA a e al ina i ely used in
consecu i e STLs.)
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
284
wo e ms. The solu ion o he abo e p oblem can be de i ed by he
non-linea op imisa ion sol e s such as g adien -based algo i hms
[49] and a iable spli ing me hods [50,51]. Depending on he man-
ually designed egula isa ion, some models may su e om a long
econs uc ion ime o be e econs uc ion quali y. Addi ionally,
he manually selec ed spa si ying ans o m Ucould also in oduce
undesi able a e ac s, e.g., o al a ia ion based egula isa ion
which is well-known o emo ing he noise and p ese ing he
sha p edges can in oduce s ai case a e ac s [10] and he igh
wa ele ame ans o m inc eases he econs uc ion e iciency
bu may lead o he blocky a e ac s [52].
2.2. CNN-based as MRI econs uc ion
To elie e he a e ac s b ough by he hand-c a ing egula isa-
ion and he long econs uc ion ime o classic models, he deep
CNNs which a e well-known as he powe ul ea u es ex ac o s,
we e i s ly applied in he CS-MRI in [22]. In his wo k, a deep
CNN was applied o lea n he mapping om down-sampled econ-
s uc ion images o ully sampled econs uc ion images di ec ly.
Following ha , se e al ne wo ks ha e been p oposed o u he
imp o e he econs uc ion quali y.
Some wo ks a emp ed o b idge he classic models wi h deep
CNNs by mimicking he i e a i e algo i hm in hei ne wo k a chi-
ec u es. Deep ADMM Ne [23] was i s ly ained by un olding he
op imisa ion algo i hm ADMM o de i e he solu ion o he gene al
model Eq. (4) by ne wo k blocks. In [24], he econs uc ion o he
deep CNN om lowe -quali y images was adop ed as he p io
in o ma ion o app oxima e in a classic CS-model as ollows:
min
x
1
2jjyF
u
xjj
2
2
þkjjx
CNN
ðx
u
jhÞjj
2
2
;ð5Þ
whe e he solu ion o he abo e unc ion was u he adop ed in o
he ne wo k a chi ec u e i e a i ely o imp o e he econs uc ion
esul o
CNN
which akes he ze o- illed econs uc ion x
u
as he
inpu .
On op o he CNNs, condi ional gene a i e ad e sa ial ne wo ks
(cGANs) exploi ed he ad an ages o deep lea ning u he and
p o ed o enhance he quali y o he MR image econs uc ion o
a la ge ex en [53,54]. Such a compe i i e ne wo k in oduced a
wo-playe gene a o -disc imina o aining mechanism o com-
pe i i ely imp o e econs uc ion pe o mance by al e na ingly
op imising h
G
and h
D
o he gene a o Gand he disc imina o D,
in a gene al o m as:
min
h
G
max
h
D
E
xp
g
logD
h
D
ðxÞ

þE
x
u
p
u
log 1 D
h
D
G
h
G
ðx
u
Þ

;ð6Þ
whe e G
h
G
and D
h
D
deno e he gene a o and he disc imina o wi h
pa ame e s h
G
and h
D
, espec i ely. xand x
u
deno e he g ound u h
MR images and unde sampled ze o- illed MR images wi h aliasing
Table 1
Quan i a i e esul s o he compa ison expe imen wi h o he me hods using Gaussian 1D 30% mask (mean (s d)).
y
:p<0:05;
yy
:p<0:01 (compa ed wi h SwinMR (PI) by pai ed
-Tes ).
à
:p<0:05;
àà
:p<0:01 (compa ed wi h SwinMR (nPI) by pai ed -Tes ).
q
: #PARAMs o only he gene a o / o bo h he gene a o and disc imina o . PSNR: Peak signal-
o-noise a io; SSIM: S uc u al simila i y index; FID: F éche incep ion dis ance; In e ence Time: The a e age ime o one in e ence in an In el Co e i9-10980XE CPU o an NVIDIA
RTX 3090 GPU; #PARAMs: The pa ame e s numbe o models; MACs: Mul iply-Accumula e Ope a ions.
Me hods PSNR SSIM FID In e ence Time #PARAMs MACs
CPU (s) GPU (s) (M) (G)
ZF 27.81 (0.83)
yyàà
0.884 (0.012)
yyàà
156.39 – – – –
Deep ADMM Ne 29.24 (0.99)
yyàà
0.922 (0.012)
yyàà
54.56 0.459 (0.052) – – –
U-Ne 31.48 (0.86)
yyàà
0.939 (0.009)
yyàà
46.90 0.166 (0.007) 0.006 (0.000) 32.31 56.44
DAGAN 30.41 (0.83)
yyàà
0.924 (0.010)
yyàà
56.05 0.089 (0.003) 0.003 (0.000) 98.59/127.18
q
33.97
PIDDGAN 31.23 (0.93)
yyàà
0.936 (0.010)
yyàà
17.55 0.166 (0.007) 0.006 (0.000) 32.31/89.50
q
56.44
SwinMR (nPI) 33.06 (1.09)
yy
0.956 (0.009)
yy
21.03 19.310 (0.115) 0.041 (0.001) 11.40 800.73
SwinMR (PI) 32.07 (1.02)
àà
0.945 (0.010)
àà
8.70 19.310 (0.115) 0.041 (0.001) 11.40 800.73
Fig. 4. Samples o he compa ison expe imen wi h g ound u h images (GT), unde sampled ze o- illed images (ZF) and econs uc ed images by o he me hods. Row 1: GT,
ZF and econs uc ed images by di e en me hods; Row 2: Edge in o ma ion ex ac ed by Sobel ope a o ; Row 3: Gaussian 1D 30% mask and he absolu e di e ences be ween
econs uc ed (o ZF) images and GT images (10).
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
285

a e ac s. A e he aining, he gene a o can yield he co espond-
ing econs uc ion om x
u
o econs uc ed images G
h
G
ðx
u
Þ.
Va ian s o gene a o s and disc imina o s ha e been de eloped
o cope wi h mul iple laws in he o iginal a chi ec u e o GAN –
o imp o ed gene a o [55], imp o ed disc imina o [56], loss
unc ions [57], egula isa ion [58], aining s abili y by Wasse -
s ein GAN [59,60] and a en ion mechanism [61]. DAGAN [37],by
subs i u ing he esidual ne wo ks wi h a modi ied U-Ne [62],
combined he ad an age o U-Ne in la en in o ma ion ex ac ion
wi h compe i i e aining and p e- ained VGG based ans e
lea ning. Fu he mo e, PIDDGAN [56] conside ed edge in o ma ion
in o hei model and u he enhance he edge in o ma ion in he
econs uc ion, which a e clinically impo an when in e p e ing
MR images. The u ilisa ion o ans e lea ning imp o ed he gen-
e alisabili y o a ne wo k ained wi h a small da ase [63].
CNN-based MR econs uc ion me hods showed hei supe io -
i y bo h on econs uc ion quali y and e iciency compa ed o clas-
sical MR econs uc ion me hods. Howe e , he pe o mance o
Fig. 5. Peak signal- o-noise a io (PSNR) and S uc u al simila i y index (SSIM) o he expe imen on di e en masks. Fi e unde sampling ajec o ies including Gaussian 1D
10% (G1D10%), Gaussian 1D 30% (G1D30%), Gaussian 1D 50% (G1D50%), adial 10% (R10%) and spi al 10% (S10%) we e applied in his expe imen . (Box ange: in e qua ile
ange; :1% and 99% con idence in e al; : maximum and minimum; : mean; j: median.) The SwinMR (PI) ou pe o ms he DAGAN using di e en unde sampling masks
wi h signi ican ly highe PSNR, SSIM (p<0:05 by pai ed -Tes ).
Table 2
F éche incep ion dis ance (FID) o he expe imen on di e en masks. Fi e
unde sampling masks including Gaussian 1D 10% (G1D10%), Gaussian 1D 30%
(G1D30%), Gaussian 1D 50% (G1D50%), adial 10% (R10%) and spi al 10% (S10%) we e
applied in his expe imen .
Mask SwinMR (PI) DAGAN ZF
G1D10% 28.27 169.83 326.00
G1D30% 8.70 56.05 156.38
G1D50% 5.11 19.26 86.25
R10% 34.19 132.58 319.45
S10% 28.97 115.98 333.40
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
286
hose CNN-based me hods was limi ed by he local sensi i i y o
he con olu ional ope a ion. Mo i a ed by his limi a ion, we p o-
posed a Swin ans o me based MR econs uc ion me hod
SwinMR.
2.3. SwinMR: Swin ans o me o MRI econs uc ion
2.3.1. O e all a chi ec u e
The o e all a chi ec u e is shown in Fig. 1(C) and he da a low
o SwinMR is shown in Fig. 2. Roo sum squa e (RSS) is applied o
combine he mul i-channel g ound u h MR images x
q
o single-
channel g ound u h MR images x(qdeno es he q
h
coil). Sensi i -
i y maps S
q
a e es ima ed by ESPIRiT [64] om mul i-channel
g ound u h MR images x
q
. Unde sampling and noise in e up ion
a e pe o med in k-space using as Fou ie ans o m (FFT) and
in e se as Fou ie ans o m (iFFT) (Gaussian noise is added in
he noise expe imen s), which con e s single-channel g ound
u h MR images x o unde sampled ze o- illed MR images x
u
.
The p oposed SwinMR model can p oduce econs uc ed MR
images ^
x
u
om unde sampled ze o- illed MR images x
u
, whe e
he esidual connec ion is applied o accele a e he con e gence
and s able he aining p ocessing. I can be exp essed by
^
x
u
¼SwinMRðx
u
jhÞþx
u
;ð7Þ
whe e he SwinMR ne wo k is pa ame e ised by h.
Fig. 6. Samples o he expe imen on di e en masks. Fi e unde sampling ajec o ies including Gaussian 1D 10% (G1D10%), Gaussian 1D 30% (G1D30%), Gaussian 1D 50%
(G1D50%), adial 10% (R10%) and spi al 10% (S10%) we e applied in his expe imen . Row 1: Unde sampled ze o- illed MR images (ZF) using di e en masks; Row 2: G ound
u h MR images (GT); Row 3: Recons uc ed MR images by DAGAN; Row 4: Recons uc ed MR images by SwinMR (PI); Row 5: Unde sampling masks. The Peak signal- o-
noise a io (PSNR) and S uc u al simila i y index (SSIM) o econs uc ed and ZF images a e shown in he op-le co ne .
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
287
Fig. 3(A) shows he s uc u e o SwinMR, which is composed o
an inpu module (IM), a ea u e ex ac ion module (FEM) and an
ou pu module (OM). The IM and OM a e a he beginning and
he end o he whole s uc u e, and he FEM is placed be ween
he IM and OM wi h a esidual connec ion. The s uc u e can be
exp essed by
F
IM
¼H
IM
ðx
u
Þ;ð8Þ
F
FEM
¼H
FEM
ðF
IM
Þ;ð9Þ
F
OM
¼H
OM
ðF
FEM
þF
IM
Þ;ð10Þ
whe e he H
IM
ðÞ,H
FEM
ðÞ and H
OM
ðÞ deno e he IM, FEM and OM
espec i ely. F
IM
;F
FEM
and F
OM
deno e he ou pu o he IM, FEM
and OM espec i ely.
2.3.2. Inpu module and ou pu module
The IM is used o ea ly isual p ocessing and mapping om he
inpu image space o highe dimensional ea u e space o he ol-
lowing FEM. The IM applies a 2D con olu ional laye (Con 2D)
mapping x
u
2R
HW1
o F
IM
2R
HWC
. In con as , he OM is used
o map he highe dimensional ea u e space o he ou pu image
space by a Con 2D mapping F
FEM
2R
HWC
o F
OM
2R
HW1
.
In he aining s age, he inpu image is andomly c opped o a
ixed size HW(H¼W). In he in e ence s age, H;Wdeno e he
heigh and weigh o he inpu image. He e we de ine H(o W)
as he pa ch numbe and Cas he channel numbe o he sel -
a en ion p ocessing.
2.3.3. Fea u e ex ac ion module
The FEM is composed o a cascade o esidual Swin ans o me
blocks (RSTBs) and a Con 2D a he end. I can be exp essed as
F
0
¼F
IM
;ð11Þ
F
i
¼H
RSTB
i
ðF
i1
Þ;i¼1;2;...;P;ð12Þ
F
FEM
¼H
CONV
ðF
P
Þ;ð13Þ
Fig. 7. Edge in o ma ion o he expe imen on di e en masks. Fi e unde sampling ajec o ies including Gaussian 1D 10% (G1D10%), Gaussian 1D 30% (G1D30%), Gaussian
1D 50% (G1D50%), adial 10% (R10%) and spi al 10% (S10%) we e applied in his expe imen . Row 1: Edge in o ma ion o unde sampled ze o- illed MR images (ZF) using
di e en masks; Row 2: Edge in o ma ion o g ound u h MR images (GT); Row 3: Edge in o ma ion o econs uc ed MR images by DAGAN; Row 4: Edge in o ma ion o
econs uc ed MR images by SwinMR (PI). The edge in o ma ion was ex ac ed by he Sobel ope a o .
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
288
whe e F
IM
and F
FEM
a e he inpu and ou pu o he FEM. H
RSTB
i
ðÞ
deno es he i
h
RSTB (PRSTBs in o al) in he FEM. H
CONV
ðÞ deno es
he Con 2D a e a se ies o RSTBs.
Fig. 3(B) shows he s uc u e o he RSTB. An RSTB consis s o Q
Swin ans o me laye s (STLs) and a Con 2D, and a esidual con-
nec ion is linked be ween he inpu and ou pu o he RSTB. I
can be exp essed as
F
i;0
¼H
Emb
i
ðF
i1
Þ;ð14Þ
F
i;j
¼H
STL
i;j
ðF
i;j1
Þ;j¼1;2;...;Q;ð15Þ
F
i
¼H
CONV
i
ðH
Unemb
i
ðF
i;Q
ÞþF
i1
Þ;ð16Þ
whe e H
Emb
i
ðÞ is he pa ch embedding om F
i1
2R
HWC
o
F
i;0
2R
HWC
, and H
Unemb
i
ðÞ is he pa ch unembedding om
F
i;Q
2R
HWC
o R
HWC
.
H
STL
i;j
ðÞand H
CONV
i
ðÞdeno e he j
h
STL and he Con 2D in he i
h
RSTB, espec i ely.
2.3.4. Swin ans o me laye
The whole p ocess o he STL can be exp essed as
X
0
¼H
ðSÞWMSA
ðH
LN
ðXÞÞþX;ð17Þ
X
00
¼H
MLP
ðH
LN
ðX
0
ÞÞþX
0
;ð18Þ
whe e Xand X
00
a e he inpu and ou pu o he STL. H
MLP
ðÞ and
H
LN
ðÞdeno e he mul ilaye pe cep on and he laye no malisa ion
laye . Windows mul i-head sel -a en ion (W-MSA) and shi ed
windows mul i-head sel -a en ion (SW-MSA) H
ðSÞWMSA
ðÞa e al e -
na ingly applied in consecu i e STLs.
Spa ial cons ain s a e added in he Swin ans o me laye
compa ed o he o iginal ans o me s. Fig. 3(B) shows he W-
MSA and he SW-MSA compa ed wi h he o iginal MSA. O iginal
MSA pe o ms sel -a en ion in he whole image space. Al hough
he in o ma ion o he en i e pic u e is in ol ed in each a en ion
calcula ion, i agg a a es compu a ional cos s and edundan con-
nec ions. The compu a ional complexi y o he o iginal MSA is as
ollows:
XðH
MSA
Þ¼4HWC
2
þ2ðHWÞ
2
C:ð19Þ
In Swin ans o me laye s, a R
HWC
ea u e map a e di ided
in o
HW
M
2
non-o e lapped windows wi h he size o M
2
C. (S)W-
MSA is calcula ed in each window, ins ead o he whole image
space. The compu a ional complexi y o (S)W-MSA is as ollows:
XðH
ðSÞWMSA
Þ¼4HWC
2
þ2M
2
HWC;ð20Þ
which is signi ican ly educed compa ed o he o iginal MSA. How-
e e , i he sepa a ion o windows is ixed be ween STLs, he ne -
wo k will lose he link be ween di e en windows. No mal
windows and shi ed windows a e al e na ingly u ilised in consec-
u i e STLs o enable in o ma ion communica ion om di e en
windows.
(S)W-MSA o each non-o e lap window Xcan be exp essed by
Q¼XP
Q
;K¼XP
K
;V¼XP
V
;ð21Þ
whe e he P
Q
;P
K
;P
V
a e sha ed p ojec ion ma ices o e all he win-
dows. The que y Q, key K, alue Vand lea nable ela i e posi ion
encoding B(R
M
2
d
) a e used in he calcula ion o he sel -a en ion
mechanism in a local window, which can be exp essed by
A en ionðQ;K;VÞ¼So Max QK
T
=ffiffiffi
d
pþB

V:ð22Þ
Such sel -a en ion mechanism calcula ions a e pe o med o h
imes and conca ena ed o (S)W-MSA. The pseudo-code o STL and
(S)W-MSA a e shown in Algo i hm 1 and Algo i hm 2.
Fig. 8. Absolu e di e ences o s anda dised pixel in ensi ies (10) o he expe imen on di e en masks. Fi e unde sampling ajec o ies including Gaussian 1D 10%
(G1D10%), Gaussian 1D 30% (G1D30%), Gaussian 1D 50% (G1D50%), adial 10% (R10%) and spi al 10% (S10%) we e applied in his expe imen . Row 1: Absolu e di e ences
be ween unde sampled ze o- illed MR images (ZF) using di e en masks and g ound u h MR images (GT); Row 2: Absolu e di e ences be ween econs uc ed MR images
by DAGAN and GT; Row 3: Absolu e di e ences be ween econs uc ed MR images by SwinMR (PI) and GT.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
289
wise and Pe cep ual loss; (3) PF: Pixel-wise and F equency loss;
(4) P: only Pixel-wise loss.
Fig. 16 shows he SSIM, PSNR and FID o SwinMR ained wi h
di e en loss unc ions. Fig. 17 displays he samples o econ-
s uc ed images o SwinMR ained wi h di e en loss unc ions.
Acco ding o Fig. 16, o SwinMR (PI), he u ilisa ion o e-
quency loss ends o imp o e SSIM/PSNR and dec eases he FID
(PFP s PP; PF s P). Fo SwinMR (nPI), he u ilisa ion o equency
loss leads o imp o emen only on SSIM and PSNR, bu sca cely on
FID. In mos cases, he u ilisa ion o he equency loss has a posi-
i e impac on econs uc ion quali y me ics – bo h SSIM/PSNR
and FID.
Fo SwinMR (PI), he u ilisa ion o pe cep ual loss ends o
sligh ly dec ease SSIM and PSNR, bu subs an ially dec eases he
FID (PFP s PF; PP s P). Fo SwinMR (nPI), he u ilisa ion o pe cep-
ual loss ends o achie e a be e FID bu sca cely change SSIM and
PSNR (PFP s PF; PP s P). In mos cases, he u ilisa ion o he pe -
cep ual loss has a posi i e impac on FID, bu a nega i e impac on
SSIM/PSNR when using mul i-channel da a.
3.9. Downs eam ask expe imen s: b ain segmen a ion expe imen s
on B aTS17 da ase
In his expe imen , we pe o med a downs eam ask using a
econs uc ed MR image, in o de o measu e he econs uc ion
quali y. Speci ically, we chose an open-access mul i-modali ies
b ain umou segmen a ion ne wo k
3
[73] o he downs eam ask
Fig. 15. Samples o he abla ion expe imen on he channel numbe using Gaussian 1D 30% mask. Row 1: Recons uc ed MR images by SwinMR (nPI) wi h he di e en
channel numbe s and ze o- illed MR images (ZF); Row 2: Absolu e di e ences (10) be ween econs uc ed MR images by SwinMR (nPI) and g ound u h MR images (GT),
and absolu e di e ences (10) be ween ZF and GT; Row 3: Recons uc ed MR images by SwinMR (PI) wi h he di e en channel numbe and GT; Row 4: Absolu e di e ences
(10) be ween econs uc ed MR images by SwinMR (PI) and GT, and he Gaussian 1D 30% mask.
Fig. 16. S uc u al simila i y index (SSIM), Peak signal- o-noise a io (PSNR) and F éche incep ion dis ance (FID) o he abla ion expe imen on he loss unc ion using
Gaussian 1D 30% mask. PFP: pixel-wise, equency and pe cep ual loss; PP: pixel-wise and pe cep ual loss; PF: pixel-wise and equency loss; P: only pixel-wise loss.
3
h ps://gi hub.com/Meh dad-Noo i/B ain-Tumo -Segmen a ion.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
296

expe imen s. This segmen a ion ne wo k adop ed a U-Ne [62] based
a chi ec u e wi h he u ilisa ion o esidual blocks and s ided con-
olu ion downsampling compa ed o he anilla U-Ne . In addi ion,
his segmen a ion ne wo k also employed he Squeeze-and-
Exci a ion Block [74] on conca ena ed mul i-le el ea u es o chan-
nel a en ion mechanism.
The segmen a ion ne wo k was ained on he B aTS17 da ase
( ou modali ies a e equi ed including FLAIR, T1, T1CE and T2).
Then, we ained ou SwinMR weigh s using B aTS17 FLAIR, T1,
T1CE and T2 da a, espec i ely. A e ha , segmen a ion asks we e
conduc ed on GT MR images, SwinMR econs uc ed MR images
and ZF MR images di ec ly using he p e- ained segmen a ion ne -
Fig. 17. Samples o he abla ion expe imen on he loss unc ion using Gaussian 1D 30% mask. PFP: pixel-wise, equency and pe cep ual loss; PP: pixel-wise and pe cep ual
loss; PF: pixel-wise and equency loss; P: only pixel-wise loss. Row 1: Recons uc ed MR images by SwinMR (nPI) and ze o- illed MR images (ZF); Row 2: Edge in o ma ion o
econs uc ed MR images by SwinMR (nPI) and edge in o ma ion o ZF; Row 3: Absolu e di e ences (10) be ween econs uc ed MR images by SwinMR (nPI) and g ound
u h MR images (GT), and absolu e di e ences (10) be ween ZF and GT; Row 4: Recons uc ed MR images by SwinMR (PI) and GT; Row 5: Edge in o ma ion o
econs uc ed MR images by SwinMR (PI) and edge in o ma ion o GT; Row 6: Absolu e di e ences (10) be ween econs uc ed MR images by SwinMR (PI) and GT, and he
Gaussian 1D 30% mask.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
297
wo k. Ideally, he segmen a ion sco e o econs uc ed images and
GT images should be as close as possible.
Table 4 shows he esul o SwinMR ained wi h B aTS17 FLAIR,
T1, T1CE and T2 espec i ely. Fig. 18 displays he samples o he
econs uc ion o di e en modali ies. Tables 5 and 6 show he
IoU and Dice sco e o he segmen a ion ask. Fig. 19 displays he
sample o he segmen a ion ask.
Acco ding o Tables 5 and 6, he IoU and Dice sco e o econ-
s uc ed MR images a e imp o ed compa ed wi h ZF MR images
and much close o he sco e o GT MR images. Acco ding o he
Mann–Whi ney Tes , he IoU and Dice sco e dis ibu ions o he
econs uc ed MR images using he Gaussian 1D 30% mask a e
no signi ican ly di e en om he dis ibu ions o he GT MR
images (p>0:05).
4. Discussion
In his wo k, a no el Swin ans o me based model, i.e.,
SwinMR, o as MRI econs uc ion has been p oposed. Mos
exis ing deep lea ning based image es o a ion me hods, including
MRI econs uc ion app oaches, a e based on CNNs. The con olu-
ion is a e y e ec i e ea u e ex ac o bu lacks long- ange
dependency. The ecep i e ield o CNNs is limi ed by he size o
he ke nel and he dep h o he ne wo k. To ackle his p oblem,
esea che s ha e de eloped ans o me s based image es o a ion
me hods ha ha e been o iginally used o sol ing NLP asks.
The co e o he ans o me is MSA, which has global sensi i i y.
In MSA ope a ion, each pa ch can link wi h any o he pa ches in
he whole image space bu also agg a a es he compu a ional
bu den.
Howe e , we ha e belie ed ha in MRI econs uc ion, he
MSA, which is ope a ed in he whole image space, is edundan
and no necessa y. I is no di icul o unde s and ha in NLP asks
he i s and he las wo ds may ha e a s ong connec ion in a sen-
ence. Howe e , his may no be applicable in CV asks. Visual ele-
men s (e.g., pixels) in CV asks can a y subs an ially in scale
unlike language elemen s (e.g., wo d okens) in NLP asks [36].
Since in mos cases, o example, he op-le co ne pa ch has no
ela ionship wi h he bo om- igh co ne pa ch wi hin an image.
Mo eo e , o MRI econs uc ion, he bigges di icul y is he
eco e y o de ailed in o ma ion and ex u e in o ma ion. Focusing
oo much on global in o ma ion and igno ing he de ailed (local)
in o ma ion may make he image smoo he and lose mo e de ails.
The u ilisa ion o a Swin ans o me can achie e a ade-o o CV
asks. In Swin ans o me , ope a ions a e conduc ed in shi ed
windows ins ead o he whole images. I has a la ge ecep i e ield
compa ed o CNNs bu is no o e ly conce ned wi h global in o -
ma ion. This is he eason why we ha e de eloped a Swin ans-
o me o MRI econs uc ion.
To e alua e ou p oposed me hods, se e al compa ison expe i-
men s and abla ion s udies ha e been conduc ed. In his s udy, we
ha e compa ed ou p oposed SwinMR wi h benchma k MRI econ-
s uc ion me hods. The esul s in Table 1 ha e demons a ed ha
ou SwinMR has achie ed he highes SSIM/PSNR and lowes FID
compa ed o CNN-based and GAN-based models. F om Fig. 4,we
ha e shown clea ly ha ou SwinMR has ob ained be e econ-
s uc ion quali y, especially in he zoom-in a ea, whe e he de ails
o he ce ebellum ha e been well-p ese ed.
In his s udy, we ha e also compa ed SwinMR (PI) ha has been
ained wi h mul i-channel b ain da a wi h SwinMR (nPI) ha has
been ained wi h single-channel b ain da a. The esul s ha e led o
a simila conclusion in ou p e ious s udy [56], whe e FID o he
model ained wi h mul i-channel da a has been be e compa ed
o he model ained wi h single-channel da a, and he SSIM/PSNR
has shown he opposi e (i.e., SSIM/PSNR: nPI >PI; FID: PI <nPI).
This phenomenon can also be obse ed in he subsequen abla ion
expe imen s. F om Fig. 4, we can ind ha he econs uc ed
images o SwinMR (PI) ha e shown mo e de ails and ex u e in o -
ma ion, bu he econs uc ed images o SwinMR (nPI) ha e shown
smoo he .
The expe imen al esul s ha e demons a ed ha he h ee
me ics ha compa ed PI and nPI ga e di e en answe s. We ha e
specula ed ha his migh be due o he di e en p inciples o
hese me ics. PSNR is a classic me ic based on pe -pixel compa -
isons, which a e no able o e lec he s uc u e in o ma ion o
images. SSIM is a pe cep ual me ic ha measu es s uc u e simi-
la i y. Howe e , bo h o hem a e based on simple and shallow
unc ions and di ec compa isons be ween images, which is insu -
icien o accoun o many nuances o human pe cep ion [71]. Fo
FID, he compa ison is based on pe cep ion and pe o med on wo
se s o images. Images a e mapped o high-dimension ep esen a-
ions by a p e- ained Incep ionV3 ne wo k, which is well- ela ed
o human isual pe cep ion. The SwinMR (PI) econs uc ed images
ha e demons a ed mo e de ails and ex u e in o ma ion. E en
hough hese de ails and ex u e in o ma ion may no be so accu-
a e, hey make he econs uc ed images mo e isually simila wi h
Table 4
Quan i a i e esul s o econs uc ed images by SwinMR (Recon) and ze o- illed images (ZF) on B aTS17 da ase (mean (s d)). PSNR: Peak signal- o-noise a io; SSIM: S uc u al
simila i y index; FID: F éche incep ion dis ance. G1D10%: Gaussian 1D 10% mask; G1D30%: Gaussian 1D 30% mask.
Mask Me ics Recon
FLAIR T1 T1CE T2
G1D10% PSNR 30.07 (1.99) 33.80 (2.30) 33.80 (1.84) 32.20 (1.81)
SSIM 0.751 (0.043) 0.760 (0.046) 0.797 (0.049) 0.745 (0.039)
FID 38.02 32.97 31.46 21.84
G1D30% PSNR 37.97 (2.42) 41.08 (3.36) 42.29 (2.12) 38.37 (2.02)
SSIM 0.942 (0.013) 0.953 (0.012) 0.953 (0.015) 0.937 (0.016)
FID 5.94 4.80 4.39 8.95
Mask Me ics ZF
FLAIR T1 T1CE T2
G1D10% PSNR 23.87 (1.64) 25.92 (1.48) 25.92 (1.70) 23.92 (1.79)
SSIM 0.388 (0.070) 0.414 (0.061) 0.414 (0.068) 0.431 (0.057)
FID 225.70 234.52 227.51 219.09
G1D30% PSNR 28.74 (1.78) 28.82 (1.60) 30.60 (1.82) 29.46 (2.01)
SSIM 0.597 (0.046) 0.602 (0.051) 0.602 (0.051) 0.632 (0.038)
FID 91.18 100.98 106.28 85.49
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
298
he g ound u h images. Howe e , he SwinMR (nPI) econ-
s uc ed images ha e shown smoo he in pixel-wise scale, a he
cos o less de ail and ex u e in o ma ion. The e o e, SwinMR
(PI) ha e ended o ha e be e FID and wo se SSIM/PSNR com-
pa ed o SwinMR (nPI), due o he p inciple di e ences o he e al-
ua ion me hods.
Fig. 18. Samples o econs uc ion esul s o SwinMR on B aTS17 da ase including FLAIR, T1, T1CE and T2 MR images. Row 1: G ound u h MR images (GT); Row 2: Ze o-
illed MR images (ZF) unde sampled by Gaussian 1D 10% mask (G1D10%); Row 3: Recons uc ed MR images unde sampled by G1D10%; Row 4: ZF unde sampled by Gaussian
1D 30% mask (G1D30%); Row 5: Recons uc ed MR images unde sampled by G1D30%.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
299
Table 5
In e sec ion o e union (IoU) o he segmen a ion expe imen (median/mean [Q1,Q3]).
q
:p<0:05;
qq
:p<0:01 (compa ed wi h GT by Mann–Whi ney Tes ). GT: g ound u h
MR images; Recon: econs uc ed MR images by SwinMR; ZF: unde sampled ze o- illed MR images. G1D10%: Gaussian 1D 10% mask; G1D30%: Gaussian 1D 30% mask. WT: Whole
umou ; TC: Enhancing umou ; ET: Tumou co e.
IoU GT Recon ZF
G1D10% WT 0.930/0.924 [0.900,0.954] 0.898/0.899 [0.868,0.940]
qq
0.838/0.836 [0.795,0.881]
qq
TC 0.821/0.771 [0.726,0.903] 0.758/0.722 [0.661,0.890]
qq
0.617/0.539 [0.393,0.733]
qq
ET 0.772/0.735 [0.625,0.889] 0.740/0.652 [0.471,0.846]
qq
0.570/0.527 [0.336,0.694]
qq
G1D30% WT 0.930/0.924 [0.900,0.954] 0.924/0.921 [0.895,0.953] 0.897/0.897 [0.862,0.945]
qq
TC 0.821/0.771 [0.726,0.903] 0.811/0.766 [0.719,0.904] 0.763/0.728 [0.669,0.895]
qq
ET 0.772/0.735 [0.625,0.889] 0.770/0.725 [0.616,0.883] 0.748/0.697 [0.573,0.859]
qq
Table 6
Dice sco e o he segmen a ion expe imen (median/mean [Q1,Q3]).
q
:p<0:05;
qq
:p<0:01 (compa ed wi h GT by Mann–Whi ney Tes ). GT: g ound u h MR images; Recon:
econs uc ed MR images by SwinMR; ZF: unde sampled ze o- illed MR images. G1D10%: Gaussian 1D 10% mask; G1D30%: Gaussian 1D 30% mask. WT: Whole umou ; TC:
Enhancing umou ; ET: Tumou co e.
Dice GT Recon ZF
G1D10% WT 0.968/0.965 [0.952,0.981] 0.950/0.950 [0.933,0.974]
qq
0.916/0.914 [0.892,0.940]
qq
TC 0.904/0.857 [0.845,0.951] 0.863/0.819 [0.800,0.944]
qq
0.767/0.653 [0.566,0.847]
qq
ET 0.874/0.835 [0.777,0.941] 0.852/0.766 [0.640,0.917]
qq
0.725/0.665 [0.503,0.820]
qq
G1D30% WT 0.968/0.965 [0.952,0.981] 0.964/0.963 [0.948,0.980] 0.949/0.949 [0.930,0.975]
qq
TC 0.904/0.857 [0.845,0.951] 0.897/0.854 [0.838,0.951] 0.868/0.826 [0.803,0.947]
qq
ET 0.874/0.835 [0.777,0.941] 0.871/0.827 [0.765,0.939] 0.857/0.808 [0.729,0.925]
qq
Fig. 19. Samples o segmen a ion esul s o SwinMR on he B aTS17 da ase . Col 1: Segmen a ion e e ence; Col 2: Segmen a ion p edic ion using GT images; Col3:
Segmen a ion p edic ion using ze o- illed MR images (ZF) unde sampled by Gaussian 1D 10% mask (G1D10%); Col 4: Segmen a ion p edic ion using econs uc ed MR images
unde sampled by G1D10%; Col 5: Segmen a ion p edic ion using ZF unde sampled by Gaussian 1D 30% mask (G1D30%); Col 6: Segmen a ion p edic ion using econs uc ed
MR images unde sampled by G1D30%. Blue a ea: Whole umou (WT); Red a ea: Enhancing umou (ET); G een a ea: Tumou co e (TC).
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
300
F om Table 1, we can ind a common p oblem o ans o me -
based me hods, which is he highe compu a ional cos compa ed
o o he CNN-based and GAN-based me hods. Eq. (20) ha e shown
ha he compu a ional complexi y is p opo ional o he HW o he
inpu o (S)W-MSA. The ime shown in Table 1 has been he in e -
ence ime, whe e he o iginal heigh and weigh ha e been ea ed
as Hand W(256 256 he e). Fo aining, andomly c opping ha e
been applied o ease he long p ocessing ime.
Expe imen s using di e en unde sampling masks wi h a ious
noise le els ha e demons a ed ha ou p oposed me hod SwinMR
ha e shown supe io i y o DAGAN in all he es s. The e alua ion
me ics change as expec ed when he condi ion changes (di e en
masks and noise le els).
Abla ion s udies on he pa ch numbe and he channel numbe
ha e demons a ed ha econs uc ion quali y has been imp o ed
as he pa ch numbe has been inc eased and has g adually been
sa u a ed, acco ding o Fig. 13(A) and (C). Howe e , acco ding o
Eq. (20), he compu a ional complexi y also has been inc eased
as he pa ch numbe has been inc eased. As a ade-o , we ha e
se he pa ch numbe o 96. Beyond ou expec a ions, he changing
o channel numbe has no been posi i ely co ela ed wi h he
e alua ion me ics in his expe imen , acco ding o Fig. 13(B) and
(D). We ha e assumed ha he e alua ion me ics ha e sa u a ed
in he ange o channel numbe in his expe imen . Empi ically,
we ha e se he channel numbe o 180 acco ding o he de aul
se ing o SwinIR.
Abla ion s udies on di e en loss unc ions ha e been con-
duc ed. As expec ed, he u ilisa ion o he pixel-wise loss and he
equency loss has mainly cons ained he ideli y o econs uc-
ion, and he u ilisa ion o pe cep ual VGG loss has ocused on pe -
cep ion, which has been well- ela ed o he human isual sys em.
The e o e, he u ilisa ion o equency loss has had a posi i e
impac on SSIM and PSNR, which has been mo e sensi i e o he
ideli y o econs uc ion. The u ilisa ion o pe cep ual loss has
had a posi i e impac on FID, which has been based on pe cep ion.
The e a e s ill some limi a ions o ou wo k. Fi s , in he (S)W-
MSA ope a ion, he size o windows is ixed. Inspi ed by Google-
Ne , mul i-scale windows could be inco po a ed and esul s om
di e en scales could be me ged in he (S)W-MSA. Second, he
hea y compu a ional cos is s ill an obs acle o he de elopmen
o ans o me s. The imp o emen ha ans o me s b ing is a
he sac i ice o inc eased compu a ional cos . A ligh weigh ans-
o me model could be a po en ial u u e esea ch di ec ion.
5. Conclusion
In his wo k, we ha e de eloped he SwinMR, a no el pa allel
imaging coupled Swin ans o me -based model o as mul i-
channel MRI econs uc ion. The p oposed me hod has ou pe -
o med o he benchma k CNN-based and GAN-based MRI econ-
s uc ion me hods. I has also shown excellen obus ness using
di e en unde sampling ajec o ies wi h a ious noises.
Decla a ion o Compe ing In e es
The au ho s decla e ha hey ha e no known compe ing inan-
cial in e es s o pe sonal ela ionships ha could ha e appea ed
o in luence he wo k epo ed in his pape .
Acknowledgemen s
This wo k was suppo ed in pa by he UK Resea ch and Inno-
a ion Fu u e Leade s Fellowship [MR/V023799/1], in pa by he
Medical Resea ch Council [MC/PC/21013], in pa by he Eu opean
Resea ch Council Inno a i e Medicines Ini ia i e [DRAGON,
H2020-JTI-IMI2 101005122], in pa by he AI o Heal h Imaging
Awa d [CHAIMELEON, H2020-SC1-FA-DTS-2019-1 952172], in pa
by he B i ish Hea Founda ion [P ojec Numbe : TG/18/5/34111,
PG/16/78/32402], in pa by he NVIDIA Academic Ha dwa e G an
P og am, in pa by he P ojec o Shenzhen In e na ional Coope -
a ion Founda ion [GJHZ20180926165402083], in pa by he Bas-
que Go e nmen h ough he ELKARTEK unding p og am [KK-
2020/00049], and in pa by he consolida ed esea ch g oup
MATHMODE [IT1294-19].
Re e ences
[1] J. Zbon a , F. Knoll, A. S i am, T. Mu ell, Z. Huang, M.J. Muckley, A. De azio, R.
S e n, P. Johnson, M. B uno, M. Pa en e, K.J. Ge as, J. Ka snelson, H. Chanda ana,
Z. Zhang, M. D ozdzal, A. Rome o, M. Rabba , P. Vincen , N. Yakubo a, J.
Pinke on, D. Wang, E. Owens, C.L. Zi nick, M.P. Rech , D.K. Sodickson, Y.W. Lui,
Fas MRI: An open da ase and benchma ks o accele a ed MRI, a Xi e-p in s
(2018) a Xi :1811.08839.
[2] M.K. S ehling, R. Tu ne , P. Mans ield, Echo-plana imaging: Magne ic
esonance imaging in a ac ion o a second, Science 254 (5028) (1991) 43–50.
[3] J. Hennig, A. Naue h, H. F iedbu g, RARE imaging: A as imaging me hod o
clinical MR, Magn. Reson. Med. 3 (6) (1986) 823–833, h ps://doi.o g/10.1002/
m m.1910030602.
[4] M. Blaime , F. B eue , M. Muelle , R.M. Heidemann, M.A. G iswold, P.M. Jakob,
SMASH, SENSE, PILS, GRAPPA: How o choose he op imal me hod, Top. Magn.
Reson. Imaging 15 (4) (2004) 223–236.
[5] D.K. Sodickson, W.J. Manning, Simul aneous acquisi ion o spa ial ha monics
(SMASH): Fas imaging wi h adio equency coil a ays, Magn. Reson. Med. 38
(4) (1997) 591–603.
[6] K.P. P uessmann, M. Weige , M.B. Scheidegge , P. Boesige , SENSE: Sensi i i y
encoding o as MRI, Magne ic Resonance in Medicine: An O icial Jou nal o
he In e na ional Socie y o , Magn. Reson. Med. 42 (5) (1999) 952–962.
[7] M.A. G iswold, P.M. Jakob, R.M. Heidemann, M. Ni ka, V. Jellus, J. Wang, B.
Kie e , A. Haase, Gene alized au ocalib a ing pa ially pa allel acquisi ions
(GRAPPA), Magn. Reson. Med. 47 (2002) 1202–1210, h ps://doi.o g/10.1002/
m m.10171.
[8] D. Donoho, Comp essed sensing, IEEE T ans. In . Theo y 52 (2006) 1289–1306,
h ps://doi.o g/10.1109/TIT.2006.871582.
[9] K.T. Block, M. Uecke , J. F ahm, Unde sampled adial MRI wi h mul iple coils.
I e a i e image econs uc ion using a o al a ia ion cons ain , Magn.
Resonance Med. 57 (6) (2007) 1086–1098.
[10] M. Beladgham, I.B. Hacene, A. Taleb-Ahmed, M. Khéli , MRI images
comp ession using cu ele s ans o ms, in: AIP Con e ence P oceedings, Vol.
1019, Ame ican Ins i u e o Physics, 2008, pp. 249–253.
[11] Z. Zhu, K. Wahid, P. Babyn, R. Yang, Comp essed sensing-based MRI
econs uc ion using complex double-densi y dual- ee DWT, J. Biomed.
Imaging (2013 ( 2013).), h ps://doi.o g/10.1155/2013/907501.
[12] S. Ra ishanka , Y. B esle , MR image econs uc ion om highly unde sampled
k-space da a by dic iona y lea ning, IEEE T ans. Med. Imaging 30 (5) (2011)
1028–1041, h ps://doi.o g/10.1109/TMI.2010.2090538.
[13] N. Zeng, Z. Wang, H. Zhang, K.-E. Kim, Y. Li, X. Liu, An imp o ed pa icle il e
wi h a no el hyb id p oposal dis ibu ion o quan i a i e analysis o gold
immunoch oma og aphic s ips, IEEE T ans. Nano echnol. 18 (2019) 819–829,
h ps://doi.o g/10.1109/TNANO.2019.2932271.
[14] N. Zeng, H. Li, Y. Peng, A new deep belie ne wo k-based mul i- ask lea ning
o diagnosis o Alzheime ’s disease, Neu al Compu . Appl. (2021).
[15] P. Wu, H. Li, N. Zeng, F. Li, FMD-Yolo: An e icien ace mask de ec ion me hod
o COVID-19 p e en ion and con ol in public, Image Vis. Compu . 117 (2022)
104341.
[16] Y. Chen, C.-B. Schönlieb, P. Liò, T. Leine , P.L. D ago i, G. Wang, D. Ruecke , D.
Fi min, G. Yang, AI-based econs uc ion o as MRI-a sys ema ic e iew and
me a-analysis, P oceedings o he IEEE 110 (2) (2022) 224–245.
[17] M. Baka o , D. Radosa , Deep lea ning and medical diagnosis: A e iew o
li e a u e, Mul imodal Technol. In e ac ion 2 (3) (2018).
[18] R. Gi shick, J. Donahue, T. Da ell, J. Malik, Rich ea u e hie a chies o accu a e
objec de ec ion and seman ic segmen a ion, in: P oceedings o he IEEE
Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), 2014.
[19] C. Szegedy, W. Liu, Y. Jia, P. Se mane , S. Reed, D. Anguelo , D. E han, V.
Vanhoucke, A. Rabino ich, Going deepe wi h con olu ions, in: P oceedings o
he IEEE Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR),
2015.
[20] J. Long, E. Shelhame , T. Da ell, Fully con olu ional ne wo ks o seman ic
segmen a ion, in: P oceedings o he IEEE Con e ence on Compu e Vision and
Pa e n Recogni ion (CVPR), 2015.
[21] C. Dong, C.C. Loy, K. He, X. Tang, Image supe - esolu ion using deep
con olu ional ne wo ks, IEEE T ans. Pa e n Anal. Mach. In ell. 38 (2016)
295–307, h ps://doi.o g/10.1109/TPAMI.2015.2439281.
[22] S. Wang, Z. Su, L. Ying, X. Peng, S. Zhu, F. Liang, D. Feng, D. Liang, Accele a ing
magne ic esonance imaging ia deep lea ning, in: 2016 IEEE 13 h
In e na ional Symposium on Biomedical Imaging (ISBI), 2016, pp. 514–517,
h ps://doi.o g/10.1109/ISBI.2016.7493320.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
301

[23] Y. Yang, J. Sun, H. Li, Z. Xu, Deep ADMM-Ne o comp essi e sensing MRI, in:
Ad ances in Neu al In o ma ion P ocessing Sys ems, Vol. 29, Cu an Associa es
Inc, 2016.
[24] J. Schlempe , J. Caballe o, J.V. Hajnal, A.N. P ice, D. Ruecke , A deep cascade o
con olu ional neu al ne wo ks o dynamic MR image econs uc ion, IEEE
T ans. Med. Imaging 37 (2018) 491–503, h ps://doi.o g/10.1109/
TMI.2017.2760978.
[25] B. Zhu, J.Z. Liu, S.F. Cauley, B.R. Rosen, M.S. Rosen, Image econs uc ion by
domain- ans o m mani old lea ning, Na u e 555 (2018) 487–492, h ps://doi.
o g/10.1038/na u e25988.
[26] I. Su ske e , O. Vinyals, Q.V. Le, Sequence o sequence lea ning wi h neu al
ne wo ks, in: Ad ances in Neu al In o ma ion P ocessing Sys ems, Vol. 27,
Cu an Associa es Inc, 2014.
[27] A. Vaswani, N. Shazee , N. Pa ma , J. Uszko ei , L. Jones, A.N. Gomez, Ł. Kaise , I.
Polosukhin, A en ion is all you need, in: Ad ances in neu al in o ma ion
p ocessing sys ems, 2017, pp. 5998–6008.
[28] A.P. Pa ikh, O. Täcks öm, D. Das, J. Uszko ei , A Decomposable A en ion
Model o Na u al Language In e ence, a Xi e-p in s (2016)
a Xi :1606.01933.
[29] J. Cheng, L. Dong, M. Lapa a, Long Sho -Te m Memo y-Ne wo ks o Machine
Reading, a Xi e-p in s (2016) a Xi :1601.06733.
[30] C. Ma soukas, J. F edin Haslum, M. Söde be g, K. Smi h, Is i ime o eplace
CNNs wi h ans o me s o medical images?, a Xi e-p in s (2021)
a Xi :2108.09038.
[31] N. Pa ma , A. Vaswani, J. Uszko ei , L. Kaise , N. Shazee , A. Ku, D. T an, Image
ans o me , in: P oceedings o he 35 h In e na ional Con e ence on Machine
Lea ning, Vol. 80 o P oceedings o Machine Lea ning Resea ch, PMLR, 2018,
pp. 4055–4064.
[32] T. Salimans, A. Ka pa hy, X. Chen, D.P. Kingma, PixelCNN++: Imp o ing he
PixelCNN wi h disc e ized logis ic mix u e likelihood and o he modi ica ions,
a Xi e-p in s (2017) a Xi :1701.05517.
[33] X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, X. Huang, P e- ained models o na u al
language p ocessing: A su ey, Science China Technological Sciences (2020) 1–
26.
[34] N. Ca ion, F. Massa, G. Synnae e, N. Usunie , A. Ki illo , S. Zago uyko, End- o-
end objec de ec ion wi h ans o me s, in: Compu e Vision – ECCV 2020,
Sp inge In e na ional Publishing, Cham, 2020, pp. 213–229.
[35] A. Doso i skiy, L. Beye , A. Kolesniko , D. Weissenbo n, X. Zhai, T. Un e hine ,
M. Dehghani, M. Minde e , G. Heigold, S. Gelly, J. Uszko ei , N. Houlsby, An
image is wo h 16x16 wo ds: T ans o me s o image ecogni ion a scale,
a Xi e-p in s (2020) a Xi :2010.11929.
[36] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, B. Guo, Swin T ans o me :
Hie a chical Vision T ans o me using Shi ed Windows, a Xi e-p in s (2021)
a Xi :2103.14030.
[37] G. Yang, S. Yu, H. Dong, G. Slabaugh, P.L. D ago i, X. Ye, F. Liu, S. A idge, J.
Keegan, Y. Guo, D. Fi min, DAGAN: Deep de-aliasing gene a i e ad e sa ial
ne wo ks o as comp essed sensing MRI econs uc ion, IEEE T ans. Med.
Imaging 37 (2018) 1310–1321, h ps://doi.o g/10.1109/TMI.2017.2785879.
[38] H.-C. Shin, A. Ihsani, S. Manda a, S. Tu u eke e S eeni as, C. Fo s e , J. Cha, A.
Disease Neu oimaging Ini ia i e, GANBERT: Gene a i e ad e sa ial ne wo ks
wi h bidi ec ional encode ep esen a ions om ans o me s o MRI o PET
syn hesis, a Xi e-p in s (2020) a Xi :2008.04393.
[39] X. Zhang, X. He, J. Guo, N. E ehadi, N. Aw, D. Semanek, J. Posne , A. Laine, Y.
Wang, PTNe : A high- esolu ion in an MRI syn hesize based on ans o me ,
a Xi e-p in s (2021) a Xi :2105.13993.
[40] O. Dalmaz, M. Yu , T. Çuku , ResViT: Residual ision ans o me s o mul i-
modal medical image syn hesis, a Xi e-p in s (2021) a Xi :2106.16031.
[41] Y. Ko kmaz, M. Yu , S.U.H. Da , T. Cuku , Deep MRI econs uc ion wi h
gene a i e ision ans o me s, in: Machine Lea ning o Medical Image
Recons uc ion, Sp inge In e na ional Publishing, Cham, 2021, pp. 54–64.
[42] Y. Ko kmaz, S.U. Da , M. Yu , M. Özbey, T. Çuku , Unsupe ised MRI
econs uc ion ia ze o-sho lea ned ad e sa ial ans o me s, IEEE T ans.
Med. Imaging (2022), h ps://doi.o g/10.1109/TMI.2022.3147426, 1–1.
[43] C.-M. Feng, Y. Yan, H. Fu, L. Chen, Y. Xu, Task ans o me ne wo k o join MRI
econs uc ion and supe - esolu ion, in: Medical Image Compu ing and
Compu e Assis ed In e en ion – MICCAI 2021, Sp inge In e na ional
Publishing, Cham, 2021, pp. 307–317.
[44] C.-M. Feng, Y. Yan, G. Chen, H. Fu, Y. Xu, L. Shao, Accele a ed mul i-modal MR
imaging wi h ans o me s, a Xi e-p in s (2021) a Xi :2106.14248.
[45] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, R. Timo e, SwinIR: Image
es o a ion using swin ans o me , in: P oceedings o he IEEE/CVF
In e na ional Con e ence on Compu e Vision (ICCV), Wo kshops, 2021, pp.
1833–1844.
[46] X. Qu, D. Guo, B. Ning, Y. Hou, Y. Lin, S. Cai, Z. Chen, Unde sampled MRI
econs uc ion wi h pa ch-based di ec ional wa ele s, Magne ic esonance
imaging 30 (7) (2012) 964–977.
[47] T. Wu, D.Z. Wang, Z. Jin, J. Zhang, Sol ing cons ained TV2L1-L2 MRI signal
econs uc ion ia an e icien al e na ing di ec ion me hod o mul iplie s,
Nume ical Ma hema ics: Theo y, Me hods and Applica ions 10 (4) (2017)
895–912.
[48] S. Ra ishanka , Y. B esle , MR image econs uc ion om highly unde sampled
k-space da a by dic iona y lea ning, IEEE T ans. Med. Imaging 30 (5) (2010)
1028–1041.
[49] M. Lus ig, D. Donoho, J.M. Pauly, Spa se MRI: The applica ion o comp essed
sensing o apid MR imaging, Magne ic Resonance in Medicine: An O icial
Jou nal o he In e na ional Socie y o , Magn. Reson. Med. 58 (6) (2007) 1182–
1195.
[50] J. Yang, Y. Zhang, W. Yin, A as al e na ing di ec ion me hod o TVL1-L2 signal
econs uc ion om pa ial ou ie da a, IEEE J. Sel. Top. Signal P ocess. 4 (2)
(2010) 288–297.
[51] L. Wang, K. Lu, P. Liu, Comp essed sensing o a emo e sensing image based on
he p io s o he e e ence image, IEEE Geosci. Remo e Sens. Le . 12 (4) (2014)
736–740.
[52] J.-F. Cai, J.K. Choi, K. Wei, Da a d i en igh ame o comp essed sensing MRI
econs uc ion ia o - he-g id egula iza ion, SIAM J. Imag. Sci. 13 (3) (2020)
1272–1301.
[53] G. Yang, J. L , Y. Chen, J. Huang, J. Zhu, Gene a i e Ad e sa ial Ne wo k
Powe ed Fas Magne ic Resonance Imaging—Compa a i e S udy and New
Pe spec i es, Sp inge In e na ional Publishing, Cham, 2022, pp. 305–339.
[54] J. L , J. Zhu, G. Yang, Which GAN? A compa a i e s udy o gene a i e
ad e sa ial ne wo k-based as MRI econs uc ion, Philos. T ans. R. Soc. A 379
(2021) 20200203, h ps://doi.o g/10.1098/ s a.2020.0203.
[55] R. Shaul, I. Da id, O. Shi i , T.R. Ra i , Subsampled b ain MRI econs uc ion
by gene a i e ad e sa ial neu al ne wo ks, Med. Image Anal. 65 (2020),
h ps://doi.o g/10.1016/j.media.2020.101747 101747.
[56] J. Huang, W. Ding, J. L , J. Yang, H. Dong, J. Del Se , J. Xia, T. Ren, S. Wong, G.
Yang, Edge-enhanced dual disc imina o gene a i e ad e sa ial ne wo k o
as MRI wi h pa allel imaging using mul i- iew in o ma ion, Appl. In ell.
(2021), h ps://doi.o g/10.1007/s10489-021-03092-w.
[57] T.M. Quan, T. Nguyen-Duc, W.-K. Jeong, Comp essed sensing MRI
econs uc ion using a gene a i e ad e sa ial ne wo k wi h a cyclic loss,
IEEE T ans. Med. Imaging 37 (2018) 1488–1497, h ps://doi.o g/10.1109/
TMI.2018.2820120.
[58] Y. Ma, J. Liu, Y. Liu, H. Fu, Y. Hu, J. Cheng, H. Qi, Y. Wu, J. Zhang, Y. Zhao,
S uc u e and illumina ion cons ained gan o medical image enhancemen ,
IEEE T ans. Med. Imaging (2021), h ps://doi.o g/10.1109/TMI.2021.3101937,
1–1.
[59] M. A jo sky, S. Chin ala, L. Bo ou, Wasse s ein gene a i e ad e sa ial
ne wo ks, in: P oceedings o he 34 h In e na ional Con e ence on Machine
Lea ning, Vol. 70 o P oceedings o Machine Lea ning Resea ch, PMLR, 2017,
pp. 214–223.
[60] Y. Guo, C. Wang, H. Zhang, G. Yang, Deep a en i e wasse s ein gene a i e
ad e sa ial ne wo ks o MRI econs uc ion wi h ecu en con ex -
awa eness, in: Medical Image Compu ing and Compu e Assis ed
In e en ion – MICCAI 2020, Sp inge In e na ional Publishing, Cham, 2020,
pp. 167–177.
[61] M. Jiang, M. Zhi, L. Wei, X. Yang, J. Zhang, Y. Li, P. Wang, J. Huang, G. Yang, FA-
GAN: Fused a en i e gene a i e ad e sa ial ne wo ks o MRI image supe -
esolu ion, Compu . Med. Imaging G aph. 92 (2021), h ps://doi.o g/10.1016/
j.compmedimag.2021.101969 101969.
[62] O. Ronnebe ge , P. Fische , T. B ox, U-Ne : Con olu ional ne wo ks o
biomedical image segmen a ion, in: Medical Image Compu ing and
Compu e -Assis ed In e en ion – MICCAI 2015, Sp inge In e na ional
Publishing, Cham, 2015, pp. 234–241.
[63] J. L , G. Li, X. Tong, W. Chen, J. Huang, C. Wang, G. Yang, T ans e lea ning
enhanced gene a i e ad e sa ial ne wo ks o mul i-channel MRI
econs uc ion, Compu . Biol. Med. 134 (2021), h ps://doi.o g/10.1016/
j.compbiomed.2021.104504 104504.
[64] M. Uecke , P. Lai, M.J. Mu phy, P. Vi ue, M. Elad, J.M. Pauly, S.S. Vasanawala,
M. Lus ig, ESPIRiT-an eigen alue app oach o au ocalib a ing pa allel MRI:
Whe e SENSE mee s GRAPPA, Magn. Reson. Med. 71 (3) (2014) 990–1001.
[65] W.-S. Lai, J.-B. Huang, N. Ahuja, M.-H. Yang, Fas and accu a e image Supe -
Resolu ion wi h deep laplacian py amid ne wo ks, IEEE T ans. Pa e n Anal.
Mach. In ell. 41 (11) (2019) 2599–2613, h ps://doi.o g/10.1109/
TPAMI.2018.2865304.
[66] R. Souza, O. Lucena, J. Ga a a, D. Gobbi, M. Saluzzi, S. Appenzelle , L. Ri ne , R.
F ayne, R. Lo u o, An open, mul i- endo , mul i- ield-s eng h b ain MR
da ase and analysis o publicly a ailable skull s ipping me hods
ag eemen , Neu oImage 170 (2018) 482–494, segmen ing he B ain. doi:
10.1016/j.neu oimage.2017.08.021.
[67] B.H. Menze, A. Jakab, S. Baue , J. Kalpa hy-C ame , K. Fa ahani, J. Ki by, Y.
Bu en, N. Po z, J. Slo boom, R. Wies , e al., The mul imodal b ain umo image
segmen a ion benchma k (BRATS), IEEE T ans. Med. Imaging 34 (10) (2014)
1993–2024.
[68] S. Bakas, H. Akba i, A. So i as, M. Bilello, M. Rozycki, J.S. Ki by, J.B. F eymann, K.
Fa ahani, C. Da a zikos, Ad ancing he cance genome a las glioma MRI
collec ions wi h expe segmen a ion labels and adiomic ea u es, Scien i ic
Da a 4 (1) (2017) 1–13.
[69] S. Bakas, M. Reyes, A. Jakab, S. Baue , M. Remp le , A. C imi, R. Takeshi
Shinoha a, C. Be ge , S.M. Ha, M. Rozycki, M. P as awa, E. Albe s, J. Lipko a, J.
F eymann, J. Ki by, M. Bilello, H. Fa hallah-Shaykh, R. Wies , J. Ki schke, B.
Wies le , R. Colen, A. Ko o sou, P. Lamon agne, D. Ma cus, M. Milchenko, A.
Naze i, M.-A. Webe , A. Mahajan, U. Baid, E. Ge s ne , D. Kwon, G. Acha ya, M.
Aga wal, M. Alam, A. Albiol, A. Albiol, F.J. Albiol, V. Alex, N. Allinson, P.H.A.
Amo im, A. Am u ka , G. Anand, S. Ande ma , T. A bel, P. A belaez, A. A e y,
M. Azma , B. P anjal, W. Bai, S. Bane jee, B. Ba h, T. Ba chelde , K.
Ba manghelich, E. Ba is ella, A. Bee s, M. Belyae , M. Bendszus, E. Benson, J.
Be nal, H. Naga aja Bha a h, G. Bi os, S. Bisdas, J. B own, M. Cabezas, S. Cao, J.
M. Ca doso, E.N. Ca e , A. Casami jana, L. Sil ana Cas illo, M. Ca à, P. Ca in, A.
Ce igues, V.S. Chagas, S. Chand a, Y.-J. Chang, S. Chang, K. Chang, J. Chazalon, S.
Chen, W. Chen, J.W. Chen, Z. Chen, K. Cheng, A.R. Choudhu y, R. Chylla, A.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
302
Clé igues, S. Colleman, R. Ge man Rod iguez Colmei o, M. Combalia, A. Cos a,
X. Cui, Z. Dai, L. Dai, L.A. Daza, E. Deu sch, C. Ding, C. Dong, S. Dong, W. Dudzik,
Z. Ea on-Rosen, G. Egan, G. Escude o, T. Es ienne, R. E e son, J. Fab izio, Y. Fan,
L. Fang, X. Feng, E. Fe an e, L. Fidon, M. Fische , A.P. F ench, N. F idman, H. Fu,
D. Fuen es, Y. Gao, E. Ga es, D. Ge ing, A. Gholami, W. Gie ke, B. Glocke , M.
Gong, S. González-Villá, T. G osges, Y. Guan, S. Guo, S. Gup a, W.-S. Han, I.S.
Han, K. Ha mu h, H. He, A. He nández-Saba é, E. He mann, N. Him hani, W.
Hsu, C. Hsu, X. Hu, X. Hu, Y. Hu, Y. Hu, R. Hua, T.-Y. Huang, W. Huang, S. Van
Hu el, Q. Huo, V. HV, K.M. I ekha uddin, F. Isensee, M. Islam, A.S. Jackson, S.R.
Jambawalika , A. Jesson, W. Jian, P. Jin, V.J.M. Jose, A. Jungo, B. Kainz, K.
Kamni sas, P.-Y. Kao, A. Ka nawa , T. Kelle meie , A. Ke mi, K. Keu ze , M.
Ta ek Khadi , M. Khened, P. Kickinge ede , G. Kim, N. King, H. Knapp, U.
Knech , L. Kohli, D. Kong, X. Kong, S. Koppe s, A. Ko i, G. K ishnamu hi, E.
K i o , P. Kuma , K. Kushiba , D. Lachino , T. Lamb ou, J. Lee, C. Lee, Y. Lee, M.
Lee, S. Le ko i s, L. Le ko i s, J. Le i , T. Li, H. Li, W. Li, H. Li, X. Li, Y. Li, H. Li, Z.
Li, X. Li, Z. Li, X. Li, W. Li, Z.-S. Lin, F. Lin, P. Lio, C. Liu, B. Liu, X. Liu, M. Liu, J. Liu,
L. Liu, X. Llado, M. Mo eno Lopez, P. Ribal a Lo enzo, Z. Lu, L. Luo, Z. Luo, J. Ma,
K. Ma, T. Mackie, A. Madabushi, I. Mahmoudi, K.H. Maie -Hein, P. Maji, C.
Mammen, A. Mang, B.S. Manjuna h, M. Ma cinkiewicz, S. McDonagh, S.
McKenna, R. McKinley, M. Mehl, S. Meh a, R. Meh a, R. Meie , C. Meinel, D.
Me ho , C. Meye , R. Mille , S. Mi a, A. Moiyadi, D. Molina-Ga cia, M.A.B.
Mon ei o, G. M ukwa, A. My onenko, J. Nalepa, T. Ngo, D. Nie, H. Ning, C. Niu,
N.K. Nuech e lein, E. Oe mann, A. Oli ei a, D.D.C. Oli ei a, A. Oli e , A.F.I.
Osman, Y.-N. Ou, S. Ou selin, N. Pa agios, M.S. Pa k, B. Paschke, J.G. Pauloski, K.
Pawa , N. Pawlowski, L. Pei, S. Peng, S.M. Pe ei a, J. Pe ez-Be e a, V.M. Pe ez-
Ga cia, S. Pezold, B. Pham, A. Phophalia, G. Piella, G.N. Pillai, M. Pi aud, M.
Piso , A. Popli, M.P. Pound, R. Pou eza, P. P asanna, V. P ko ska, T.P. P idmo e,
S. Puch, É. Puyba eau, B. Qian, X. Qiao, M. Rajchl, S. Rane, M. Rebsamen, H. Ren,
X. Ren, K. Re anu u, M. Rezaei, O. Rippel, L.C. Ri e a, C. Robe , B. Rosen, D.
Ruecke , M. Sa wan, M. Salem, J. Sal i, I. Sanchez, I. Sánchez, H.M. San os, E.
Sa o , D. Schellinge hou , K. Scheu ele, M.R. Sco , A.A. Scussel, S. Sedla , J.P.
Se ano-Rubio, N.J. Shah, N. Shah, M. Shaikh, B.U. Shanka , Z. Shboul, H. Shen,
D. Shen, L. Shen, H. Shen, V. Shenoy, F. Shi, H.E. Shin, H. Shu, D. Sima, M.
Sinclai , O. Smedby, J.M. Snyde , M. Sol aninejad, G. Song, M. Soni, J. S awiaski,
S. Sub amanian, L. Sun, R. Sun, J. Sun, K. Sun, Y. Sun, G. Sun, S. Sun, Y.R. Su e , L.
Szilagyi, S. Talba , D. Tao, D. Tao, Z. Teng, S. Thaku , M.H. Thaku , S. Tha akan, P.
Tiwa i, G. Tochon, T. T an, Y.M. Tsai, K.-L. Tseng, T.A. Tuan, V. Tu lapo , N.
Tus ison, M. Vakalopoulou, S. Val e de, R. Vangu i, E. Vasilie , J. Ven u a, L.
Ve a, T. Ve cau e en, C.A. Ve as o, L. Vidya a ne, V. Vilaplana, A.
Vi ekanandan, G. Wang, Q. Wang, C.J. Wang, W. Wang, D. Wang, R. Wang, Y.
Wang, C. Wang, G. Wang, N. Wen, X. Wen, L. Weninge , W. Wick, S. Wu, Q. Wu,
Y. Wu, Y. Xia, Y. Xu, X. Xu, P. Xu, T.-L. Yang, X. Yang, H.-Y. Yang, J. Yang, H. Yang,
G. Yang, H. Yao, X. Ye, C. Yin, B. Young-Moxon, J. Yu, X. Yue, S. Zhang, A. Zhang,
K. Zhang, X. Zhang, L. Zhang, X. Zhang, Y. Zhang, L. Zhang, J. Zhang, X. Zhang, T.
Zhang, S. Zhao, Y. Zhao, X. Zhao, L. Zhao, Y. Zheng, L. Zhong, C. Zhou, X. Zhou, F.
Zhou, H. Zhu, J. Zhu, Y. Zhuge, W. Zong, J. Kalpa hy-C ame , K. Fa ahani, C.
Da a zikos, K. an Leempu , B. Menze, Iden i ying he bes machine lea ning
algo i hms o b ain umo segmen a ion, p og ession assessmen , and o e all
su i al p edic ion in he BRATS challenge, a Xi e-p in s (2018)
a Xi :1811.02629.
[70] M. Heusel, H. Ramsaue , T. Un e hine , B. Nessle , S. Hoch ei e , Gans ained
by a wo ime-scale upda e ule con e ge o a local nash equilib ium,
Ad ances in neu al in o ma ion p ocessing sys ems 30 (2017).
[71] R. Zhang, P. Isola, A.A. E os, E. Shech man, O. Wang, The un easonable
e ec i eness o deep ea u es as a pe cep ual me ic, in: P oceedings o he
IEEE Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), 2018.
[72] M.S. Hansen, P. Kellman, Image econs uc ion: An o e iew o clinicians, J.
Magn. Reson. Imaging 41 (3) (2015) 573–585, h ps://doi.o g/10.1002/
jm i.24687.
[73] M. Noo i, A. Bah i, K. Mohammadi, A en ion-guided e sion o 2D UNe o
au oma ic b ain umo segmen a ion, in: 2019 9 h In e na ional Con e ence
on Compu e and Knowledge Enginee ing (ICCKE), 2019, pp. 269–275, h ps://
doi.o g/10.1109/ICCKE48569.2019.8964956.
[74] J. Hu, L. Shen, G. Sun, Squeeze-and-exci a ion ne wo ks, in: P oceedings o he
IEEE Con e ence on Compu e Vision and Pa e n Recogni ion (CVPR), 2018.
Jiahao Huang is a Ph.D. s uden a Na ional Hea and
Lung Ins i u e, Impe ial College London. He ecei ed his
B.S. deg ee in Op oelec onics In o ma ion Science and
Enginee ing om Beijing Ins i u e o Technology in
2021. His esea ch in e es s include compu e ision,
machine lea ning, and medical image p ocessing and
analysis.
Yingying Fang is a Pos doc Resea ch Assis an in he
Na ional Hea and Lung Ins i u e o Impe ial College
London. She ob ained he Ph.D. deg ee om Hong Kong
Bap is Uni e si y in 2020. He esea ch in e es s
include image es o a ion, op imisa ion and deep
lea ning. He esea ch is cu en ly ocused on he
medical image p ocessing o lung disease diagnosis and
p ognosis.
Yinzhe Wu is an unde g adua e s uden a he Depa -
men o Bioenginee ing o Impe ial College London, and
also an unde g adua e esea ch s uden unde he
supe ision o D . Guang Yang a Na ional Hea and
Lung Ins i u e o Impe ial College London.
Huanjun Wu is an unde g adua e s uden s udying
Molecula Bioenginee ing a he Depa men o Bio-
enginee ing, Impe ial College London. She has joined D .
Guang Yang’s esea ch g oup a Na ional Hea and Lung
Ins i u e o Impe ial College London since June 2021 as a
UROP s uden , cu en ly doing esea ch on MRI econ-
s uc ion.
Zhi an Gao is an associa e p o esso a School o
Biomedical Enginee ing, Sun Ya -sen Uni e si y (SYSU).
Be o e joining SYSU, he was a pos doc a Schulich School
o Medicine & Den is y, Wes e n Uni e si y (UWO),
Canada.
Yang Li is a P o esso wi h he School o Au oma ion
Sciences and Elec ical Enginee ing, Beihang Uni e si y,
Beijing, China. His esea ch a ea is in ol ed sys em
iden i ica ion and modeling o complex nonlinea
p ocesses, signal p ocessing and da a modeling, image
p ocessing, and machine lea ning.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
303
Ja ie Del Se is a p incipal esea che in da a analy ics
and op imiza ion a TECNALIA (Spain), and a p o esso
a he Uni e si y o he Basque Coun y (UPV/EHU).
Jun Xia is a p o esso a he Depa men o Radiology, Shenzhen Second People’s
Hospi al, The Fi s A ilia ed Hospi al o Shenzhen Uni e si y Heal h Science Cen e .
D Guang Yang is a Fu u e Leade s Fellow (Tenu ed
Senio Resea ch Fellow) in he Na ional Hea and Lung
Ins i u e a Impe ial College London. He is also an
Hono a y Senio Lec u e in he School o Biomedical
Enginee ing & Imaging Sciences a King’s College Lon-
don. His esea ch g oup is in e es ed in de eloping
no el and ansla ional echniques o imaging and
biomedical da a analysis. His g oup ocuses on he
esea ch and de elopmen on da a-d i en as imaging,
da a ha monisa ion, image segmen a ion, image syn-
hesis, ede a ed lea ning, explainable AI e c. He is
cu en ly wo king on a wide ange o clinical applica-
ions in ca dio ascula disease, lung disease and oncology. Read mo e in o ma ion
abou Yang’s Lab a : h ps://www.yanglab. yi/.
J. Huang, Y. Fang, Y. Wu e al. Neu ocompu ing 493 (2022) 281–304
304