Llomba e al. EURASIP Jou nal on Audio, Speech, and Music
P ocessing (2021) 2021:1
h ps://doi.o g/10.1186/s13636-020-00191-3
RESEARCH Open Access
P og essi e loss unc ions o speech
enhancemen wi h deep neu al ne wo ks
Jo ge Llomba *, Dayana Ribas, An onio Miguel, Luis Vicen e, Al onso O ega and Edua do Lleida
Abs ac
The p og essi e pa adigm is a p omising s a egy o op imize ne wo k pe o mance o speech enhancemen
pu poses. Recen wo ks ha e shown di e en s a egies o imp o e he accu acy o speech enhancemen solu ions
based on his mechanism. This pape s udies he p og essi e speech enhancemen using con olu ional and esidual
neu al ne wo k a chi ec u es and explo es wo c i e ia o loss unc ion op imiza ion: weigh ed and uni o m
p og essi e. This wo k ca ies ou he e alua ion on simula ed and eal speech samples wi h e e be a ion and added
noise using REVERB and VoiceHome da ase s. Expe imen al esul s show a a ie y o achie emen s among he loss
unc ion op imiza ion c i e ia and he ne wo k a chi ec u es. Resul s show ha he p og essi e design s eng hens
he model and inc eases he obus ness o dis o ions due o e e be a ion and noise.
Keywo ds: P og essi e loss unc ion, Speech enhancemen , ResNe , CNN
1 In oduc ion
Mos deep neu al ne wo k speech enhancemen (DNN-
SE) me hods ac like a monoli hic block, whe e he noisy
signal is he inpu o he a chi ec u e and he enhanced
signal is he ou pu , while in e media e signals a e no
easily in e p e able. Howe e , SE can also be pe o med
as a g adual imp o emen p ocess, wi h a s ep-by-s ep
speech denoising. In his pa adigm, he signal is enhanced
p og essi ely a di e en sys em s ages, by inc emen ally
imp o ing he speech quali y a each s age in e ms o
noise educ ion, speech dis o ion, e c.
The inc emen al SE pa adigm has been ecen ly
app oached h ough he so-called p og essi e speech
enhancemen (PSE) [1–3]. In his mechanism, he ne -
wo k lea ning p ocess is decomposed in mul iple s ages,
such ha he a ge is p og essi ely op imized. This way,
he subp oblem sol ed a each s age can boos he sub-
sequen lea ning in he nex s ages. P e ious wo ks ol-
lowing his s a egy ha e shown imp o ed esul s o
he p og essi e a chi ec u es compa ed o usual DNN-SE
me hods.
*Co espondence: [email p o ec ed]
ViVoLab, A agón Ins i u e o Enginee ing Resea ch (I3A), Uni e si y o
Za agoza, C/ Ped o Ce buna 12, 50009 Za agoza, Spain
P e ious p og essi e p oposals ha e ocused on he
inc emen al signal- o-noise a io (SNR) econs uc ion
a di e en deg ees. In [2], a eed o wa d deep neu al
ne wo k implemen ed a eg ession scheme, whe e he ne -
wo k a ge was lea ning an ideal bina y mask esponsible
o imp o ing he SNR h ee imes in 10 dB. The same
example was used wi h di e en SNR o achie e he p o-
g essi e enhancemen . In [3], he au ho s ex ended his
wo k by es ing mo e ad anced a chi ec u es. Ini ially, a
ep oduc ion o he p ocedu e in [2]usingalongsho -
e m memo y cell (LSTM) showed a deg ada ion o he
SE pe o mance wi h he numbe o a ge laye s. Then, a
each cleaning s ep, hey used addi ional knowledge om
he p e ious s eps, inally achie ing an imp o emen in
pe o mance.
Mo e ecen ly and mo i a ed by he in e p e abili y o
he enhancemen p ocess, we ha e p esen ed a p og es-
si e a chi ec u e based on wide esidual ne wo ks [1]. Ou
main goal was o unde s and he enhancemen p ocess,
s ep by s ep, by using a isualiza ion p obe a each ne -
wo k block. Insigh s p o ided by he in e p e a ion o he
enhancemen p ocess led o he modi ica ion o he ne -
wo k a chi ec u e, which p o ided imp o ed esul s o
he SE p ocess. In he p oposed a chi ec u e, he mean
© The Au ho (s). 2021 Open Access This a icle is licensed unde a C ea i e Commons A ibu ion 4.0 In e na ional License, which
pe mi s use, sha ing, adap a ion, dis ibu ion and ep oduc ion in any medium o o ma , as long as you gi e app op ia e c edi
o he o iginal au ho (s) and he sou ce, p o ide a link o he C ea i e Commons licence, and indica e i changes we e made. The
images o o he hi d pa y ma e ial in his a icle a e included in he a icle’s C ea i e Commons licence, unless indica ed
o he wise in a c edi line o he ma e ial. I ma e ial is no included in he a icle’s C ea i e Commons licence and you in ended
use is no pe mi ed by s a u o y egula ion o exceeds he pe mi ed use, you will need o ob ain pe mission di ec ly om he
copy igh holde . To iew a copy o his licence, isi h p://c ea i ecommons.o g/licenses/by/4.0/.
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 2o 16
squa e e o (MSE) o he log-spec al ampli ude (LSA)
be ween he enhanced signal and he e e ence is com-
pu ed a e e y ne wo k s age and e eshes he backp op-
aga ion g adien s. Fu he mo e, he econs uc ion e o
o each block con ibu es o he op imiza ion loss unc ion
wi h a weigh ed p og essi e mechanism.
Ou p elimina y app oach o his p oblem had he
in en ion o jus p esen ing a p og essi e app oach o
DNN speech enhancemen [1]. Now, his wo k deeply
s udies he p og essi e s a egy o DNN-SE. This pape
explo es he gene aliza ion o he aining me hod on
wo consolida ed DNN a chi ec u es used o SE asks:
a con olu ional neu al ne wo k (CNN) and a esidual
neu al ne wo k (ResNe ). This s udy analyzes wo di e -
en c i e ia o implemen he p og essi e pa adigm: he
weigh ed p og essi e (WP) c i e ion in [1]andanewly
p oposed uni o m p og essi e c i e ion (UP). The UP
c i e ion implemen s he inal op imiza ion o he loss
unc ion, conside ing ha he econs uc ion e o s om
all blocks con ibu e in he same way. Mo eo e , in his
wo k, we conside no only he de e e be a ion p oblem
bu he whole enhancemen p oblem. Also, a wide expe -
imen al se up is implemen ed, including simula ed and
eal da ase s.
Mo e ecen DNN a chi ec u es used o SE such as
gene a i e ad e sa ial ne wo ks (GAN) [4], U-Ne [5], o
esidual hou glass ecu en neu al ne wo ks (RHR-Ne )
[6] ha e demons a ed hei capabili ies and cu en ly hey
o e he bes esul s. Despi e hese a chi ec u es could
also bene i om he use o he p oposed me hod, in his
wo k, we concen a e on he pe o mance on a selec ed
se o e y well-known, simple, and es ablished a chi ec-
u es o show he bene i s in e ms o pe o mance wi h-
ou negligible inc ease in compu a ional complexi y ( e y
educed a aining ime and no compu a ional inc ease a
in e ence ime) o he p og essi e app oach dis ega ding
he speci ic me hod o ne wo k a chi ec u e.
The con ibu ions o his wo k a e:
•S udy o he PSE on wo consolida ed deep neu al
ne wo k (DNN) a chi ec u es: CNN and ResNe .
•Assessmen o wo c i e ia o p og essi e loss
unc ion op imiza ion: weigh ed and uni o m.
•Explo ing he space o inpu ea u es.
•Analysis o he p og essi e mechanism e ec on
g adien s and speech quali y measu es.
The es o he pape is o ganized as ollows. Sec ion 2
summa izes he an eceden s o his wo k. Sec ion 3goes
deepe in o he applica ion o he p og essi e pa adigm
o he loss unc ion. Sec ion 4desc ibes he expe imen al
condi ions. Sec ion 5p esen s some p elimina y esul s
on he anishing g adien p oblem, and Sec ion 6analyzes
he beha io o he CNN/ResNe a chi ec u es when hey
a e using he p og essi e pa adigm by p esen ing ob ained
esul s. Finally, Sec ion 7concludes he pape .
2 An eceden s
The a chi ec u es conside ed in his wo k a e CNN and
ResNe . In o de o adap hese a chi ec u es o he
p og essi e pa adigm, i is necessa y o add addi ional
es ic ions and modi y he loss unc ion. In he ollowing
subsec ions, we p o ide an o e iew o he a chi ec u e
design and he loss unc ion ha will be he base o his
wo k.
2.1 A chi ec u e
A chi ec u es based on CNN a e capable o exploi ing
local pa e ns in he spec um om bo h equency and
empo al domains [7,8]. The e ec o noise and e e be -
a ion appea s as a pe u ba ion o he signal spec al shape
ex ended h ough a speci ic ime- equency a ea. The
na u al s uc u e o he speech signal o he dis o ion pa -
e ns can show co ela ion in consecu i e ime- equency
bins in a con ex . CNN-based a chi ec u es e ec i ely
deal wi h his cha ac e is ic o he speech signal s uc u e,
wha makes hem app op ia e o speech enhancemen
pu poses. CNN has also appea ed combined wi h ecu -
en blocks o u he model he dynamic co ela ions
among consecu i e ames [9]. In Fig. 1,weshowa ypical
s uc u e o a CNN whe e each a chi ec u e block could
ha e di e en con igu a ions in e ms o con olu ional
laye s, ba ch no maliza ion, o non-linea i ies.
The inco po a ion o esidual connec ions b ough
a egula iza ion po en ial o he CNN app oach [10].
ResNe a chi ec u e makes use o sho cu connec ions
be ween neu al ne wo k laye s, allowing sys ems o han-
dle mo e dep h, wi h as e con e gence and a smalle
g adien anishing e ec . Since hey can manage deepe
ne wo ks, hey can be mo e exp essi e, p o ide mo e
de ailed ep esen a ions o he unde lying s uc u e o he
co up ed signal and manage longe con ex s. All o his
esul s in mo e accu a ely enhanced speech. We show his
modi ica ion in Fig. 2, whe e we desc ibe he connec ion
be ween con olu ional blocks in a esidual app oach.
In [1], we added o he ResNe an addi ional cons ain :
he a chi ec u e kep a cons an numbe o channels
along all he blocks o he DNN. The cons an num-
be o channels allowed he ou pu econs uc ion and
a isualiza ion p obe a any in e nal block. The manda-
o y p og essi e signal econs uc ion o ced an inc e-
men al p ocess o he SE ha ended o imp o e he
obus ness o he model. Besides, his a chi ec u e uses a
weigh ed composi ion o econs uc ion e o s by block
o pe o m he loss unc ion op imiza ion. This way, each
block makes pa ial econs uc ion, and he nex block
has as inpu a p e iously enhanced ep esen a ion o he
signal.
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 3 o 16
Fig. 1 Con olu ional neu al ne wo k (CNN). The con olu ional block
can ha e any dis ibu ion o con olu ional laye s and auxilia y laye s
such as he ba ch no maliza ion, o non-linea i ies
2.2 Loss unc ion
In [1], we p oposed an SE sys em based on he econ-
s uc ion o he LSA o a noisy signal: he audio sig-
nal was econs uc ed, by means o he o e lap-add
mechanism, using he enhanced loga i hmic ou pu spec-
um wi h he phase o he o iginal noisy speech. The
loss unc ion was he classical MSE be ween he LSA
o he e e ence and he LSA o he enhanced signal,
MSE yn,τ,ˆ
xn,τ=1
D
D−1

d=0yd,n,τ−ˆ
xd,n,τ2(1)
whe e Dis he signal inpu dimension, yd,n,τ,ˆ
xd,n,τa e he
equency bins o he loga i hmic spec um a he aining
example nand ame τ.yn,τis he a ge ec o o he clean
LSA e e ence, and ˆ
xn,τis he econs uc ed ec o o he
enhanced signal.
F om ou p e ious expe ience [1,11,12], ins ead o
using a ame-by- ame loss unc ion, his loss uses he
whole inpu as a sequence. Namely, he base loss unc ion
Fig. 2 Residual neu al ne wo k (ResNe ). The con olu ional block can
ha e any dis ibu ion o con olu ional laye s and auxilia y laye s such
as he ba ch no maliza ion, o non-linea i ies. The main di e ence
be ween CNN and ResNe is he esidual pa h in ResNe
is he MSE o he LSA o e all he examples and sequence
leng h o an upda e s ep,
JY,ˆ
X=1
N
N−1

n=0
1
T
T−1

τ=0
MSE yn,τ,ˆ
xn,τ(2)
whe e Yand ˆ
Xa e he LSA ep esen a ion o he aining
upda e.
Eachexampleisasequenceo all he ameso he
inpu signal, whe e Nis henumbe o examplesin he
aining p ocedu e s ep, and Tis he numbe o ames
o he example. In o de o simpli y he aining p oce-
du e, all he aining examples ha e he same numbe o
ames. The e o e, he aining keeps ixing he same seg-
men size, which is ob ained by andomly c opping he
inpu signals. This way, any example selec ed o a aining
upda e is an a bi a y segmen o he inpu example.
Finally, [1] implemen s he p og essi e pa adigm mod-
i ying he objec i e loss unc ion composing he MSE
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 4 o 16
be ween noisy inpu LSA and he enhanced LSA a di e -
en ne wo k le els o blocks. This p og essi e loss unc-
ion is a pa icula case o his pape p oposal, and i will
be s udied in de ail in he ollowing sec ion.
3 Speech enhancemen
This pape aims o s udy he unde lying po en ial o
he PSE pa adigm. P e ious wo ks ha e poin ed ou
he pe o mance imp o emen o he SE ask in p o-
g essi e a chi ec u e designs. Beyond hese esul s, his
pape b ings he hypo hesis ha he p og essi e pa adigm
ob ains be e SE pe o mance because hese mechanisms
also e esh g adien s du ing he neu al ne wo k ain-
ing. In he ollowing, we will desc ibe he PSE a chi-
ec u e p oposed in his pape , which is based on ou
p e ious wo k [1], bu addi ionally includes a se o no el-
ies/con ibu ions designed explici ly o his s udy.
3.1 A chi ec u e
This pape s udy will be based on wo DNN a chi ec u es:
p og essi e con olu ional neu al ne wo k (P-CNN) and
p og essi e esidual neu al ne wo k (P-ResNe ). Beyond
ou p e ious p oposal in [1] using he ResNe opology,
his pape includes he CNN opology wi h compa a-
i e pu poses and o ex end he s udy o gene alize he
p og essi e pa adigm o di e en a chi ec u es.
Figu e 3 ep esen s he on -end o bo h a chi ec u es.
The inpu signal, x( ), is i s windowed, and hen, we
ob ain he loga i hm o he absolu e alue o i s sho - e m
Fou ie ans o m (STFT), yielding he LSA X.Wealso
ob ain he Mel-scaled il e bank (FB), and Mel- equency
ceps al coe icien s (MFCC) wi h di e en windowing
p ocesses o p o ide addi ional in o ma ion o he ne -
wo k, XC.
Fig. 3 F on -end. P ep ocessing block a beginning o P-CNN and
P-ResNe a chi ec u es
Bo h a chi ec u es keep he same numbe o channels
along all hei con olu ional blocks. Also, hey use he
same basic con olu ional block (Fig. 4) o emain as com-
pa able as possible. This con olu ional block is composed
o wo successi e iden ical s uc u es. This s uc u e s a s
wi h ba ch no maliza ion, ollowed by a pa ame ic ec-
i ied linea uni (PReLU), and a 1D-Con olu ional laye
wi h hesamenumbe o channelsa heinpu and he
ou pu . In Fig. 4,Csis henumbe o channels.The
dimension o he ke nel (k) is 3 in all con olu ions o he
a chi ec u e. The ou pu o his s uc u e has he same
dimensions as he enhanced ou pu . Thus, we can ob ain
a pa ially enhanced signal a each block ou pu o P-CNN
and P-ResNe .
Fo his wo k, we used 1D-con olu ional laye s. Unlike
2D-con olu ional laye s ha combine empo al and e-
quency dimensions locally, 1D-con olu ional laye s pe -
o m a global combina ion o e all he equency dimen-
sions in a sho - e m empo al con ex . Recen wo ks sug-
ges ha when con olu ional a chi ec u es a e employed,
Fig. 4 Con olu ional block. Basic block o composing P-CNN and
P-ResNe a chi ec u es
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 5 o 16
he use o con olu ional laye s compu ed h ough he
single empo al dimension a e mo e app op ia e o he
speech enhancemen p ocessing [13,14].
The mul i esolu ion windowing p ocessing o he signal
con ibu es o he de e e be a ion ask, especially when
he impulse esponse is longe han he window leng h
used in he LSA analysis [15]. Xcou pu is only used as
inpu o he i s con olu ional block as shown in Figs. 1
and 2. The ollowing blocks ha e he same inpu and ou -
pu dimensions o allow he use o he p og essi e loss
unc ion. By p o iding he MFCC, he ne wo k has he
possibili y o calcula ing a e age ceps al ep esen a ions
o help wi h he channel iden i ica ion and imp o e he
de e e be a ion. The il e bank can also play a ole in he
iden i ica ion o use ul speech s uc u es in a pe cep ual
scale. As we can see in he expe imen s, hei combined
use makes a signi ican imp o emen .
3.2 Loss unc ion
In [1], we designed a neu al ne wo k o ha e he same
numbe o channels as he inpu signal a ce ain p obe
poin s. To induce he desi ed beha io , we o ced he
desi ed enhanced signal o be ob ained a hese poin s
by adding hei econs uc ion e o s o he aining loss,
which p o ided a p og essi e educ ion o he di e ence
be ween he e e ence signal and he econs uc ion a e
each block. Unlike he classical laye -wise aining, whe e
a s acking echnique is used, we ain he whole ne -
wo k agains he inal objec i e in he p oposed me hod
bu wi h he addi ional cons ain ha a ull econs uc-
ion a e each a chi ec u e block mus be ca ied ou .
Ou p e ious wo k demons a ed ha i we do no o ce
he econs uc ion a e each block, in e media e block
ou pu s a e en i ely di e en om ou objec i e and no
in e p e able. The inclusion o he econs uc ion con-
s ain h ough ou loss unc ion allows he isualiza ion
o he enhancemen p ocedu e. We can choose an in e -
media e esul o educe he e alua ion compu a ional
cos depending on he applica ion and help he aining
p ocedu e o ob ain be e esul s.
Wi h he p oposed loss unc ion, we add he ull econ-
s uc ion cons ain a e each con olu ional block min-
imizing he MSE be ween he clean e e ence Yand he
block ou pu ˆ
Xb(Fig. 5). Equa ion 3shows a gene al de -
ini ion o he p og essi e loss unc ion as a weigh ed sum
o e he econs uc ion loss o each con olu ional block
Fig. 5 PSE gene al a chi ec u e o P-CNN and P-ResNe . This igu e illus a es he applica ion o he p og essi e loss ha allows o di ec ly ep esen
he ou pu a e each block

Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 6 o 16
Table 1 T aining da ase s desc ip ion
Da ase Timi Lib ispeech TedLium
Files 6299 292329 56704
Speake s 630 2484 698
Speech ype Read speech Con e ence
In e ace Close mic ophone Audi o ium mic ophone
JPY,ˆ
X=
B

b=1
Wb·JY,ˆ
Xb.(3)
Depending on he weigh s in Equa ion 3, i is possible o
de ine di e en p og essi e loss unc ion c i e ia. In [1],
we p oposed he WP loss unc ion and he e we also p o-
pose he UP c i e ion. In he nex sec ions, bo h c i e ia
a e expe imen ally e alua ed in combina ion wi h P-CNN
and P-ResNe .
•Weigh ed p og essi e (WP): The main weigh o he
loss unc ion is he inal cos , as usual in
app oxima ion asks. Then, he cos o all he
a chi ec u e blocks is uni o mly dis ibu ed and
added in a weigh ed sum,
JWP Y,ˆ
XB=JY,ˆ
XB+α1
B
B

b=1
JY,ˆ
Xb(4)
whe e B is he numbe o blocks o he a chi ec u e.
No e ha Equa ion 4is a pa icula case o he
gene al p og essi e loss unc ion in Equa ion 3,
whe e Wb=α/B o b=1, ...,B−1and
WB=1+α/B. This loss unc ion implemen s
p og essi e p ocessing along blocks, i.e., e e y
in e media e block econs uc s he enhanced signal.
This design o ces he enhancemen p ocess o be
inc emen al, om sligh ly o de ailed cleaning. In he
end, his p ocessing complemen s he adi ional
p ocess o ob ain he inal sys em ou pu , namely he
s anda d back-p opaga ion o g adien s h oughou
he ull a chi ec u e (ou pu -inpu ).
•Uni o m p og essi e (UP): This loss unc ion
p oposes a uni o m dis ibu ion o he block losses
along he a chi ec u e,
JUP Y,ˆ
XB=1
B
B

b=1
JY,ˆ
Xb,(5)
which is a special case o Equa ion 3whe e
Wb=1/B o b=1, ...,B.
Wi h his s a egy, all he ou pu s ha e he same
impac in he econs uc ion. This way, e e y block
can equally con ibu e o he inal loss, and he ull
a chi ec u e makes he same e o in he signal
econs uc ion.
4 Expe imen al se up
4.1 T aining da a
Fo DNN aining, we ha e used h ee di e en public
da ase s: Tedlium [16] om Ted alks; Lib ispeech [17],
audio-books; and Timi [18], a phone ically balanced dis-
ibu ed ead speech. These da ase s a e ully employed,
wi hou any pa i ion. See Table 1 o he cha ac e is ics
o he da ase s.
4.2 Da a augmen a ion: e e be a ed and noisy aining
da a
Da a augmen a ion using e e be a ion and addi i e noise
was pe o med a he aining se . Fo each andom ain-
ing example, he e a e h ee ans o ma ions (See Table 2
o u he de ails):
1 Impulse esponses: We simula ed andom ooms and
sou ce- ecei e dis ances desc ibed h ough he
oom impulse esponses (RIR) using he py hon
package i -gene a o 1[19]. Fo he da a
augmen a ion loop, he e a e h ee di e en kinds o
simula ed ooms: small, medium, and la ge, selec ed
wi h a p obabili y o 0.5, 0.3, and 0.2.
2 Addi i e noise: We add some noise, wi h SNR
uni o mly sampled be ween 5 and 25 dB, om he
music and noise iles in he Musan da ase [20]. No e
ha among he noise iles, he e is c owd noise, bu
he e is no any in elligible speech.
3 Time scaling: We andomly selec a scale be ween 0.8
and 1.2. The e a e signals wi h no scaling, i.e., he
o iginal speed. Some o he s a e slowed down o sped
up.
4.3 E alua ion da a
Fo e alua ion pu poses, we use wo da abases: (1)
REVERB [21] and (2) VoiceHome 0.2 [22]and 1.0
[23]. REVERB is di ided in a de elopmen se (REVERB-
De ), gene ally used o e alua ing in e media e esul s
du ing he s udy, and an e alua ion se (REVERB-E al),
o con i ming he esul s and e alua ion o he sys em.
VoiceHome e alua es he sys em in a ealis ic domes ic
en i onmen wi h noise and e e be a ion. So, wi h hese
woda abases,wecansepa a e wocondi ions:
1h ps://gi hub.com/Ma in182/ i -gene a o
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 7 o 16
Table 2 RIR and noise o aining da a augmen a ion
Room impulse esponses
Small Medium La ge
P obabili y 0.5 0.3 0.2
Size (x,y,z)[m] x∼U(1, 6),y∼U(1, 6),z∼U(2, 3.5)x∼U(6, 10),y∼U(6, 10),z∼U(3, 5)x∼U(10, 20),y∼U(10, 20),z∼U(4, 6)
RT60[s] RT60 ∼U(0.1, 0.25)
Dis ance[m] 0.5, 1.0, 1.5, 2.0, 2.5
Mic ophone ype Bidi ec ional, hype ca diodid, ca dioid, subca doid, omnidi ec ional
Noise
Music 659 iles
Noise 929 iles
SNR [dB] SNR ∼U(5, 25)
Simula ed da a Pa o he REVERB da ase co esponds
o simula ed condi ions. They a e speech samples om
he WSJCAM0 co pus [24]combinedwi h h eekindso
RIR: small, medium, and big oom (RT60 =0.25, 0.5, 0.7s).
Fo each one, he e a e wo sou ce-mic dis ances: a (2m)
and nea (0.5m). Also, a s a iona y noise was added om
he same ooms (SNR =20dB). Fo his s udy, we only
use he i s channel o he eigh a ailable. We also add
i e noises (SNR =0, 5, 10, 15, 20, and 25dB) o all signals
a he simula ed condi ion o REVERB. These a e babble
noise, ca e en i onmen noise, music, s ee en i onmen
wi h lo o a ic, and noise cap u ed inside a mo ing
am.
Real da a We used wo e alua ion se s wi h eal condi-
ions: he eal pa in REVERB and VoiceHome da ase
( 0.2 and 1.0). REVERB was eco ded in a mee ing oom
wi h RT60 =0.7sa wo dis ances: a (2.5 m)and nea
(1m), omMC-WSJ-AV[25]. VoiceHome co esponds o
a ealis ic domes ic en i onmen wi h e e yday noises like
a acuum cleane , dish-washing, o sound o TV shows.
4.4 Speech quali y measu es
To measu e he le el o denoising and de e e be a ion
achie ed by he PSE me hod, we es ima e he segmen-
al SNR [26] and he speech- o- e e be a ion modula-
ion ene gy a io (SRMR) [27,28]. In hese me ics, he
highe he alues, he be e speech quali y. Howe e ,
i is well-known ha he SE p ocessing migh gene a e
dis o ion on he ou pu speech. The e o e, o he sim-
ula ed da ase , we also measu e he dis o ion be ween
he clean e e ence and enhanced speech using he log-
likelihood a io (LLR) [29]. In his case, lowe alues mean
less dis o ion, so he be e quali y o he speech. The
combina ion o bo h speech quali y iewpoin s, i.e., he
ade-o be ween noise/ e e be a ion educ ion and dis-
o ion, p o ides a gene al assessmen o he SE me hod
pe o mance. This way, he bes enhancemen sys em is
he one which imp o es SNR o SRMR, bu e ains he
dis o ion, in his case, measu ed wi h LLR, as low as pos-
sible. Addi ionally we use he well-known PESQ measu e
[30] o simula ed da a. PESQ measu e is in ange 0-5
whe e he highe he be e pe o mance.
4.5 Neu al ne wo k con igu a ion
The inpu p o ided o he CNN, ResNe , P-CNN, and
P-ResNe a chi ec u es consis s o he loga i hm o he
magni ude o he 512-STFT o he co up ed signal, sam-
pled a 16 kHz. The STFT is compu ed e e y 10 ms o a
25 ms sliding Hamming window. We also conca ena e he
Mel-Scaled Fil e -bank and he MFCC as auxilia y inpu s,
wi h il e bank sizes 32, 50, and 100, e e y 10 ms. MFCC
a e compu ed using he disc e e cosine ans o m (DCT)
wi hou unca ion. Howe e , each equency esolu ion
has a di e en sliding Hamming window o 25 ms, 50 ms,
and 75 ms espec i ely. These auxilia y ea u es p o ide
di e en equency and empo al esolu ions, which can
bene i he speech enhancemen p ocess [15]. Taking in o
accoun ha he LSA dimension is 512, he o e all inpu
size is 876.
Fo all he expe imen s, we use adap i e momen es i-
ma o (Adam) as he upda e unc ion. Each laye has 512
neu ons o ollow he philosophy o main aining unal-
e ed he numbe o channels along he a chi ec u e. The
aining consis s o 900 epochs. Fo each epoch, 10,000
inpu iles a e andomly selec ed om he aining se . As
long as he e a e unused aining examples, no ile can
be selec ed mo e han once. Ba ch no maliza ion mo ing
pa ame e s a e blocked a e epoch 700. Fo he JWP loss
unc ion, we use α=0.1 as in [1], which p o ided he bes
SRMR alue on REVERB-De .
5 P elimina y g adien s udy
This sec ion p esen s a p elimina y s udy o he beha -
io o he g adien o explo e how he injec ion o new
esh g adien s a di e en a chi ec u e le els imp o e
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 8 o 16
he aining p ocedu e. When g adien s back-p opaga e
h ough a la ge numbe o laye s, hey end o lose ene gy.
Thus, hei abili y o mo e weigh s o he laye s nea o
he inpu is educed. The p oposed PSE me hod eeds
a esh and s onge g adien a e each block o mo e
he weigh s o each laye . In o de o check his, we
design an expe imen o obse e he ene gy o he g a-
dien s ha modi y he weigh s o he i s con olu ional
block du ing he 100 i s op imiza ion upda es. This
p ocedu e is epea ed 100 imes wi h di e en weigh
ini ializa ions o obse e he a iance among di e en
s a s and he a ia ion o his g adien ene gy du ing
op imiza ion.
Figu e 6p esen s he esul s ob ained o P-CNN and P-
ResNe a chi ec u es, o non-p og essi e baselines, and
o each p oposed p og essi e c i e ia. The e is a no ice-
able di e ence in he beha io o he wo s uc u es. In P-
CNN, he e is a signi ican di e ence among he g adien
ene gy o each compa ed sys em. The lowe ene gy co e-
sponds o he baseline a chi ec u e, he one wi hou any
p og essi e assump ion. On he o he hand, he p og es-
si e mechanisms show a signi ican li ing o he g adien
ene gy. These boos ed g adien s ha e mo e s eng h o
mo e he weigh s allowing a be e lea ning a inne laye s
o he whole a chi ec u e.
In con as , in P-ResNe , he e is no ele an di e ence
be ween he g adien ene gy o he p og essi e echniques
and ha o he no p og essi e baseline a he i s con o-
lu ional block. Conside ha P-ResNe is an a chi ec u e
designed o deal wi h he anishing p oblem, and hanks
o esidual connec ions, he g adien s ha e a sho cu o
p opaga e up o he i s laye s wi hou anishing. In his
case, injec ing new g adien s does no push much mo e
he p e ious g adien s. Howe e , he new g adien s a e
mo e accu a e because hey di ec ly come om he a ge
e alua ion a he ou pu o each a chi ec u e block.
6 Resul s and discussion
6.1 Analysis o al e na i es o he DNN inpu
In his sec ion, we p esen a s udy o asses ha he com-
bined use o complemen a y inpu s o he co up ed LSA
may imp o e he pe o mance o he sys em. We use mul-
i esolu ion in he MFCC and FB inpu s as desc ibed
in Sec ion 4.5, bu we pe o m an abla ion s udy abou
he use o each ea u e ype. Fo his s udy, we ocus on
he de e e be a ion pe o mance o he P-ResNe wi h
Fig. 6 Mean and s anda d de ia ion (shaded a ea) o he log-ene gy o he g adien s o 100 andom ne wo k ini ializa ion du ing he 100 i s
mini-ba ches in aining. The log-ene gy is measu ed on he nea es block o he inpu
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 9 o 16
Table 3 E alua ion o he use o complemen a y in o ma ion a he inpu o he P-ResNe wi h WP a chi ec u e o e he REVERB-De
da ase in e ms o e e be a ion measu ed wi h SRMR
Complemen a y in o. Real condi ion Simula ed condi ion
Wi hou 6.55 7.99
FB 7.25 8.31
MFCC 7.07 8.44
FB + MFCC 7.14 8.41
Bold ex ema ks on he bes esul pe condi ion and i alic ex he second bes
WP o e he REVERB-De da ase in eal and simula ed
condi ions.
Table 3shows ha he bes esul s in simula ed condi-
ions a e a ained using only MFCC, bu o eal condi-
ions hey a e ob ained wi h FB ea u es. On a e age, he
combined use o bo h ea u es, FB and MFCC, p o ides
he bes pe o mance, especially compa ed o he use o
LSA wi hou any auxilia y inpu s.
6.2 A chi ec u e dep h analysis
SE p og essi e me hods use a sequence o s eps o pe -
o m he enhancemen . We ha e o de e mine he numbe
o s eps o he numbe o blocks ha composes he a chi-
ec u e. Table 4shows he a chi ec u e dep h s udy in
e ms o SRMR o e he REVERB-De da ase . This s udy
shows he esul s o simula ed and eal condi ions and
he a e age o bo h.
Resul indica e ha he con igu a ion wi h 16 blocks
achie es he bes pe o mance o all he e alua ed con-
di ions. No e how p og essi e sys ems can achie e high
SRMR, bo h o simula ed and eal condi ions. This con-
sis ency among di e en condi ions demons a es how
he p og essi e s a egy can p o ide a be e gene aliza-
ion o he DNN aining.
Fo CNN opology, he e e ence sys em in eal condi-
ions quickly deg ades he pe o mance wi h he dep h o
he a chi ec u e. Besides, esul s o P-CNN wi h UP a e
be e han he CNN e e ence sys em, i.e., P-CNN wi h
UP does no deg ade as as as CNN e e ence sys em as
dep h inc eases.
Fo ResNe opology, he a ailabili y o esidual con-
nec ions wo ks well wi h a high numbe o blocks.
Fo ins ance, he esul s o he ResNe e e ence sys-
em achie e he bes pe o mance on simula ed condi-
ions wi h he deepe a chi ec u e (32 blocks). Howe e ,
no e ha in eal condi ions, he ResNe e e ence sys-
em achie es he bes esul wi h 8 blocks e sus he 32
blocks o simula ed condi ions. Ne e heless, P-ResNe
Table 4 Speech quali y in e ms o SRMR o simula ed and eal e e be a ed speech samples h ough a chi ec u e dep h o
REVERB-De da ase . The las ows ep esen s he mean and s anda d de ia ion along he expe imen s p esen ed o each column
Re e ence sys ems P og essi e sys ems
Condi ion Blocks dep h CNN ResNe P-CNN wi h WP P-CNN wi h UP P-ResNe wi h WP P-ResNe wi h UP
Simula ed
8 7.33 8.23 6.49 7.53 8.31 7.91
16 7.60 8.27 8.96 7.70 8.41 8.05
24 8.87 8.14 6.18 8.09 8.03 8.02
32 7.01 8.56 7.65 7.41 7.98 7.78
Real
8 6.05 6.82 4.90 6.32 7.06 6.91
16 5.98 5.81 3.74 7.26 7.14 6.85
24 4.76 5.77 2.07 6.90 6.53 6.91
32 3.35 6.33 2.33 6.34 5.97 6.62
AVG5±STD
86.69±0.64 7.52±0.70 5.69±0.79 6.92±0.60 7.68±0.62 7.41±0.50
16 6.79±0.81 7.04±1.23 6.35±2.61 7.48±0.22 7.77±0.63 7.45±0.60
24 6.81±2.05 6.97±1.16 4.12±2.05 7.49±0.59 7.28±0.75 7.46±0.55
32 5.18±1.83 7.44±1.11 4.99±2.66 6.87±0.53 6.97±1.00 7.20±0.58
Bold alues show he bes esul o each condi ion
Llomba e al. EURASIP Jou nal on Audio, Speech, and Music P ocessing (2021) 2021:1 Page 16o 16
21. K. Kinoshi a, M. Delc oix, T. Yoshioka, T. Naka ani, E. Habe s, R.
Haeb-Umbach, V. Leu nan , A. Seh , W. Kelle mann, R. Maas, S. Ganno , B.
Raj, in P oceedings o he IEEE Wo kshop on Applica ions o Signal P ocessing
o Audio and Acous ics (WASPAA-13). The REVERB challenge: a common
e alua ion amewo k o de e e be a ion and ecogni ion o e e be an
speech (IEEE, New Pal z, 2013), pp. 1–4
22. N. Be in, E. Cambe lein, E. Vincen , R. Leba benchon, S. Peillon, É.
Lamandé, S. Si asanka an, F. Bimbo , I. Illina, A. Tom, e al,inIn e speech
2016. A F ench co pus o dis an -mic ophone speech p ocessing in eal
homes, (San F ancisco, 2016), pp. 2781–2785
23. N. Be in, E. Cambe lein, R. Leba benchon, E. Vincen , S. Si asanka an, I.
Illina, F. Bimbo . VoiceHome-2, an ex ended co pus o mul ichannel
speech p ocessing in eal homes, ol. 106, (2019), pp. 68–78
24. T. Robinson, J. F ansen, D. Pye, J. Foo e, S. Renals, in IEEE In e na ional
Con e ence on Acous ic, Speech and Signal P ocessing (ICASSP).WSJCAM0:a
B i ish English speech co pus o la ge ocabula y con inuous speech
ecogni ion (IEEE, De oi , 1995), pp. 81–84
25. M. Lincoln, I. McCowan, J. Vepa, H. K. Magan i, in P oceedings o he 2005
IEEE Wo kshop on Au oma ic Speech Recogni ion and Unde s anding
(ASRU-05). The mul i-channel Wall S ee Jou nal audio isual co pus
(MC-WSJ-AV): speci ica ion and ini ial expe imen s (IEEE, Philadelphia,
2005), pp. 357–362
26. C. Kim, R. M. S e n, in Nin h Annual Con e ence o he In e na ional Speech
Communica ion Associa ion (In e speech 2008). Robus signal- o-noise a io
es ima ion based on wa e o m ampli ude dis ibu ion analysis, (B isbane,
2008)
27. T. H. Falk, C. Zheng, W. Y. Chan, A non-in usi e quali y and in elligibili y
measu e o e e be an and de e e be a ed speech. IEEE T ans. Audio
Speech Lang. P ocess. 18(7), 1766–1774 (2010)
28. J. F. San os, M. Senoussaoui, T. H. Falk, in P oc. In . Wo kshop Acous . Signal
Enhancemen (IWAENC 2014). An upda ed objec i e in elligibili y
es ima ion me ic o no mal hea ing lis ene s unde noise and
e e be a ion, (An ibes - Jaun les Pins, 2014), pp. 55–59
29. P. C. Loizou, Speech quali y asssessmen . in: mul imedia analysis, p ocessing
and communica ions. (Sp inge , Be lin, 2011), pp. 623–654
30. A. W. Rix, J. G. Bee ends, M. P. Hollie , A. P. Heks a, in 2001 IEEE
In e na ional Con e ence on Acous ics, Speech, and Signal P ocessing.
P oceedings (Ca . No. 01CH37221), ol. 2. Pe cep ual e alua ion o speech
quali y (PESQ)-a new me hod o speech quali y assessmen o elephone
ne wo ks and codecs (IEEE, Sal Lake Ci y, 2001), pp. 749–752
31. L. D ude, J. Heymann, C. Boeddeke , R. Haeb-Umbach, NARA-WPE: a
Py hon package o weigh ed p edic ion e o de e e be a ion in Numpy and
Tenso low o online and o line p ocessing, (S u ga , 2018), pp. 1–5
32. T. Naka ani, T. Yoshioka, K. Kinoshi a, M. Miyoshi, B.-H. Juang. Speech
de e e be a ion based on a iance-no malized delayed linea p edic ion,
ol. 18, (2010), pp. 1717–1731
Publishe ’s No e
Sp inge Na u e emains neu al wi h ega d o ju isdic ional claims in
published maps and ins i u ional a ilia ions.