Ci a ion: Ga mendia-O begozo, A.;
Nuñez-Gonzalez, J.D.; An on, M.A.
SLRP op: A Back-P opaga ion
Va ian o Spa se Low Rank Me hod
o DNNs Reduc ion. Senso s 2023,
23, 2718. h ps://doi.o g/10.3390/
s23052718
Academic Edi o : Juan M. Co chado
Recei ed: 19 Janua y 2023
Re ised: 16 Feb ua y 2023
Accep ed: 28 Feb ua y 2023
Published: 2 Ma ch 2023
Copy igh : © 2023 by he au ho s.
Licensee MDPI, Basel, Swi ze land.
This a icle is an open access a icle
dis ibu ed unde he e ms and
condi ions o he C ea i e Commons
A ibu ion (CC BY) license (h ps://
c ea i ecommons.o g/licenses/by/
4.0/).
senso s
A icle
SLRP op: A Back-P opaga ion Va ian o Spa se Low Rank
Me hod o DNNs Reduc ion
Asie Ga mendia-O begozo 1,† , Jose Da id Nuñez-Gonzalez 1,*,† and Miguel Angel An on 2,†
1Depa men o Applied Ma hema ics, Uni e si y o he Basque Coun y UPV/EHU, 20600 Eiba , Spain
2TECNALIA, Basque Resea ch and Technology Alliance (BRTA), 20009 San Sebas ian, Spain
*Co espondence: [email p o ec ed]
† These au ho s con ibu ed equally o his wo k.
Abs ac :
Applica ion o deep neu al ne wo ks (DNN) in edge compu ing has eme ged as a con-
sequence o he need o eal ime and dis ibu ed esponse o di e en de ices in a la ge numbe
o scena ios. To his end, sh edding hese o iginal s uc u es is u gen due o he high numbe
o pa ame e s needed o ep esen hem. As a consequence, he mos ep esen a i e componen s
o di e en laye s a e kep in o de o main ain he ne wo k’s accu acy as close as possible o he
en i e ne wo k’s ones. To do so, wo di e en app oaches ha e been de eloped in his wo k. Fi s ,
he Spa se Low Rank Me hod (SLR) has been applied o wo di e en Fully Connec ed (FC) laye s
o wa ch hei e ec on he inal esponse, and he me hod has been applied o he la es o hese
laye s as a duplica e. On he con a y, SLRP op has been p oposed as a a ian case, whe e he
ele ances o he p e ious FC laye ’s componen s we e weighed as he sum o he p oduc s o each
o hese neu ons’ absolu e alues and he ele ances o he neu ons om he las FC laye ha a e
connec ed wi h he neu ons om he p e ious FC laye . Thus, he ela ionship o ele ances ac oss
laye was conside ed. Expe imen s ha e been ca ied ou in well-known a chi ec u es o conclude
whe he he ele ances h oughou laye s ha e less e ec on he inal esponse o he ne wo k han
he independen ele ances in a-laye .
Keywo ds: p uning; deep lea ning; edge compu ing
1. In oduc ion
The use o deep neu al ne wo ks (DNN) in di e en scena ios ela ed o Machine
Lea ning (ML) applica ions has de eloped in such a way ha cu en ly neu al ne wo k
designs ha e billions o pa ame e s wi h a g ea capabili y o p edic ion, as one o he
mos used ypes o a chi ec u e in p edic ion asks. Speci ically, some o hose applica ions
include image, sound, and ex ual da a ecogni ion. In con as o o he ML algo i hms,
he DNNs ha e achie ed a ema kable accu acy. Howe e , he use o hese ne wo ks in
memo y and p ocessing esou ce cons ained de ices is limi ed due o he amoun o da a
needed o de elop hese a chi ec u es and he high compu a ion cos s o aining hem.
Consequen ly, di e en educ ion echniques a e essen ial o i hese o me ne wo ks in
esou ce cons ained de ices, such as edge de ices.
Among o he s, he mos used and e ec i e way o sh ink hese ne wo ks is he use
o echniques such as p uning and quan iza ion. The o me one consis s o emo ing
pa ame e s (neu ons o weigh s) ha ha e negligible con ibu ion while main aining he
accu acy o he classi ie . On he o he hand, quan iza ion in ol es eplacing da a ypes
o educed wid h da a ypes, by ans o ming da a o i in o new da a ypes’ shapes. In
his way, educed ne wo ks a e able o compe e wi h he o iginal ones in e ms o accu acy
and e en imp o e hese in some cases in which o e i ing issues we e hinde ing hei
p edic abili y. Mo eo e , by educing he wid h o he da a, edge de ices could ace he
s o age issue men ioned abo e and collec la ge da ase s in cons ained memo y sizes.
Senso s 2023,23, 2718. h ps://doi.o g/10.3390/s23052718 h ps://www.mdpi.com/jou nal/senso s
Senso s 2023,23, 2718 2 o 14
Mainly con olu ional neu al ne wo ks (CNN) became a widely used ne wo k s uc u e
in image ecogni ion asks. Such a success is buil upon a la ge numbe o model pa ame e s
and con olu ional ope a ions. As a esul , he huge s o age and compu a ion cos s make
hese models di icul o be deployed on esou ce-cons ained de ices, such as phones and
obo s, needing o adop di e en educ ion echniques.
In his wo k, we in oduce a new a ian o he Spa se Low Rank (SLR) me hod o
de elop weigh p uning in well-known a chi ec u es, SLRP op. We judge ha he las
Fully Connec ed (FC) Laye , Final Response Laye (FRL), is he mos ele an o he inal
decision. Mo eo e , he ele ance o weigh s o his inal laye a e p opaga ed o he
p e ious laye s, making each neu on non-independen o he p e ious laye s in e ms o
ele ance. Consequen ly, he connec ions o each neu on has a di ec ela ionship wi h
neu on’s p edic abili y in he inal decision o he ne wo k, needing o conside hem. A e
ac o izing he weigh ma ices o FC Laye s, we spa si ied hem only conside ing he mos
ele an pa s and p opaga e hese ele ances o he p e ious FC laye s by conside ing
he connec ions be ween di e en FC laye s. Simila ly, we pe o med a pa allel p ocess in
which he spa si ica ion o ma ices has been ca ied ou independen ly be ween laye s,
only conside ing he ele ance in a-laye . Finally, we s a e he alidi y o he supposi ion o
backp opaga ing he ele ance wi hin laye s. As a esul , he p uning p ocess is op imized
by de e mining he less ele an componen s o each laye , as a consequence o he addi ion
o he backp opaga ion concep o he Spa se Low Rank Me hod con ibu ed in his wo k.
S a e o he A
The e ha e been se e al a emp s o educe DNNs dimensionali y by applying he
echniques men ioned abo e. P uning echniques consis o emo ing pa o connec ions
(weigh s) o neu ons om he o iginal ne wo k so as o educe he dimension o he o iginal
s uc u e by main aining i s abili y o p edic . The co e o hese echniques eside on he
edundancy ha some elemen s add o he en i e a chi ec u e. Memo y size and bandwid h
educ ion a e add essed wi h hese echniques. Redundancy is lowe ed and o e i ing is
aced in some scena ios. Di e en classi ica ions o wo ks based on his abili y a e made
depending on elemen p uned, s uc u ed/uns uc u ed (symme y), and s a ic/dynamic.
S a ic p uning is he p ocess o emo ing elemen s o a ne wo k s uc u e o line
be o e aining and in e ence p ocesses. Du ing hese las p ocesses no changes a e made
o he ne wo k p e iously modi ied. Howe e , emo al o di e en componen s o he
a chi ec u e equi es a ine- uning o e aining o he p uned ne wo k. This is a di ec
consequence o he changes ha su e he ne wo k by emo ing big pa o i s elemen s.
Thus, some compu a ion e o is needed in o de o each compa able accu acy o he
o iginal ne wo k.
The p uning has been ca ied ou by ollowing di e en c i e ia. In [
1
,
2
], hey used
he second de i a i e o he Hessian ma ix o educe he dimension o he o iginal a -
chi ec u e. Op imal B ain Damage (OBD) and Op imal B ain Su geon (OBS) wo k unde
h ee assump ions. Quad a ic: he cos unc ion is nea quad a ic. Ex emal: he p uning
is conduc ed a e he ne wo k con e ged. Diagonal: sums up he e o o indi idual
weigh s by p uning he esul o he e o caused by hei co-consequence. Addi ionally,
OBS a oids he diagonal assump ion and imp o es neu on emo al p ecision by up o 90%
educ ion in weigh s o XOR ne wo ks. Using Taylo expansions o i s o de [
3
,
4
] was
also an al e na i e o he p e ious ones o ackle ne wo ks’ dimension issues, as a c i e ion
o app oxima e he change o loss in he objec i e unc ion as an e ec o p uning.
Some wo ks we e based on he magni ude o he elemen s hemsel es. I is undoub -
edly ue ha nea -ze o alues o weigh s make a less con ibu ions o he esul s han
o he s ha su pass a ce ain h eshold alue. In his way, emo ing connec ions ha may
appea unnecessa y, he o iginal ne wo k is sh unk. The bes way o accomplish his is he
emo al o weigh s laye by laye o no ab up ly dec ease he pe o mance o he esul ing
ne wo k. LASSO [
5
] was in oduced as a penal y e m. I sh inks he leas absolu e alued
ea u e’s co esponding weigh s by inc easing weigh spa si y. This ope a ion has been
Senso s 2023,23, 2718 3 o 14
shown o o e a be e pe o mance han adi ional p ocedu es such as OBS by selec ing
he mos signi ican ly con ibu ed a iables ins ead o using all he a iables, achie ing
app oxima ely 60% mo e spa si y han OBS. The p oblem wi h LASSO is ha is an elemen-
wise p uning echnique leading o an uns uc u ed ne wo k and spa se weigh ma ices.
By pe o ming his echnique di iding he p ocess by g oups—as G oup LASSO [
6
] does,
emo ing en i e g oups o neu ons and main aining he o iginal ne wo k’s s uc u e— his
las issue was sol ed. G oups a e made based on geome y, compu a ional complexi y, o
g oup spa si y, among o he s.
Singula Value Decomposi ion (SVD) is an e ec i e and p omising echnique o sh ed
con olu ional o FC laye s by educing he numbe o pa ame e s needed o ep esen hem.
No only i has been use ul o image classi ica ion asks, bu also in objec de ec ion [
7
]
scena ios and o he s ela ed wi h DNN-based acous ic modeling [
8
,
9
]. Low- ank decom-
posi ion o con olu ion laye s as well as ully connec ed laye s we e applied in se e al
wo ks. Kholia chenko e al. [
10
] p oposed an i e a i e app oach o low- ank decomposi ion
by applying dynamic ank selec ion o image classi ica ion and objec de ec ion models.
One o i s nega i e aspec s was ha i e a i ely applying low- ank decomposi ion needs
longe ime and highe compu a ional esou ces o ank selec ion in deepe models. The
al e na i e p oposed by [
11
] assumes he p ope ies o bo h low- ank and spa seness o
weigh ma ices while aiming o econs uc he o iginal ma ix. In [
12
], h ough mixing he
concep s o spa si y and exis ence o unequal con ibu ions o neu ons owa ds achie ing
he a ge , he Spa se Low Rank (SLR) me hod is p oposed—a me hod ha sca e s SVD
ma ices o comp ess hem by conse ing lowe ank o unimpo an neu ons. As a esul ,
i is easible o educe he 3.6
×
s o age space o SVD wi hou much a iance on he model
accu acy. Speedup in he compu a ion was ano he ad an age ha has he s uc u ed
spa si y ob ained by he p esen ed app oach.
The majo i y o he p e ious wo ks had paid a en ion o he indi idual p uning o
laye s while no conside ing he connec ion be ween di e en laye s. In [
13
], hey claimed
ha he las FC laye is he mos ele an o he en i e ne wo k ega ding he e ec on
he inal esponse o he en i e ne wo k. Conside ing his las , hey p oposed o p une
he p e ious laye o he ne wo k when conside ing he connec ions o neu ons wi h
he neu ons o his las FC laye called he Final Response Laye (FRL). In his way, he
ele ances o he neu ons conside ed independen ly o he FRL we e backp opaga ed o
he p e ious laye ’s neu ons. The p uning o he es o he laye s was ca ied ou simila ly,
sco ing he ele ance o he neu ons when conside ing he connec ions wi h he pos e io
laye s’ neu ons.
O he al e na i es ha e been p oposed o ca y ou s a ic p uning. In [
14
], hey p o-
posed an inno a i e me hod o CNNs p uning called laye wise ele ance p opaga ion.
Each uni ’s ele ance o he inal decision is measu ed, and he uni s ha a e below a p ede-
ined h eshold a e emo ed om he o iginal s uc u e. As a las s ep, each componen ’s
ele ance is ecalcula ed by calcula ing he o al ele ance pe laye o keep i cons an
h ough he i e a ions. Thus, each uni ’s ele ance is ecalcula ed o main ain his alue.
In [
15
], a me hod o p uning edundan ea u es along wi h hei ela ed ea u e maps,
acco ding o hei ela i e cosine dis ances in he ea u e space, is p oposed, and he au ho s
achie e smalle ne wo ks wi h a signi ican download in pos - aining in e ence compu-
a ional cos s and achie ing a decen pe o mance. Redundancy can be minimized while
in e ence cos (FLOPS) is educed by 40% o VGG-16, 28%/39% o ResNe -56/110 models
ained on CIFAR-10, and 28% o ResNe -34 ained on ImageNe da abase wi h almos
negligible loss o accu acy. To ix he dec ease in accu acy a e p uning, models we e
e ained o some i e a ions main aining all hype -pa ame e s.
2. Ma e ial and Me hods
In his sec ion, we desc ibe he me hodology p oposed in o de o a emp o imp o e
he esul s ob ained in he li e a u e o di e en neu al ne wo ks and da ase s. Addi ionally,
we p esen he da ase s and models used o expe imen a ion.
Senso s 2023,23, 2718 4 o 14
2.1. Me hodology
The app oach we p esen in his s udy ollows his me hodology. Fi s , adi ional
low- ank decomposi ion SVD is applied o he weigh ma ix o he inal FC laye , called
FRL. Nex , inpu and ou pu weigh s in he laye a e selec ed o spa si ica ion using
di e en neu on selec ion s a egies. Then, spa si ica ion is applied o he selec ed inpu
and ou pu neu on componen s in he decomposed ma ices. Wi h he mos ele an
neu ons o he inal FC laye ob ained we back p opaga e hei ele ance o he p io FC
laye , ollowing he idea p oposed by [
13
], and we ob ained he ele ance o he neu ons
composing he p io FC laye . Finally, we epea ed he p ocess o spa si ica ion o he
decomposed ma ices o he p io FC laye . In pa allel, we pe o med he same p ocess
o spa si ica ion bu only conside ing he ele ance o each indi idual laye o he las
wo FC laye s. The esul s and compa a i e o bo h me hodologies a e summa ized in
Sec ion 4.
2.1.1. Single Value Decomposi ion (SVD)
One way o decomposing ma ices ep esen ing he weigh s o neu al ne wo ks is
he use o low- ank ac o iza ion. A con olu ional neu al ne wo k is composed o a la ge
numbe o con olu ional laye s and ully connec ed laye s. By applying his echnique
o con olu ional ke nels weigh s op imiza ion o he in e ence speed, he con olu ion
ope a ion could be ob ained due o he educ ion in he ime needed o mul iplica ion wi h
ac o ized ma ices compa ed o ha o mul iplica ion wi h 3D weigh s o ke nels.
In a FC laye ha ing m inpu and n ou pu neu ons, ac i a ion
a∈Rn
o he laye
wi h n nodes is ep esen ed as
a=g(WTX+b)(1)
whe e
X
ep esen s he inpu o he laye , and
g
() ep esen s any o he possible ac i a ion
unc ions. FC laye s connec ions o m a weigh ma ix
W∈Rmn
and a bias ec o
b∈Rn
whe e each pa ame e in he weigh ma ix
W
is
wij ∈R(
1
≤i≤m
, 1
≤j≤n)
, and bias
ma ix
b
is
bj∈R(
1
≤j≤n)
. The p oposed app oach is applied o he weigh ma ix
W
a e aining he en i e model. The SVD app oach decomposes he weigh ma ix
W
as
W=USVT
whe e
U∈Rm×m
,
VT∈Rn×n
a e o hogonal ma ices and
S∈Rm×n
is a
diagonal ma ix.
2.1.2. Spa se Low Rank Decomposi ion
The ma ix
S
is a diagonal ma ix con aining n non-nega i e singula alues in a
dec easing o de . The
k
singula alues ha a e mos signi ican a e kep by T unca ed
SVD whe e he decomposed ma ices
U
,
S
, and
VT
become
Û
,
ˆ
S
,
ˆ
VT∈Rm×k
,
Rk×k
,
Rk×n
.
By his way, he o iginal weigh s
W
a e eplaced in o econs uc ed app oxima ed weigh
ˆ
Was ˆ
W=Ûˆ
Sˆ
VT.
In SVD we ha e diagonal ma ix sigma
S
wi h he mos signi ican singula alues
om he uppe le o lowe igh in a dec easing o de . In he unca ion p ocess he i s k
ows o Uand columns o ˆ
VTa e kep .
Simula ing he app oach d i en by [
12
] we comp essed unca ed ma ices
Û
,
ˆ
S
, and
ˆ
V
based on he impo ance o he m inpu and n ou pu neu ons, i.e., we ep esen ed a
ew columns o
Û
and ows o
ˆ
VT
wi h a ank lowe han
k
, called educed ank
k
. In his
way, only
k
mos signi ican ows and columns a e kep in
Û
and
ˆ
VT
, espec i ely, due
o he o de o impo ance o
W
ha s a s om le o igh h ough columns o
Û
and
op o bo om h ough ows o
ˆ
VT
. We conside ed only he mos signi ican ows (
m
) and
columns (
n
) om each column and ow om
Û
and
ˆ
VT
, espec i ely, ollowing he cos
c i e ia, b ie ly explained in he nex subsec ion.
When he ma ices
Û
,
ˆ
S
and
ˆ
VT
a e spa si ied wi h s and , he o al numbe o
non-ze o pa ame e s o he
Û
,
ˆ
S
,
ˆ
VT
become
k(m− m +n− n +
1
) + k( m + n)
, which
is less han he numbe o non-ze o pa ame e s o unca ed SVD k(m+n+1).
Senso s 2023,23, 2718 5 o 14
P uning ully connec ed laye s is much mo e e ec i e in e ms o accu acy, ime, and
ene gy e iciency han p uning con olu ional laye s as shown in [
16
], which con ibu es o
bigge losses in p edic ion capabili y wi h he same a e o educ ion in pa ame e s. Those
a e usually placed in he i s posi ions in DNNs, and hey a e mo e sensi i e han he ones
ha a e placed in he las posi ions in many cases. In his s udy, we ollowed he app oach
di ec ed by [
12
] spa si ying SVD ma ices achie ing a low comp ession a e wi hou big
losses in accu acy. We used as a me ic o spa si ica ion he comp ession a e de ined in [
12
],
as he a io be ween he pa ame e s needed o de ine he spa si ied decomposed ma ices
and he o iginal weigh s’ ma ix pa ame e s. In ou case, we analyzed hei 3 a ian s o
applying SLR, ha we e based in cos , weigh s, and ac i a ions, and we p oposed wo new
a ian s ha sum he impo ance o cos and weigh s and cos and ac i a ions due o he
ac ha each o hem pe o med as he bes a ian in di e en comp ession a e egimes.
O e all, he mos ele an a ibu e was he cos , so we decided o es ablish his as he
c i e ia o selec ion o he ows and columns o spa si ica ion. An explana ion o he ull
p ocess o his me hod is gi en in Algo i hm 1.
Algo i hm 1 SLRP op
Weigh s1←FRL weigh s
Weigh s2←P e ious FC laye weigh s
U,S,V←SVD(Weigh s1)
U, S, V ←U[:, 0 : ank],S[0 : ank],V[0 : ank, :]
o N ows do
empU[ ow, k :] = 0
Weigh s ← empU ∗ S ∗ V
Sco e ←Accu acy(Weigh s )
is ←Ranking o ows
end o
o Ncolumns do
empV[ k :, column] = 0
Weigh s ← U ∗ S ∗ empV
Sco e ←Accu acy(Weigh s )
os ←Ranking o columns
end o
U( m ows)←0 whe e m=s *m
V( ncolumns)←0 whe e n=s *n
U2, S2, V2←SVD(Weigh s2)
U2, S2, V2←U2[:, 0 : ank],S2[0 : ank],V2[0 : ank, :]
o N ows do
Sco e ←∑Abs(U2[i,j]∗is[j])
is2←Ranking o ows
end o
o Ncolumns do
empV2[ :, column] = 0
Weigh s 2← U2∗ S2∗ empV2
Sco e ←Accu acy(Weigh s 2)
os2←Ranking o columns
end o
U2( m ows)←0
V2( mcolumns)←0
2.1.3. Selec ion o Rows and Columns Based on Cos
A neu on’s impo ance is de ined by whe he he e is a change o no in he ne wo k
pe o mance a e emo ing i . Le
c
be he de aul cos o he neu al ne wo k wi h o iginal
ained weigh
W
es ima ed o he p aining samples, compu ed using any loss unc ion.
Le
ˆ
c
be he alue o cos o he ne wo k wi h spa si ied weigh s
ˆ
W
. By unca ing wi h
Senso s 2023,23, 2718 6 o 14
educed ank
a speci ic ow o
ˆ
U
o column o
ˆ
VT
we ha e he absolu e change in cos is
o os. Those a e calcula ed as ollows:
isi=|c−ˆ
ci|(2)
osj=|c−ˆ
cj|(3)
As he spa si ica ion p ocess pu pose is o ensu e ha he unc ionali y o he ne wo k
does no change a e comp ession, and no o educe he o e all ne wo k cos o imp o e
accu acy, only he absolu e change in he cos alue is conside ed.
2.1.4. P opaga ion o Rele ance be ween Laye s
As i is known, he majo i y o neu al ne wo ks can be o mula ed as a nes ed unc ion.
Thus, we can de ine a ne wo k wi h
n
hidden laye s as a
F(n)= (n)◦ (n−1)◦
...
◦ (1)
.
Each laye can be ep esen ed as ollows:
(n)(x) = σ(n)(w(n)x+b(n))(4)
whe e
σ(n)
is he ac i a ion unc ion o each laye ,
w(n)
is he co esponding laye s connec-
ions’ weigh unc ion, and
b(n)
is he bias o each laye . A his s age i is possible o say
ha all o hese laye s a e in e connec ed and each o hem has di ec ele ance on he inal
decision o he en i e ne wo k. Consequen ly, weigh s om he FRL, ha is he las Fully
Connec ed Laye , backp opaga e hei ele ance o he p io laye s as p oposed in [
13
]. As
a esul , he ele ance o each neu on in he inal decision is he composi ion o weigh s
ha a e in e connec ed un il he FRL co esponding elemen ’s ele ance. The summa ion
o he co esponding ele ances is gi en by Equa ion (5).
sk=|w(k+1)|>|w(k+2)|>...|w(n)|>sn(5)
The absolu e alue o he weigh s ha a e connec ed o each o he neu ons o he FRL a e
mul iplied by he ele ance o hese in he FRL.
sk,j=∑
i
|w(k+1)
i,j|sk+1,i(6)
Equa ion
(6)
shows he ele ance o he j- h neu on in he
k
- h laye , which p opaga es he
ele ances o he neu ons om he pos e io k+1- h laye ha a e connec ed wi h i .
By in oducing his idea o he SVD ma ices, keeping only he mos ele an ows o
U ma ices, we can conside only he mos ele an neu ons o ha laye . The p ocedu e
in he FC laye s ha a e no he FRL, is simila o he o iginal SLR me hod excep o he
spa si ica ion o he U ma ices whe e he ele ance p opaga ed h ough he pos e io
laye s is conside ed o de e mine he mos ele an neu ons. This ele ance is p opaga ed
ollowing Equa ion (6).
In summa y, he main con ibu ions made by his wo k a e he ollowing. The p uning
o weigh s ca ied ou in hese FC laye s is mo e op imal han in he o iginal SLR me hod.
Consequen ly, he pe o mance o he esul ing ne wo k is aised, ob aining sub-op imal
esul s in e ms o di e en pe o mance me ics de ined in
Sec ion 4
wi h a less weigh s
needed compa ed wi h he o iginal s uc u e. Thus, in scena ios in which o iginal ne wo k
s uc u es canno i end use de ices due o memo y es ic ions a e c ucial o such
educ ion echniques.
2.2. Ma e ials
Rega ding he ma e ials, we used wo well-known models o image ecogni ion, VGG-
16 [
17
] and Lene 5 [
18
], whe e VGG a chi ec u e is much known o i s memo y in ensi e
FC laye s. I is wo h no ing ha VGG is he commonly used a chi ec u e wi h FC laye s
whe e o he popula image ecogni ion models, such as ResNe , Incep ion, MobileNe ,
Senso s 2023,23, 2718 7 o 14
ResNe , DenseNe , and objec de ec ion models, do no ha e FC laye s excep he inal
so max laye . Tables 1and 2show he speci ica ions o each ne wo k s uc u e.
These wo di e en app oaches we e es ed on di e en well-known da ase s, Ci a 10
(VGG16), Ci a 100 (VGG16), and MNIST (Lene 5). Each o hem con ain 32
×
32 images
(colo images in Ci a 10/Ci a 100 and g ayscale images in MNIST). In case o Ci a 10 and
MNIST he e a e 10 di e en classes and 100 in Ci a 100. All o hem ha e been ained
using de aul 10,000 es images and 50,000 and 60,000 aining images o he Ci a and
MNIST da ase s, espec i ely. Di e en comp ession a es we e applied o spa si ying
SVD ma ices; he e o e, o each da ase we ob ained di e en pe o mance me ics o
each me hod. O e all, we we e able o s a e which me hod was he bes in each case. The
da ase s used o expe imen s comp ise a good mix o di e en image ypes, sizes, and
numbe o classes. CIFAR-10 and CIFAR-100 ha e gene al pu pose image classes whe e
MNIST da ase con ains handw i en digi images.
Mo eo e , o demons a e he use ulness o ou app oach in senso ela ed da a we
es ed ou app oach in a model consis ing o 3 FC laye s o he Room Occupancy Es ima ion
Da a Se om he UCI Machine Lea ning Reposi o y. I is a da ase o es ima ing he p ecise
numbe o occupan s in a oom using mul iple non-in usi e en i onmen al senso s such as
empe a u e, ligh , sound,
CO2
, and PIR. The e a e 10,129 ins ances using 1000 o es ing
and he es o aining. Table 3shows he speci ica ions o he ne wo k s uc u e.
Table 1. VGG16 model ained o 32 ×32 images.
Laye Name Laye Type Fea u e Map Ou pu Size o Images Ke nel Size S ide Ac i a ion
Inpu Image 1 32 ×32 ×3 - - -
Con -1 2 ×Con 64 32 ×32 ×64 3 ×3 1 elu
Pool1 Maxpool 64 16 ×16 ×64 3 ×3 2 elu
Con -2 2 ×Con 128 16 ×16 ×128 3 ×3 1 elu
Pool2 Maxpool 128 8 ×8×128 3 ×3 2 elu
Con -3 2 ×Con 256 8 ×8×256 3 ×3 1 elu
Pool3 Maxpool 256 4 ×4×256 3 ×3 2 elu
Con -4 2 ×Con 512 4 ×4×512 3 ×3 1 elu
Pool4 Maxpool 512 2 ×2×512 3 ×3 2 elu
Con -5 2 ×Con 512 2 ×2×512 3 ×3 1 elu
Pool5 Maxpool 512 1 ×1×512 3 ×3 2 elu
Fla en Fla en - 512 - - elu
FC6 Dense - 4096 - - elu
FC7 Dense - 4096 - - elu
FC8 Dense - # o classes - - so max
Table 2. Lene 5 model ained o 32 ×32 images.
Laye Name Laye Type Fea u e Map Ou pu Size o Images Ke nel Size S ide Ac i a ion
Inpu Image 1 32 ×32 ×3 - - -
Con -1 1 ×Con 6 28 ×28 ×6 5 ×5 1 anh
Pool1 A gppool 6 14 ×14 ×6 2 ×2 2 anh
Con -2 1 ×Con 16 10 ×10 ×16 5 ×5 1 anh
Pool2 A gppool 16 5 ×5×16 2 ×2 2 anh
Fla en Fla en - 400 - - anh
FC3 Dense - 120 - - elu
FC4 Dense - 84 - - elu
FC5 Dense - # o classes - - so max
Table 3. FC laye s model.
Laye Name Laye Type Ou pu Size Ac i a ion
Inpu Da a # o a ibu es -
FC1 Dense 4000 elu
FC2 Dense 4000 elu
FC3 Dense 4000 elu
FC4 Dense # o classes sigmoid
Senso s 2023,23, 2718 8 o 14
The en i onmen in which all de elopmen o ou wo k had been p ocessed is a
×
64
Ubun u 20.04.4 LTS Ope a ing Sys em equipped wi h an In el Co e i7-11850H wo king a
2.5 GHz
×
16 and 32 GB DDR-4 RAM and a NVIDIA T1200 Lap op GPU (d i e e sion:
510.47.03, CUDA e sion:11.6).
3. P oposed App oach
As ci ed abo e, he in en ion o his esea ch was o ealize he connec ion o ele ances
be ween di e en laye s. To do so, we op ed o applying he app oach p esen ed by [
12
]
in wo di e en FC laye s. Fi s , we applied i independen ly. To show ha he e is a di ec
ela ionship be ween neu ons om di e en laye s, we conside ed he ele ance o he FRL
and backp opaga e i un il he second FC laye ha we p uned in he pa allel p ocess. In
his way, we could see he e ec o backp opaga ing he ele ance h oughou laye s and
see he co ela ion be ween hem.
We applied he SLR app oach p oposed by [
12
] o ob ain in o ma ion abou he mos
ele an pa s o ming he FRL. In his way, we we e able o know he ele ances o he
inal decision o each o he neu ons comp ising his las FC laye . To calcula e he ele ance
p opaga ed o he p e ious laye s we used he insigh in oduced in Sec ion 2.1.4 and
mul iplied each o he absolu e alue o weigh s ha was connec ed wi h each neu on o
he nex laye wi h he ele ance o hese neu ons om he nex laye , o each neu on
comp ising he laye in ques ion. Finally, a e ob aining he ele ances o each neu on
om he laye , we spa si ied he weigh ma ix o his laye he same way as o he FRL
bu while spa si ying he U ma ix in he ollowing way. We conside ed only he ows ha
ob ained he highes alue a e he summa ion o mul iplica ions o absolu e weigh s o
connec ions wi h each o he ele ances o neu ons connec ed om he nex laye , ins ead
o conside ing he o iginal ele ances o neu ons as we implemen ed o he FRL.
A he same ime, we ca ied ou spa si ica ion o he same numbe o laye s only
conside ing he independen ele ances o each laye , ollowing he c i e ia p oposed
by [
12
]. In his wo k, hey p esen h ee di e en c i e ia o de e mine which elemen s o
each laye we e mo e ele an o he inal decision o he ne wo k. O e all, he c i e ia
based in he cos o weigh s was he mos adequa e o educe he dimensionali y o he
p oblem and main ain he pe o mance o he a chi ec u e o be as high as possible. The
g aphical ep esen a ion o bo h app oaches is gi en by Figu e 1.
In case o VGG16, he FRL co esponds o FC7, and he backp opaga ion o he
ele ances has been ca ied ou un il FC6. FRL and p e ious FC laye o Lene 5 a e FC4
and FC3, espec i ely. In case o he a o emen ioned h ee FC laye s’ a chi ec u e hese
laye s a e FC3 and FC2, espec i ely.
Figu e 1. Compa ison o he p oposed app oaches.
Senso s 2023,23, 2718 9 o 14
4. Expe imen
In his sec ion, de ails abou he en i e expe imen a ion p ocess a e desc ibed. The
esul s ob ained a e summa ized as well.
Pe o mance Me ics
E alua ion me ics used o de e mining which o he me hods used is bes o keeping
he pe o mance o he o me ne wo k as high as possible a e he accu acy s. comp ession
a e, AUC s. comp ession, ecall s. comp ession, p ecision s. comp ession, and
speci ici y s. comp ession, whe e he comp ession a e was de ined in [
12
]. This las me ic
de e mines he ela ionship o he numbe o pa ame e s be ween spa si ied decomposed
ma ices and he o iginal ne wo k’s weigh ma ices. AUC is he a ea below he ROC
cu e—i.e., a g aph showing he pe o mance o a classi ica ion model a all classi ica ion
h esholds. Wha is plo ed in he cu e is he FPR and TPR in he x and y axes, espec i ely,
whose de ini ions a e gi en in Equa ion
(10)
and
(11)
. The de ini ions o he es o he
me ics men ioned abo e a e gi en in Equa ions
(7)
–
(9)
, whe e TP, TN, FP, and FP s and o
T ue Posi i es, T ue Nega i es, False Posi i es, and False Nega i es, espec i ely. We used
FRL’s p e ious FC laye ’s comp ession a e o check he accu acy o he esul an ne wo k
on di e en comp ession a e egimes.
Each o he a ian s p oposed in his wo k, conside ing o no he ele ance be ween
laye s, ha e been es ed on well-known open sou ce da ase s o image ecogni ion Ci a 10,
Ci a 100, and MNIST. All o hem ha e been ained using de aul 10,000 es images
and 50,000 and 60,000 aining images o he Ci a and MNIST da ase s, espec i ely.
To show hei e ec i eness in senso ela ed da ase s, hey we e applied o he Room
Occupancy De ec ion Da ase oo. In his case, 1,000 samples we e used o es ing and
he es (9,129 samples) o aining he ne wo k. In each case, we op ed o es ablishing
he same educ ion a e (0.5) and spa si y a e (0.5) de ined in [
12
], and we es ed each
a ian wi h di e en ank k, which de e mines he numbe o columns and ows kep in
he spa si ied
ˆ
U
and
ˆ
VT
ma ices. We inc emen ed he ank k un il he pe o mance me ics
we e equal o he ones ob ained by he o iginal ne wo k s uc u e. In he es ing phase 10
di e en seeds we e es ablished o es ing each me hodology in each da ase .
Accu acy(Acc) = TP +TN
TP +TN +FP +FN (7)
Recall(Re) = TP
TP +FN (8)
P ecision(P ) = TP
TP +FP (9)
T uePosi i eRa e(TPR) = TP
TP +FN (10)
FalsePosi i eRa e(FPR) = FP
FP +TN (11)
Speci ici y(Spec) = FP
FP +TN (12)
5. Resul s
Figu e 2shows he accu acies ob ained a e es ing bo h p uning echniques o he
VGG16 a chi ec u e on he Ci a 10 da ase . As is clea , he e was no signi ican di e ence
be ween he me hods when applying an ex emely low comp ession a e, which means ha
e y ew pa ame e s o he o iginal ma ices we e kep . Simila ly, we could obse e he
same pa e n when a highe numbe o pa ame e s we e kep in he o iginal decomposed
ma ices, bu he e we e signi ican di e ences be ween bo h comp ession a e egimes. In
his case, applying he SLR me hod independen ly o di e en FC laye s o e s a highe
