IFAC Pape sOnLine 58-9 (2024) 299–304
ScienceDi ec
ScienceDi ec
A ailable online a www.sciencedi ec .com
2405-8963 Copy igh © 2024 The Au ho s. This is an open access a icle unde he CC BY-NC-ND license
.
Pee e iew unde esponsibili y o In e na ional Fede a ion o Au oma ic Con ol.
10.1016/j.i acol.2024.07.413
10.1016/j.i acol.2024.07.413 2405-8963
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a ∗Ond ej Mih´alik ∗Mi osla Ji gl ∗
∗Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa oso a@ u .cz).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•
SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a ∗Ond ej Mih´alik ∗Mi osla Ji gl ∗
∗Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa oso a@ u .cz).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a ∗Ond ej Mih´alik ∗Mi osla Ji gl ∗
∗Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa oso a@ u .cz).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a ∗Ond ej Mih´alik ∗Mi osla Ji gl ∗
∗
Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa os[email p o ec ed]).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a
∗
Ond ej Mih´alik
∗
Mi osla Ji gl
∗
∗Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa oso a@ u .cz).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
CNN A chi ec u e o Pos u e
Classi ica ion on Small Da a
Michaela Mes´a oˇso ´a ∗Ond ej Mih´alik ∗Mi osla Ji gl ∗
∗Depa men o Con ol and Ins umen a ion, Facul y o Elec ical
Enginee ing, B no Uni e si y o Technology, B no, Czech epublic
(e-mail: Michaela.Mesa oso a@ u .cz).
Abs ac : A con olu ional neu al ne wo k is o en men ioned as one o he deep lea ning
me hods ha equi es a la ge amoun o aining da a. Ques ioning his belie , his pape
explo es he applicabili y o classi ica ion based on a shallow ne s uc u e ained on a small
da a se in he con ex o pa ien pos u e classi ica ion based on da a om a p essu e ma ess.
Designing a CNN o en p esen s a complex p oblem, especially wi hou a uni e sally applicable
app oach, allowing many di e se s uc u al possibili ies and aining se ings. We es ed a ious
aining op ions and laye con igu a ions o p o ide an o e iew o in luen ial pa ame e s
o pos u e classi ica ion. Expe imen s show encou aging esul s wi h he lea e-one-ou c oss-
alida ion accu acy o 93.1% o one o he e alua ed CNN s uc u es and i s hype pa ame e
se ings.
Keywo ds: CNN, ine uning, ne wo k s uc u e, op imiza ion, pos u e classi ica ion
1. INTRODUCTION
Deep lea ning me hods bene i g ea ly om e y la ge da a
se s. Howe e , in some si ua ions, i migh be di icul
o ga he enough da a, i he da a-acquisi ion p ocess is
ime-consuming o labo ious. Choosing he s uc u e and
pa ame e s o a CNN ha ep esen s images and hei
s o ed in o ma ion uniquely and co ec ly is one o he
mos impo an aspec s o achie ing a high CNN pe -
o mance. Typically he CNN con igu a ion pa ame e s
a e di ided in o wo ca ego ies: hose conce ning he a -
chi ec u e and hose conce ning he aining p ocess, see
Fe ey a-Rami ez e al. (2019). The speci ic ini ializa ion
o he pa ame e s o en has a signi ican impac on how
long i akes o he aining p ocess o ind a solu ion and
on he gene aliza ion abili y o he esul ing ained ne -
wo k. Reg e ully, as s a ed in Bishop and Bishop (2023),
he e is no much heo y o help de e mine an ini ializa ion
s a egy. Fo his eason, we o e a compac o e iew o a
selec ion o se ings and modi iable pa ame e s and hei
impac on he classi ica ion esul s.
2. NETWORK TRAINING OPTIONS
The igh design o he ne wo k is c ucial o he applica-
ion whe e he CNN is o be applied. Howe e he same im-
po ance lies in he ask o disco e ing app op ia e aining
pa ame e s ha lead o he bes -pe o ming classi ie . We
in oduce some o hese pa ame e s along wi h an o e iew
o hei ole in he aining p ocess.
2.1 Op imiza ion algo i hms
In MATLAB 2023, he e a e a ailable he ollowing ou
op imiza ion algo i hms:
•SGDM,
•RMSP op,
•Adam,
•L-BGFS
Al hough he e exis many mo e algo i hms, hey a e o en
de i a i es o hose lis ed abo e, hence we will lea e i o
he eade o e alua e hei pe o mance a e he mos
sui able base app oach was selec ed. Only he i s h ee
will be aken in o conside a ion in his pape .
The S ochas ic G adien Descen wi h Momen um o
SGDM is an ex ension o he S ochas ic g adien descen
(SGD) algo i hm ha upda es he ne wo k pa ame e s o
minimize he loss unc ion by aking small s eps a each
i e a ion in he di ec ion o he nega i e g adien o he
loss,
θℓ+1 =θℓ−α∇E(θℓ)+γ(θℓ−θℓ−1),(1)
whe e αis he lea ning a e,γis he momen um alue,
ℓis he i e a ion numbe , θis he pa ame e ec o , and
E(θ) is he loss unc ion. The g adien is e alua ed using
a subse o he aining da a, also called a mini-ba ch,
whe e a di e en subse is used a each i e a ion, o
de ailed explana ion see Bishop and Bishop (2023) and
Beale e al. (2023). The en i e pass o e he aining da a is
called an epoch. The con ibu ion o he p e ious g adien
s ep o he cu en i e a ion is de e mined by he lea ning
a e αand he momen um e m γ, which in addi ion
educes he oscilla ion a ound he op imum ha migh
occu in he case o a e y s eep nega i e g adien .
The unpublished Roo Mean Squa e P opaga ion o RM-
SP op is designed o accele a e he op imiza ion p ocess,
and hus, dec ease he numbe o unc ion e alua ions o
each he op imum. This is achie ed by using di e gen
lea ning a es o each weigh ha a e able o adap
au oma ically o he op imiza ion o he loss unc ion.
The lea ning a e o a pa icula weigh is di ided by a
Copy igh ©
2024 The Au ho s. This is an open access a icle unde he CC BY-NC-ND license
(
h ps://c ea i ecommons.o g/licenses/by-nc-nd/4.0/
)
300 Michaela Mesá ošo á e al. / IFAC Pape sOnLine 58-9 (2024) 299–304
unning a e age o he magni udes o ecen g adien s o
he co esponding weigh .
ℓ=β2 ℓ−1+ (1 −β2)[∇E(θℓ)]2(2)
θℓ+1 =θℓ−α∇E(θℓ)
√ ℓ+ϵ,(3)
whe e ℓis he mo ing a e age, β2is he squa ed g adien
decay ac o o he mo ing a e age, and ϵis a small
cons an o a oid di ision by ze o, o mo e de ail see
Tieleman and Hin on (2012). As s a ed by Kingma and Ba
(2015), RMSP op is sui ed o online and non-s a iona y
se ings.
Adap i e momen es ima ion, also called Adam, was i s
in oduced in Kingma and Ba (2015). I is an e icien
s ochas ic op imiza ion ha equi es i s -o de g adien s.
Indi idual adap i e lea ning a es a e again main ained o
all pa ame e s sepa a ely. Upda es a e di ec ly es ima ed
by using a unning a e age o he i s and he second
momen s o he g adien s. Mo ing a e ages a e calcula ed
as ollows:
mℓ=β1mℓ−1+ (1 −β1)∇E(θℓ) (4)
ℓ=β2 ℓ−1+ (1 −β2)[∇E(θℓ)]2(5)
mℓ=mℓ/(1 −βℓ
1) (6)
 ℓ= ℓ/(1 −βℓ
2) (7)
whe e β1and β2a e he exponen ial decay a es o
he momen es ima es. The ne wo k pa ame e s hen a e
upda ed as
θℓ+1 =θℓ−αmℓ
√ ℓ+ϵ(8)
I g adien s a e simila h oughou many i e a ions up-
da es o he weigh s a e able o gain momen um in a
ce ain di ec ion by using a mo ing a e age o he g adien .
The Adam algo i hm is sui able o wo king wi h la ge
amoun s o da a o i s small memo y equi emen s.
2.2 Lea ning a e
Lea ning a e, also e e ed o as s ep size, de e mines he
a e o change o he weigh s. The ini ial lea ning a e
has a signi ican impac on he lea ning speed and he
o e all aining de elopmen . When he lea ning a e is
oo la ge, he aining e o may inad e en ly inc ease
a he han dec ease, ye wi h a alue oo small, aining
is subs an ially slowe and may become s uck wi h a high
aining e o , o mo e de ail see Good ellow e al. (2016).
Fu he ema ks om Bengio (2012) and Reed and Ma ks
(1999) iden i y he lea ning a e as he mos impo an
hype pa ame e . I s ypical de aul alue is se o 0.01.
Howe e , addi ional uning is always ecommended.
2.3 Regula iza ion
A easonable app oach o p oblem-sol ing would be o
adap he model complexi y acco ding o he complexi y
o he p oblem. One way o con ol gene aliza ion and
a oid he o e i ing phenomenon is by egula iza ion.
To include egula iza ion in model aining, a penal y
unc ion is added o he loss unc ion E(θ), discou aging
he pa ame e s om ha ing la ge magni udes.

E(θ)=E(θ)+λΩ (9)
whe e Ω is he egula iza ion unc ion and λis he mul i-
plica i e pa ame e . An op imal ne wo k is one in which
a comp omise be ween he bes i o he aining da a and
a smoo hness o he i is ound, i.e. minimum o e all e o

E, see Zaknich (2003).
2.4 Momen um
Du ing aining, he p oblem o widely di e ing eigen al-
ues o en occu s. One echnique o deal wi h his incon-
enience is o add momen um γ o he weigh upda e
o mula. This limi s he oscilla ions and adds ine ia o
he mo emen h ough weigh space. As may be ound in
Bishop and Bishop (2023), he e ec o he momen um is
inc easing he e ec i e lea ning a e along he cu e in he
weigh space, whe e he g adien emains unchanged. On
he con a y, in high cu a u e egions whe e he g adien
changes signi ican ly, he con ibu ion o he momen um
is supp essed and has li le o no e ec on he e ec i e
lea ning a e alue.
Fig. 1. Di e ence be ween he g adien descen wi h
he momen um e m on he le side wi h as e con-
e gence owa ds he op imum in compa ison o g a-
dien descen wi hou he momen um e m oscilla ing
along he pa h on he igh side.
2.5 Mini-Ba ch
In i s p inciple, SGD uses only one da a poin o calcula e
g adien es ima ion o he e o unc ion leading o a e y
noisy es ima e, while he compu a ion on he en i e da a
se yields accu a e g adien esul s. Howe e , o de e mine
g adien es ima ion o each da a poin is compu a ionally
demanding, hence a mini-ba ch—a small subse o da a
poin s—is used o e alua e he g adien a each i e a ion.
Acco ding o Bishop and Bishop (2023), an impo an
ac o o bea in mind when using mini-ba ches is ha
cons i uen da a poin s should be chosen andomly om
he aining se . This is due o possible co ela ions be-
ween consecu i e da a poin s a ising om he way da a
was collec ed o s o ed, o example, i is so ed in an al-
phabe ical o ch onological o de .
3. NET STRUCTURE
A g ea di e si y o ne s uc u es can be obse ed ac oss
hei applica ions as a consequence o di e en classi ica-
ion p oblems ha ing da a se s ha di e in hei o ma ,
and hus, equi ing a co esponding ne wo k a chi ec u e.
In his pape we ocus on applying CNNs in an a ypical
se ing: designing a classi ie using a small da a se o 290
samples composed o a low esolu ion images, see Fig 2.
A e ca e ul conside a ion o he da ase ’s p ope ies, a
simple s uc u e wi h a conside ably smalle numbe o
Michaela Mesá ošo á e al. / IFAC Pape sOnLine 58-9 (2024) 299–304 301
unning a e age o he magni udes o ecen g adien s o
he co esponding weigh .
ℓ=β2 ℓ−1+ (1 −β2)[∇E(θℓ)]2(2)
θℓ+1 =θℓ−α∇E(θℓ)
√ ℓ+ϵ,(3)
whe e ℓis he mo ing a e age, β2is he squa ed g adien
decay ac o o he mo ing a e age, and ϵis a small
cons an o a oid di ision by ze o, o mo e de ail see
Tieleman and Hin on (2012). As s a ed by Kingma and Ba
(2015), RMSP op is sui ed o online and non-s a iona y
se ings.
Adap i e momen es ima ion, also called Adam, was i s
in oduced in Kingma and Ba (2015). I is an e icien
s ochas ic op imiza ion ha equi es i s -o de g adien s.
Indi idual adap i e lea ning a es a e again main ained o
all pa ame e s sepa a ely. Upda es a e di ec ly es ima ed
by using a unning a e age o he i s and he second
momen s o he g adien s. Mo ing a e ages a e calcula ed
as ollows:
mℓ=β1mℓ−1+ (1 −β1)∇E(θℓ) (4)
ℓ=β2 ℓ−1+ (1 −β2)[∇E(θℓ)]2(5)
mℓ=mℓ/(1 −βℓ
1) (6)
 ℓ= ℓ/(1 −βℓ
2) (7)
whe e β1and β2a e he exponen ial decay a es o
he momen es ima es. The ne wo k pa ame e s hen a e
upda ed as
θℓ+1 =θℓ−αmℓ
√ ℓ+ϵ(8)
I g adien s a e simila h oughou many i e a ions up-
da es o he weigh s a e able o gain momen um in a
ce ain di ec ion by using a mo ing a e age o he g adien .
The Adam algo i hm is sui able o wo king wi h la ge
amoun s o da a o i s small memo y equi emen s.
2.2 Lea ning a e
Lea ning a e, also e e ed o as s ep size, de e mines he
a e o change o he weigh s. The ini ial lea ning a e
has a signi ican impac on he lea ning speed and he
o e all aining de elopmen . When he lea ning a e is
oo la ge, he aining e o may inad e en ly inc ease
a he han dec ease, ye wi h a alue oo small, aining
is subs an ially slowe and may become s uck wi h a high
aining e o , o mo e de ail see Good ellow e al. (2016).
Fu he ema ks om Bengio (2012) and Reed and Ma ks
(1999) iden i y he lea ning a e as he mos impo an
hype pa ame e . I s ypical de aul alue is se o 0.01.
Howe e , addi ional uning is always ecommended.
2.3 Regula iza ion
A easonable app oach o p oblem-sol ing would be o
adap he model complexi y acco ding o he complexi y
o he p oblem. One way o con ol gene aliza ion and
a oid he o e i ing phenomenon is by egula iza ion.
To include egula iza ion in model aining, a penal y
unc ion is added o he loss unc ion E(θ), discou aging
he pa ame e s om ha ing la ge magni udes.

E(θ)=E(θ)+λΩ (9)
whe e Ω is he egula iza ion unc ion and λis he mul i-
plica i e pa ame e . An op imal ne wo k is one in which
a comp omise be ween he bes i o he aining da a and
a smoo hness o he i is ound, i.e. minimum o e all e o

E, see Zaknich (2003).
2.4 Momen um
Du ing aining, he p oblem o widely di e ing eigen al-
ues o en occu s. One echnique o deal wi h his incon-
enience is o add momen um γ o he weigh upda e
o mula. This limi s he oscilla ions and adds ine ia o
he mo emen h ough weigh space. As may be ound in
Bishop and Bishop (2023), he e ec o he momen um is
inc easing he e ec i e lea ning a e along he cu e in he
weigh space, whe e he g adien emains unchanged. On
he con a y, in high cu a u e egions whe e he g adien
changes signi ican ly, he con ibu ion o he momen um
is supp essed and has li le o no e ec on he e ec i e
lea ning a e alue.
Fig. 1. Di e ence be ween he g adien descen wi h
he momen um e m on he le side wi h as e con-
e gence owa ds he op imum in compa ison o g a-
dien descen wi hou he momen um e m oscilla ing
along he pa h on he igh side.
2.5 Mini-Ba ch
In i s p inciple, SGD uses only one da a poin o calcula e
g adien es ima ion o he e o unc ion leading o a e y
noisy es ima e, while he compu a ion on he en i e da a
se yields accu a e g adien esul s. Howe e , o de e mine
g adien es ima ion o each da a poin is compu a ionally
demanding, hence a mini-ba ch—a small subse o da a
poin s—is used o e alua e he g adien a each i e a ion.
Acco ding o Bishop and Bishop (2023), an impo an
ac o o bea in mind when using mini-ba ches is ha
cons i uen da a poin s should be chosen andomly om
he aining se . This is due o possible co ela ions be-
ween consecu i e da a poin s a ising om he way da a
was collec ed o s o ed, o example, i is so ed in an al-
phabe ical o ch onological o de .
3. NET STRUCTURE
A g ea di e si y o ne s uc u es can be obse ed ac oss
hei applica ions as a consequence o di e en classi ica-
ion p oblems ha ing da a se s ha di e in hei o ma ,
and hus, equi ing a co esponding ne wo k a chi ec u e.
In his pape we ocus on applying CNNs in an a ypical
se ing: designing a classi ie using a small da a se o 290
samples composed o a low esolu ion images, see Fig 2.
A e ca e ul conside a ion o he da ase ’s p ope ies, a
simple s uc u e wi h a conside ably smalle numbe o
lea ning pa ame e s was selec ed. This decision is based
on he gene ally obse ed ac , ha la ge CNNs end o
o e i he model wi h such a small amoun o a ailable
da a. One app oach o handle his issue s a ed by Kesha i
e al. (2018) is o educe he numbe o lea nable pa am-
e e s.
3.1 Con olu ional laye
One o he i s s uc u al p ope ies we conside a e
con olu ional laye s, numbe o ke nels and hei size.
Con olu ional laye ex ac s in o ma ion o ea u es om
he inpu signal using con olu ion il e s. Gi en he low
esolu ion o he images in he da a se , he commonly
used il e sizes sensible o his applica ion a e 3 ×3 and
5×5. Hence, we can desc ibe he con olu ional laye s o
he ne wo k as a uple (l1,l
2), whe e li×liis he size o
he con olu ion il e in i- h laye , o i∈{1,2}. These
pa ame e combina ions we e es ed:
(l1,l
2); l1,l
2∈{3,5}.(10)
3.2 Pooling laye
Pooling is esponsible o downsizing he spa ial size o
he ou pu om he p e ious laye by compu ing an
a e age alue in he il e window—a e age pool—o he
popula app oach o picking he maximal alue in he il e
egion—max pool —see in Zhou and Chellappa (1988).
This educes compu a ional complexi y and allows he
subsequen con olu ional laye o ex ac ea u es a a
di e en scale. Simila ly o con olu ional laye s we can
desc ibe ou selec ion o pooling laye s. I we ake in o
accoun he image esolu ion and he ac ha key ea u es
o he lying posi ion a e o en concen a ed on smalle
su aces, i will also be app op ia e o adjus he il e
dimensions acco dingly. Hence he con empla ed sizes we e
2×2and3×3 wi h wo possible s ep sizes 1 and 2. The
indi idual con igu a ions compa ed in he expe imen s can
be hen desc ibed by a pa ame e ec o (p1,s
1,p
2,s
2)
whe e he dimension o he i- h pooling laye is pi×piand
siis he s ep size o he i- h il e o i∈{1,2}. Tes ed
combina ions we e
(p1,s
1,p
2,s
2)∈({2,3}×{1,2})2.(11)
2 4 6 8 10
5
10
15
20
25
30
2 4 6 8 10
5
10
15
20
25
30
2 4 6 8 10
5
10
15
20
25
30
2 4 6 8 10
5
10
15
20
25
30
Fig. 2. Da a se example o one subjec in ou lying
posi ions.
The chosen pooling me hod is he max pool. 2The simple
easoning behind his choice is o maximize he p essu e
indica ions in key a eas such as shoulde s, hips, knees and
heels ha a e impo an o a success ul classi ica ion.
Using he a e age pool would blu hese a eas, causing
a dec eased di e en iabili y be ween he ea u es o classi-
ica ion classes.
3.3 Ac i a ion unc ion
The ac i a ion unc ion de ines he ou pu o a node o
a neu on o a gi en inpu . I can be hough o as an
e alua ion o whe he o no a neu on should be ac i a ed
upon he a i al o a speci ic inpu . Ne wo ks a e capable
o sol ing non i ial p oblems when nonlinea ac i a ion
unc ions a e included in he s uc u e, see Hinkelmann
(2018). Some o he adi ional and mode n ac i a ion
unc ions wi h hei applica ions a e he sigmoid unc ion,
hype bolic angen , bina y s ep in oduced in McCulloch
and Pi s (1943), Rec i ied Linea Uni (ReLU) desc ibed
in Nai and Hin on (2010), Leaky ReLU desc ibed in
Maas (2013), Exponen ial Linea Uni (ELU) desc ibed in
Cle e e al. (2016), o name a ew. Sigmoid, hype bolic
angen , ReLU and ELU ac i a ion unc ions we e used in
he p esen ed expe imen s.
4. DATA
Fo aining pu poses p essu e map images we e used.
These maps ep esen he alues measu ed by he p essu e-
sensi i e ma ess wi h a pe son si ua ed in lying posi ions,
see Fig. 2. The da a se consis s o 290 images o size
30 ×11. Each image has an associa ed subjec and class
numbe . All classes 1–4 illus a ed in Fig. 2 co espond o
hese posi ions:
•on he back (28.2%),
•on he igh side (23.7%),
•on he le side (21.7%),
•on he s omach (26.2%),
espec i ely. Numbe s ep esen he pe cen age ep esen-
a ion o a class in he da ase . Wi hin each o he posi-
ions, sligh a ia ions may occu such as di e en posi-
ioning o he a ms and legs o loca ion on he ma ess,
i.e., lying in he middle, on he edge o diagonally. The
o al numbe o measu ed subjec s is 18. Images a e s o ed
as ma ices ha con ain alues om 0 o 1, whe e 0
ep esen s he maximum p essu e and 1 is no p essu e on
he ma ess.
5. DESIGN AND PERFORMANCE
A chi ec u es we e ained using MATLAB 2023 Deep
Lea ning Toolbox. I is necessa y o s a e ha he esul s
p esen ed in his pape a e only op imal o pa ame e
space ea ma ked in Sec ions 3 and 5. The e may exis
ne s uc u es ha would achie e simila o be e esul s,
howe e , i is no possible o sea ch he whole pa ame e
space o ind he globally bes -pe o ming CNN. Ou
esul s a e quan i ied using he Accu acy measu e (Acc)
ob ained by c oss- alida ion wi h 18 olds (numbe o
subjec s).
5.1 S uc u e
Be o e we a e able o une he ne wo k’s pa ame e s, he
ne wo k s uc u e needs o be es ablished. The s uc u al
302 Michaela Mesá ošo á e al. / IFAC Pape sOnLine 58-9 (2024) 299–304
Fig. 3. Dependence o accu acy on he il e con igu a ion
and egula iza ion. Con olu ion il e sizes we e se o
(l1,l
2)=(3,3) wi h s ide 1.
s a ing poin was se o wo con olu ional laye s be ween
which he pooling laye s we e placed. Th ee o en il e s
we e es ed in each con olu ional laye o il e sizes s a ed
in (10). As o he pa ame e s, he only one esol ed a he
beginning is he op imiza ion algo i hm. A e conside a-
ion o algo i hm p ope ies, Adam was chosen as i com-
bines he ad an ages o he g adien descen op imiza ion
and RMSP op. The lea ning a e was se o 0.05 o all
aining scena ios and he ange o egula iza ion alues
is om 0.01 o 7 ·10−5. O he aining op ions emain a
de aul alues un il he inal s uc u e is chosen.
Figs. 3 and 4 depic wo examples o il e size combina-
ions. Fig. 3 shows ha dec easing egula iza ion c ea es
a a he uns able and luc ua ing su ace accompanied by
descending a e age Acc. Fo his eason, u he es s did
no include egula iza ion wi h alues 10−4and 7 ·10−5.
None o he combina ions shown in Fig. 4 we e able o
su pass he bes esul s o il e sizes 3 ×3, which is also
he case o he emaining il e size a ia ions. The e o e,
he inal s uc u e o he con olu ional laye s is six il e s
in he i s laye and 7 il e s in he second laye . Weigh s
in he il e s we e ini ialized om he no mal dis ibu ion
wi h ze o mean and s anda d de ia ion 0.01.
The nex pa ame e analysed was he ac i a ion unc-
ion. The expe imen included sigmoid, hype bolic an-
gen , leaky ReLU and ELU. A e e alua ion o a ious
combina ions, we concluded ha he ac i a ion unc ion
Fig. 4. Dependence o Acc on il e con igu a ion and
egula iza ion. Con olu ion il e sizes we e se o
(l1,l
2)=(5,3) wi h s ide 1.
should be consis en h oughou he ne wo k. The i s
ejec ed ac i a ion was sigmoid o i s poo esul s, only
a ound 25% Acc, which is compa able wi h andom guess-
ing. Mo eo e , i is compu a ionally demanding. Al hough
hype bolic angen is also a complex unc ion o compu e,
i yields he bes esul s closely ollowed by ReLU and ELU
ac i a ion unc ions wi h c oss- alida ion Acc ≈90%.
The inal examined s uc u al aspec s a e he pooling
laye s. Fo he easons men ioned in Sec ion 3 max pooling
me hod was applied in con igu a ion (11). Di e ences in
he classi ica ion accu acy o he di e en con igu a ions
a ied only sligh ly (±2%). Wi h smalle sliding s eps
he accu acy inc eased, howe e , lowe ing he s ep size
escala es he ime equi ed o aining. I s ide 1 is
used e en in one laye , he aining ime almos doubles.
The size o he used da ase implies ha such an inc ease
in he aining ime s ill does no p esen a p oblem, bu
in he case o di e en da ase s (numbe o measu emen s
o da a ype), i is a ac o ha needs o be aken
in o accoun . Con igu a ion (p1,s
1,p
2,s
2)=(2,1,2,2)
achie ed he highes accu acy.
A e he examina ion o he pa ial success a es inal
ne wo k s uc u e was selec ed:
•6 con olu ion il e 3 ×3,
•hype bolic angen ,
•max pool 2 ×2,
•7 con olu ion il e 3 ×3,
•hype bolic angen ,
•max pool 2 ×2,
• ully connec ed laye ,
•so max wi h 4 classes.
This ne wo k con ains only 1,700 lea nable pa ame e s,
which is a no ably smalle numbe in compa ison wi h
Michaela Mesá ošo á e al. / IFAC Pape sOnLine 58-9 (2024) 299–304 303
Fig. 3. Dependence o accu acy on he il e con igu a ion
and egula iza ion. Con olu ion il e sizes we e se o
(l1,l
2)=(3,3) wi h s ide 1.
s a ing poin was se o wo con olu ional laye s be ween
which he pooling laye s we e placed. Th ee o en il e s
we e es ed in each con olu ional laye o il e sizes s a ed
in (10). As o he pa ame e s, he only one esol ed a he
beginning is he op imiza ion algo i hm. A e conside a-
ion o algo i hm p ope ies, Adam was chosen as i com-
bines he ad an ages o he g adien descen op imiza ion
and RMSP op. The lea ning a e was se o 0.05 o all
aining scena ios and he ange o egula iza ion alues
is om 0.01 o 7 ·10−5. O he aining op ions emain a
de aul alues un il he inal s uc u e is chosen.
Figs. 3 and 4 depic wo examples o il e size combina-
ions. Fig. 3 shows ha dec easing egula iza ion c ea es
a a he uns able and luc ua ing su ace accompanied by
descending a e age Acc. Fo his eason, u he es s did
no include egula iza ion wi h alues 10−4and 7 ·10−5.
None o he combina ions shown in Fig. 4 we e able o
su pass he bes esul s o il e sizes 3 ×3, which is also
he case o he emaining il e size a ia ions. The e o e,
he inal s uc u e o he con olu ional laye s is six il e s
in he i s laye and 7 il e s in he second laye . Weigh s
in he il e s we e ini ialized om he no mal dis ibu ion
wi h ze o mean and s anda d de ia ion 0.01.
The nex pa ame e analysed was he ac i a ion unc-
ion. The expe imen included sigmoid, hype bolic an-
gen , leaky ReLU and ELU. A e e alua ion o a ious
combina ions, we concluded ha he ac i a ion unc ion
Fig. 4. Dependence o Acc on il e con igu a ion and
egula iza ion. Con olu ion il e sizes we e se o
(l1,l
2)=(5,3) wi h s ide 1.
should be consis en h oughou he ne wo k. The i s
ejec ed ac i a ion was sigmoid o i s poo esul s, only
a ound 25% Acc, which is compa able wi h andom guess-
ing. Mo eo e , i is compu a ionally demanding. Al hough
hype bolic angen is also a complex unc ion o compu e,
i yields he bes esul s closely ollowed by ReLU and ELU
ac i a ion unc ions wi h c oss- alida ion Acc ≈90%.
The inal examined s uc u al aspec s a e he pooling
laye s. Fo he easons men ioned in Sec ion 3 max pooling
me hod was applied in con igu a ion (11). Di e ences in
he classi ica ion accu acy o he di e en con igu a ions
a ied only sligh ly (±2%). Wi h smalle sliding s eps
he accu acy inc eased, howe e , lowe ing he s ep size
escala es he ime equi ed o aining. I s ide 1 is
used e en in one laye , he aining ime almos doubles.
The size o he used da ase implies ha such an inc ease
in he aining ime s ill does no p esen a p oblem, bu
in he case o di e en da ase s (numbe o measu emen s
o da a ype), i is a ac o ha needs o be aken
in o accoun . Con igu a ion (p1,s
1,p
2,s
2)=(2,1,2,2)
achie ed he highes accu acy.
A e he examina ion o he pa ial success a es inal
ne wo k s uc u e was selec ed:
•6 con olu ion il e 3 ×3,
•hype bolic angen ,
•max pool 2 ×2,
•7 con olu ion il e 3 ×3,
•hype bolic angen ,
•max pool 2 ×2,
• ully connec ed laye ,
•so max wi h 4 classes.
This ne wo k con ains only 1,700 lea nable pa ame e s,
which is a no ably smalle numbe in compa ison wi h
1
Inpu
6
con
+ anh
6
max
pool
6
7
con
+ anh
42
max
pool 42
ully
conn.
So
max Ou pu
Fig. 5. The esul ing CNN s uc u e. Bigge blocks ep esen ou pu om he p e ious laye wi h newly ob ained
dimensions while small boxes illus a e il e s wi h hei numbe o channels and unc ionali y.
Fig. 6. Success a es, po ayed by colo ba , based on he uned hype pa ame e s o mini-ba ch size 128.
usually used CNNs o image p ocessing. Fo isualiza ion
o he inal s uc u e see Fig. 5.
5.2 Fine uning
To a ain op imal esul s i is essen ial o pay a en ion o
ine uning. Fo a clea summa y o he uned pa ame e s
and hei ange e e o Table 1. Fo each pa ame e , he
de aul alue and i s su ounding in e al we e es ed. Fo
ins ance, in mini-ba ch size powe s o wo a e commonly
used o op imize he wo k wi h memo y, excep he las
alue, which is he whole size o a aining se . Had
he alue been se o 256, he aining se would ha e
been passed emp y, as he algo i hm au oma ically o ms
ba ches only i enough da a is a ailable. In Table 1, he
in luence o he pa ame e s on he Acc dec eases om le
o igh .
Table 1. O e iew o uned hype pa ame e s
Lea n a e Regula iza ion Epochs Mini-ba ch G adien decay
0.05 0.001 200 32 0.9
0.035 0.0002 700 64 0.8
0.02 0.0007 2000 128 0.7
0.01 4000 247
0.005

304 Michaela Mesá ošo á e al. / IFAC Pape sOnLine 58-9 (2024) 299–304
Fig. 6 shows ha he majo i y o esul s a e in he
ange o 75–85%. This is also he case o he mini-ba ch
o size 247. The bes -pe o ming con igu a ion eached
93.1% wi h a lea ning a e o 0.005, egula iza ion o
0.001, g adien decay o 0.8 and mini-ba ch size o 247
h oughou 4,000 epochs. The ou h plo in Fig. 6 is
o e all he one wi h he highes c oss- alida ion Acc in
he whole pa ame e space anging om nea ly 80–93%,
while he es epoch combina ions go as low as 35%.
Ano he pe cep ible phenomenon is he ac s a ed in
Sec ion 2 ha he lea ning a e has he la ges impac
on he classi ica ion esul s— he Acc alues a e mos ly
changing along he lea ning a e axis. The second mos
in luen ial pa ame e is egula iza ion and in combina ion
wi h he lea ning a e i is ob ious, ha hei mu ual
dec easing also wo sens he Acc. Hence i is impo an
o no e ha wi h a small lea ning a e and a he low
egula iza ion he Acc alue always signi ican ly d ops,
due o insu icien aining o excessi ely complex models
con aining high weigh alues. I we compa e he Acc ha
di e only in hei g adien decay we obse e ha hey
a e p ac ically he same, implying ha g adien decay has
a li le e ec on he esul ing Acc. Fo u u e expe imen s,
his pa ame e may be le ou un il mo e e ec ual se ings
a e es ablished. I may be es ed as one o he las uning
op ions.
The pe o mance o he p oposed ne wo k has been com-
pa ed o he esul s ob ained by ans e lea ning employ-
ing GoogLeNe , ResNe and Squeezene . In all cases, he
p oposed small-scale ne wo k pe o med be e achie ing
highe accu acy. This con i ms he conside a ion om Sec-
ion 3 ha a simple ne wo k is be e sui ed o his
ype o ask. Conside able di e ence also lies in he ime
needed o ain he ne wo k, whe e he shallow s uc u e
wi h 1,700 lea nable pa ame e s is ained much as e in
compa ison o ans e lea ning.
6. CONCLUSION
This pape discusses he possibili ies o aining a small
con olu ional neu al ne wo k while ha ing a ailable only a
small da a se . Al hough CNNs ypically need housands o
da a samples o ope a e eliably o o p o ide sa is ac o y
accu acy, we ha e been able o ind an app op ia e ne -
wo k s uc u e design and disco e co esponding op imal
pa ame e s h ough ex ensi e ine- uning expe imen s.
Lea ning a e and egula iza ion ha e shown he s onges
impac on he o e all accu acy. Numbe o epochs also
has a s ong e ec . Howe e , his a ec s only whe he
he model had enough ime o be ained o i he model
s a s o o e - i on gi en da a. Fo his eason, i is ap o
obse e he aining p ocess in he ea ly s ages o design.
Mini-ba ch has shown a mode a e in luence and g adien
descen had only a mino in luence on he inal accu acy.
The e o e, hese wo hype pa ame e s could be uned a
he end o he ne wo k design. Wi h he highes eached
accu acy being 93.1%, we ha e shown he applicabili y o
CNNs o his ype o classi ica ion p oblem.
As a pa o u u e wo k, he in luence o o he pa ame e s
such as lea ning a e scheduling o squa ed g adien decay
ac o could be examined. Addi ional expansion o he
pa ame e space such as he numbe o il e s used in
con olu ional laye s may also un eil ne wo k s uc u e
wi h imp o ed classi ica ion pe o mance.
ACKNOWLEDGEMENTS
The comple ion o his pape was made possible by he
g an No. FEKT-S-23-8451 – “Resea ch on ad anced
me hods and echnologies in cybe ne ics, obo ics, a i i-
cial in elligence, au oma ion and measu emen ” inancially
suppo ed by he In e nal science und o B no Uni e si y
o Technology.
REFERENCES
Beale, M.H., Hagan, M.T., and Demu h, H.B. (2023).
Deep lea ning oolbox e e ence.
Bengio, Y. (2012). P ac ical ecommenda-
ions o g adien -based aining o deep
a chi ec u es. CoRR, abs/1206.5533. URL
h p://a xi .o g/abs/1206.5533.
Bishop, C.M. and Bishop, H. (2023). Deep Lea ning:
Founda ions and Concep s. Sp inge , Cham, 1s ed.
edi ion.
Cle e , D.A., Un e hine , T., and Hoch ei e , S. (2016).
Fas and accu a e deep ne wo k lea ning by exponen ial
linea uni s (elus).
Fe ey a-Rami ez, A., A iles-C uz, C., Rod iguez-
Ma inez, E., Villegas-Co ez, J., and Zu˜niga-Lopez,
A. (2019). An imp o ed con olu ional neu al ne wo k
a chi ec u e o image classi ica ion. In Lec u e No es
in Compu e Science, olume 11524, 89–101. Sp inge
In e na ional Publishing, Cham.
Good ellow, I., Bengio, Y., and Cou ille, A. (2016). Deep
Lea ning. The MIT P ess.
Hinkelmann, K. (2018). Neu al ne wo ks, p. 7.
Kesha i, R., Va sa, M., Singh, R., and Noo e, A. (2018).
Lea ning s uc u e and s eng h o cnn il e s o small
sample size aining. In 2018 IEEE/CVF Con e ence on
Compu e Vision and Pa e n Recogni ion, 9349–9358.
doi:10.1109/CVPR.2018.00974.
Kingma, D.P. and Ba, J. (2015). Adam: A
me hod o s ochas ic op imiza ion. URL
h p://a xi .o g/abs/1412.6980.
Maas, A.L. (2013). Rec i ie nonlinea i ies imp o e neu al
ne wo k acous ic models.
McCulloch, W.S. and Pi s, W. (1943). A logical calculus
o he ideas immanen in ne ous ac i i y. Bulle in o
Ma hema ical Biology, 5, 115–133.
Nai , V. and Hin on, G.E. (2010). Rec i ied linea uni s
imp o e es ic ed bol zmann machines. In P oceedings
o he 27 h In e na ional Con e ence on Machine Lea n-
ing, 807–814.
Reed, R. and Ma ks, R.J. (1999). Neu al Smi hing:
Supe ised Lea ning in Feed o wa d A i icial
Neu al Ne wo ks. The MIT P ess. doi:
10.7551/mi p ess/4937.001.0001.
Tieleman, T. and Hin on, G. (2012). Lec u e 6.5- msp op,
cou se a: Neu al ne wo ks o machine lea ning. Uni-
e si y o To on o, Technical Repo .
Zaknich, A. (2003). Neu al ne wo ks o in elligen signal
p ocessing. Wo ld Scien i ic, New Je sey.
Zhou, Y.T. and Chellappa, R. (1988). Compu a ion
o op ical low using a neu al ne wo k. IEEE 1988
In e na ional Con e ence on Neu al Ne wo ks, 2, 71–78.
doi:10.1109/ICNN.1988.23914.