A COMPARATIVE STUDY OF ACTIVATION FUNCTIONS IN DEEP LEARNING MODELS

80 “Al-Fa g‘oniy a lodla i” elek on ilmiy ju nali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendan s o Al-Fa ghani" elec onic scien i ic jou nal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 yea Электронный научный журнал "Потомки Аль- Фаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год h ps://al- a goniy.uz/ A COMPARATIVE STUDY OF ACTIVATION FUNCTIONS IN DEEP LEARNING MODELS P imbe o Abbaz, Phd s uden , Tashken Uni e si y o In o ma ion Technologies named a e Muhammad Al-Khwa izmi. Senio lec u e , Uni e si y o Tashken o applied sciences [email p o ec ed] Akba o Na uz, Phd s uden , Tashken Uni e si y o In o ma ion Technologies named a e Muhammad Al-Khwa izmi akba o na [email protected] Abs ac : Ac i a ion unc ions play a i al ole in he aining dynamics and gene aliza ion pe o mance o deep lea ning models. This s udy p esen s a compa a i e analysis o en widely used ac i a ion unc ions—ReLU, Sigmoid, Tanh, ELU, SELU, So plus, So sign, Swish, GELU, and a cus om spline-based unc ion—wi hin a uni ied con olu ional neu al ne wo k (CNN) a chi ec u e. All models we e ained and e alua ed on he CIFAR-10 da ase unde iden ical expe imen al se ings, including ixed lea ning a e, ba ch size, numbe o epochs, and a chi ec u e con igu a ion. The esul s show ha he p oposed spline ac i a ion unc ion achie ed he highes es accu acy o 71.48%, ou pe o ming popula unc ions such as ReLU (67.87%) and Swish (68.33%). In con as , adi ional unc ions like Sigmoid exhibi ed signi ican ly lowe accu acy (10.00%), ea i ming known limi a ions in deep ne wo k con ex s. These indings demons a e he po en ial o adap i e, piecewise-de ined ac i a ion unc ions o enhance model pe o mance while main aining compe i i e aining e iciency. The s udy p o ides p ac ical insigh s in o ac i a ion unc ion selec ion o image classi ica ion asks. Keywo ds: Ac i a ion Func ions, Spline Ac i a ion, Con olu ional Neu al Ne wo ks, CIFAR-10, Image Classi ica ion, ReLU, Swish, Deep Lea ning 1. Ind oduc ion Ac i a ion unc ions (AFs) a e essen ial componen s o deep lea ning a chi ec u es, in oducing non-linea i y ha enables neu al ne wo ks o app oxima e complex mappings be ween inpu s and ou pu s. Wi hou AFs, neu al ne wo ks would be educed o simple linea models, ega dless o hei dep h o s uc u e [1]. The choice o ac i a ion unc ion signi ican ly a ec s a model’s aining s abili y, con e gence speed, and gene aliza ion capaci y [2]. O e he yea s, nume ous ac i a ion unc ions ha e been p oposed, anging om adi ional unc ions like Sigmoid and Tanh o mo e ad anced o ms such as ReLU, ELU, Swish, and GELU [3]. Each unc ion exhibi s dis inc ma hema ical p ope ies, including smoo hness, mono onici y, boundedness, and di e en iabili y, which impac how well he ne wo k lea ns om da a [4]. ReLU and i s a ian s, o example, ha e become he de ac o s anda d o many con olu ional neu al ne wo ks (CNNs) due o hei simplici y and compu a ional e iciency [5]. Howe e , hese unc ions a e no wi hou limi a ions—ReLU su e s om dying neu on p oblems, while Sigmoid and Tanh can cause anishing g adien s in deepe ne wo ks [6]. In esponse o such limi a ions, adap i e and lea nable ac i a ion unc ions ha e eme ged. Among hem, spline-based ac i a ion unc ions p o ide a lexible, da a-d i en al e na i e by modeling piecewise con inuous ans o ma ions wi h ainable pa ame e s [7]. Recen s udies ha e shown ha spline ac i a ions can imp o e model exp essi eness and lea ning s abili y wi hou adding subs an ial compu a ional complexi y [8]. This pape p esen s a compa a i e empi ical s udy o en ac i a ion unc ions, including a cus om 81 “Al-Fa g‘oniy a lodla i” elek on ilmiy ju nali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendan s o Al-Fa ghani" elec onic scien i ic jou nal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 yea Электронный научный журнал "Потомки Аль- Фаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год h ps://al- a goniy.uz/ spline-based unc ion, using a consis en CNN a chi ec u e ained on he CIFAR-10 da ase . All hype pa ame e s, p ep ocessing s eps, and ne wo k s uc u es we e kep iden ical o ensu e a ai e alua ion. Ou goal is o quan i y and analyze he pe o mance o each unc ion in e ms o classi ica ion accu acy, aining ime, and loss. The indings o e p ac ical insigh s in o ac i a ion unc ion selec ion o image classi ica ion asks, pa icula ly in esou ce- cons ained o pe o mance-sensi i e applica ions. 2. Rela ed Wo k Ac i a ion unc ions (AFs) ha e been ex ensi ely s udied due o hei c ucial ole in enabling deep neu al ne wo ks o lea n non-linea mappings. T adi ional ac i a ion unc ions such as Sigmoid and Tanh we e widely used in ea ly neu al ne wo k a chi ec u es bu su e ed om anishing g adien p oblems, which hinde ed he aining o deep models [1]. This led o he eme gence o ReLU (Rec i ied Linea Uni ), which became he de aul choice in con olu ional neu al ne wo ks due o i s simplici y, spa si y, and e icien g adien low [2]. To o e come ReLU’s limi a ions such as he “dying ReLU” p oblem, se e al a ian s we e p oposed including Leaky ReLU, Pa ame ic ReLU (PReLU), and Exponen ial Linea Uni (ELU). These unc ions aim o main ain non-ze o g adien s in he nega i e inpu space while p ese ing he bene i s o ReLU [3]. In a comp ehensi e analysis, Bou aya e al. [4] ca ego ized ac i a ion unc ions in o i e majo ypes: bounded (e.g., Sigmoid, Tanh), unbounded (ReLU, ELU), exponen ial-based, adap i e, and di e si ied unc ions. Thei axonomy ocused on key ma hema ical cha ac e is ics such as mono onici y, smoo hness, and boundedness. Simila ly, Dubey e al. [5] benchma ked 18 di e en ac i a ion unc ions ac oss se e al deep lea ning asks, concluding ha no single unc ion consis en ly ou pe o ms o he s in all scena ios, ein o cing he impo ance o ask-speci ic e alua ion. Recen ly, adap i e and lea nable ac i a ion unc ions ha e gained ac ion. Among hese, Spline- based unc ions ha e shown p omise by o e ing piecewise-smoo h, pa ame e ized ans o ma ions ha can be lea ned du ing aining. Sca dapane e al. [6] and Boh a e al. [7] demons a ed ha spline-based ac i a ions ou pe o m adi ional unc ions in se e al classi ica ion benchma ks by imp o ing exp essi eness and s abili y, pa icula ly in deepe models. Howe e , despi e ex ensi e heo e ical wo k, ew s udies ha e empi ically compa ed adi ional and spline-based ac i a ion unc ions unde iden ical expe imen al condi ions. Mos p io benchma ks in ol e ei he shallow ne wo ks o a ying a chi ec u es, which complica es di ec compa ison. This s udy ills ha gap by sys ema ically e alua ing en ac i a ion unc ions—including a cus om spline— on he CIFAR-10 da ase using an iden ical CNN a chi ec u e and aining se up. 3. Me hodology To ensu e a ai and con olled compa ison o di e en ac i a ion unc ions, we designed a consis en expe imen al amewo k based on a con olu ional neu al ne wo k (CNN) ained on he CIFAR-10 image classi ica ion da ase . This sec ion ou lines he da ase cha ac e is ics, model a chi ec u e, ac i a ion unc ions es ed, and he aining con igu a ion. 3.1 Da ase The CIFAR-10 da ase is a widely used benchma k o e alua ing image classi ica ion models. I con ains 60,000 colo images o size 32×32 pixels, di ided in o 10 ca ego ies such as ai planes, au omobiles, bi ds, ca s, and mo e. The da ase is spli in o 50,000 aining images and 10,000 es images. All images we e no malized o he [0,1] ange by di iding pixel alues by 255. Label ec o s we e one-ho encoded o be compa ible wi h he ca ego ical c oss-en opy loss unc ion. 3.2 Model A chi ec u e A shallow ye exp essi e CNN a chi ec u e was adop ed o educe he isk o o e i ing while ensu ing su icien capaci y o unc ion compa ison. The a chi ec u e is as ollows: Inpu : 32×32×3 image Con 2D laye wi h 32 il e s (3×3 ke nel), ollowed by ac i a ion 82 “Al-Fa g‘oniy a lodla i” elek on ilmiy ju nali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendan s o Al-Fa ghani" elec onic scien i ic jou nal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 yea Электронный научный журнал "Потомки Аль- Фаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год h ps://al- a goniy.uz/ Con 2D laye wi h 64 il e s (3×3 ke nel), ollowed by ac i a ion MaxPooling2D (2×2) D opou ( a e = 0.25) Fla en Dense laye wi h 128 uni s, ollowed by ac i a ion D opou ( a e = 0.5) Ou pu : Dense laye wi h 10 uni s and so max ac i a ion d aw o his model 3.3 T aining Con igu a ion To ensu e ai ness and ep oducibili y in he compa ison o ac i a ion unc ions, all models we e ained unde iden ical hype pa ame e se ings. Each ne wo k was ained o 5 epochs using a ba ch size o 64. The Adam op imize was employed wi h a ixed lea ning a e o 0.001, and he loss unc ion used was ca ego ical c ossen opy, app op ia e o mul i-class classi ica ion asks. A 10% alida ion spli was applied o he aining se du ing aining o moni o he model's gene aliza ion pe o mance. The pe o mance o each ac i a ion unc ion was e alua ed based on h ee key me ics: es accu acy, es loss, and aining ime, measu ed in seconds. All aining sessions we e conduc ed in a single GPU-enabled en i onmen o main ain consis en ha dwa e condi ions and elimina e a iabili y due o p ocessing esou ces. 3.4 Ac i a ion Func ion Fo mula ions The ma hema ical de ini ions o he en ac i a ion unc ions compa ed in his s udy a e summa ized below, along wi h hei p ope ies and e e ences. 1. ReLU (Rec i ied Linea Uni ) This is one o he mos widely used ac i a ion unc ions in con olu ional neu al ne wo ks due o i s simplici y and compu a ional e iciency. I ou pu s he inpu di ec ly i i is posi i e; o he wise, i e u ns ze o. 𝑓(𝑥)=𝑚𝑎𝑥(0,𝑥) I enables spa se ep esen a ions bu su e s om he “dying ReLU” p oblem, whe e neu ons can become inac i e du ing aining and ne e eco e . 2. Sigmoid The sigmoid unc ion maps any eal- alued inpu o he ange (0, 1), making i sui able o p obabilis ic in e p e a ion. 𝑓(𝑥)=1 1+𝑒−𝑥 Howe e , i ends o sa u a e o la ge alues o |x| and causes anishing g adien s, which slows down lea ning in deep ne wo ks. 3. Tanh (Hype bolic Tangen ) Simila o sigmoid bu ze o-cen e ed, he anh unc ion maps inpu s o (−1, 1). 𝑓(𝑥)= anh(𝑥)=𝑒𝑥−𝑒−𝑥 𝑒𝑥+𝑒−𝑥 I has be e g adien dynamics han sigmoid bu s ill su e s om anishing g adien s a ex eme alues. 4. ELU (Exponen ial Linea Uni ) This unc ion ou pu s he iden i y o posi i e alues and an exponen ial cu e o nega i es, helping main ain ac i a ions close o ze o mean. 𝑓(𝑥)={𝑥, 𝑥≥0 𝛼(𝑒𝑥−1), 𝑥<0 (𝑐𝑜𝑚𝑚𝑜𝑛𝑙𝑦 𝛼=1) I helps mi iga e anishing g adien p oblems and accele a es lea ning. 5. SELU (Scaled ELU) A scaled a ian o ELU designed o sel - no malizing neu al ne wo ks. 𝑓(𝑥)=𝜆∙{𝑥, 𝑥≥0 𝛼(𝑒𝑥−1), 𝑥<0 (𝜆≈1.05, 𝛼≈1.67) Wi h app op ia e λ and α, i main ains ze o mean and uni a iance h oughou he ne wo k, aiding con e gence. 6. So plus So plus is a smoo h app oxima ion o ReLU. 𝑓(𝑥)=𝑙𝑛(1+𝑒𝑥 ) I is always posi i e and di e en iable, bu compu a ionally mo e expensi e han ReLU. 7. So sign A bounded, con inuous unc ion simila o anh bu wi h a simple exp ession. 𝑓(𝑥) = 𝑥 1 + |𝑥| I s g adien decays mo e slowly han anh, which may bene i lea ning. 8. Swish 83 “Al-Fa g‘oniy a lodla i” elek on ilmiy ju nali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendan s o Al-Fa ghani" elec onic scien i ic jou nal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 yea Электронный научный журнал "Потомки Аль- Фаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год h ps://al- a goniy.uz/ Swish is a smoo h, non-mono onic unc ion de ined as he inpu mul iplied by i s sigmoid. 𝑓(𝑥) = 𝑥 · 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝑥) = 1 1+𝑒−𝑥 I has been shown o ou pe o m ReLU in deepe ne wo ks due o i s sel -ga ing beha io . 9. GELU (Gaussian E o Linea Uni ) This unc ion weigh s he inpu by he cumula i e dis ibu ion unc ion o he s anda d no mal dis ibu ion. 𝑓(𝑥)=𝑥⋅𝛷(𝑥) 𝑤ℎ𝑒𝑟𝑒 𝛷(𝑥)=1 2(1 + e (𝑥 √2)) I in oduces s ochas ic egula iza ion e ec s and is used in models like BERT. 10. Spline (Cus om) This is a piecewise-de ined ac i a ion unc ion combining a quad a ic le segmen , a linea cen al segmen , and a gen le-slope linea igh segmen [9]. 𝑓(𝑥)={0.01∙𝑥2, 𝑥<0 𝑥, 0≤𝑥<1 0.1∙(𝑥−1)+1, 𝑥≥1 The unc ion is smoo h and con inuous, designed o p ese e g adien low while in oducing non-linea i y in a con ollable manne . 4. Resul s and Discussion This sec ion p esen s and analyzes he expe imen al esul s ob ained by aining iden ical con olu ional neu al ne wo ks using en di e en ac i a ion unc ions on he CIFAR-10 da ase . The e alua ion ocused on h ee me ics: es accu acy, es loss, and aining ime in seconds. Table 1 summa izes he quan i a i e ou comes o each ac i a ion unc ion. Ac i a ion Tes Accu acy (%) Tes Loss T aining Time (s) Spline 71.48 0.8609 53.49 Swish 68.33 0.9785 54.11 ELU 67.98 0.9827 55.47 ReLU 67.87 0.9303 57.08 GELU 67.54 0.9873 55.39 So sign 66.31 0.9861 55.82 SELU 65.29 1.0195 53.73 Tanh 63.77 1.0639 51.42 So plus 55.06 1.2819 51.55 Sigmoid 10.00 2.3059 54.66 Table 1 Quan i a i e Compa ison The esul s e eal ha he cus om spline ac i a ion unc ion achie ed he highes es accu acy (71.48%) among all en es ed unc ions. I s pe o mance su passes ha o well-es ablished ac i a ions like ReLU (67.87%), Swish (68.33%), and ELU (67.98%), while main aining a compa able aining ime (53.49 seconds). This sugges s ha he piecewise s uc u e o he spline unc ion—combining quad a ic, linea , and damped linea segmen s— p o ides bo h smoo h g adien low and lexible lea ning dynamics. Swish and GELU, known o hei smoo h and non-mono onic na u e, also showed compe i i e pe o mance, ein o cing indings om p e ious s udies [3]. T adi ional unc ions such as Tanh and So plus deli e ed lowe accu acy, likely due o anishing g adien issues, while Sigmoid yielded a me e 10.00% accu acy, indica ing comple e aining ailu e in his deep a chi ec u e—consis en wi h i s known sa u a ion limi a ions [2]. In e es ingly, aining ime ac oss all unc ions emained wi hin a igh ange (51–57 seconds), indica ing ha none o he es ed unc ions in oduce signi ican compu a ional o e head in a small-scale CNN. This con i ms ha exp essi e unc ions like spline can be p ac ical in eal- ime sys ems. Figu e 1 illus a es he e olu ion o es accu acy o e 10 epochs o all en ac i a ion unc ions. The esul s alida e he impo ance o choosing ac i a ion unc ions ha balance smoo hness, non- linea i y, and g adien p opaga ion. While ReLU emains a s ong baseline, ad anced o cus omized unc ions—such as Swish, GELU, and pa icula ly 84 “Al-Fa g‘oniy a lodla i” elek on ilmiy ju nali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendan s o Al-Fa ghani" elec onic scien i ic jou nal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 yea Электронный научный журнал "Потомки Аль- Фаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год h ps://al- a goniy.uz/ Spline—can p o ide measu able pe o mance gains wi hou sac i icing e iciency. This compa ison also suppo s ea lie s udies emphasizing he con ex -speci ic beha io o ac i a ion unc ions [1][3], and highligh s he po en ial o adap i e spline-based ac i a ions in CNN-based image classi ica ion. Conclusion This pape p esen ed a comp ehensi e empi ical compa ison o en ac i a ion unc ions, including a cus om spline-based unc ion, in a con olled deep lea ning se ing using he CIFAR-10 image classi ica ion da ase . All models sha ed he same CNN a chi ec u e and aining con igu a ion o ensu e a ai e alua ion. The esul s demons a e ha he p oposed spline ac i a ion unc ion achie ed he highes es accu acy (71.48%), ou pe o ming widely used al e na i es such as ReLU, Swish, and GELU. Mo eo e , i main ained compe i i e aining ime, indica ing i s p ac icali y o eal- ime and esou ce- cons ained applica ions. O he ad anced unc ions like Swish and ELU also pe o med well, whe eas adi ional unc ions like Sigmoid and Tanh showed limi ed e ec i eness in deep CNNs, likely due o anishing g adien issues. The indings highligh he impo ance o ac i a ion unc ion choice in neu al ne wo k design and sugges ha adap i e, smoo h, and piecewise-de ined unc ions such as spline can o e measu able imp o emen s in pe o mance. As u u e wo k, we plan o ex end his analysis o mo e complex a chi ec u es (e.g., ResNe , Vision T ans o me s), addi ional da ase s (e.g., CIFAR-100, ImageNe ), and u he explo e lea nable o pa ame e ized spline a ian s ha can adap du ing aining. Re e ences 1. Bou aya, S., & Belangou , A. (2024). A compa a i e analysis o ac i a ion unc ions in neu al ne wo ks: un eiling ca ego ies. Bulle in o Elec ical Enginee ing and In o ma ics, 13(5), 3301-3308. 2. Sha ma, S., Sha ma, S., & A haiya, A. (2017). Ac i a ion unc ions in neu al ne wo ks. Towa ds Da a Sci, 6(12), 310-316. 3. Dubey, S. R., Singh, S. K., & Chaudhu i, B. B. (2022). Ac i a ion unc ions in deep lea ning: A comp ehensi e su ey and benchma k. Neu ocompu ing, 503, 92-108. 4. Rasamoelina, A. D., Adjailia, F., & Sinčák, P. (2020, Janua y). A e iew o ac i a ion unc ion o a i icial neu al ne wo k. In 2020 IEEE 18 h wo ld symposium on applied machine in elligence and in o ma ics (SAMI) (pp. 281-286). IEEE. 5. Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empi ical e alua ion o ec i ied ac i a ions in con olu ional ne wo k. a Xi p ep in a Xi :1505.00853. 6. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Del ing deep in o ec i ie s: Su passing human-le el pe o mance on imagene classi ica ion. In P oceedings o he IEEE in e na ional con e ence on compu e ision (pp. 1026-1034). 7. Sca dapane, S., Sca pini i, M., Comminiello, D., & Uncini, A. (2017, June). Lea ning ac i a ion unc ions om da a using cubic spline in e pola ion. In I alian Wo kshop on Neu al Ne s (pp. 73-83). Cham: Sp inge In e na ional Publishing. 8. Boh a, P., Campos, J., Gup a, H., Aziznejad, S., & Unse , M. (2020). Lea ning ac i a ion unc ions in deep (spline) neu al ne wo ks. IEEE Open Jou nal o Signal P ocessing, 1, 295-309. 9. P imbe o Abbaz. (2025). DEEPFAKE DETECTION USING A HYBRID RESNEXT AND LSTM ARCHITECTURE. Al-Fa g’oniy a lodla i, 1(2), 87-94. 10. P imbe o Abbaz. (2025). DEEPFAKE TECHNOLOGY: THREAT LANDSCAPE, DETECTION TECHNIQUES, AND ETHICAL GOVERNANCE. Al-Fa g’oniy a lodla i, 1(2), 106- 112.