A COMPARATIVE STUDY OF ACTIVATION FUNCTIONS IN DEEP LEARNING MODELS

80 “Al-Farg‘oniy avlodlari” elektron ilmiy jurnali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendants of Al-Farghani" electronic scientific journal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 year Электронный научный журнал "Потомки АльФаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год https://al-fargoniy.uz/ A COMPARATIVE STUDY OF ACTIVATION FUNCTIONS IN DEEP LEARNING MODELS Primbetov Abbaz, Phd student, Tashkent University of Information Technologies named after Muhammad Al-Khwarizmi. Senior lecturer, University of Tashkent for applied sciences [email protected] Akbarov Navruz, Phd student, Tashkent University of Information Technologies named after Muhammad Al-Khwarizmi [email protected] Abstract: Activation functions play a vital role in the training dynamics and generalization performance of deep learning models. This study presents a comparative analysis of ten widely used activation functions—ReLU, Sigmoid, Tanh, ELU, SELU, Softplus, Softsign, Swish, GELU, and a custom spline-based function—within a unified convolutional neural network (CNN) architecture. All models were trained and evaluated on the CIFAR-10 dataset under identical experimental settings, including fixed learning rate, batch size, number of epochs, and architecture configuration. The results show that the proposed spline activation function achieved the highest test accuracy of 71.48%, outperforming popular functions such as ReLU (67.87%) and Swish (68.33%). In contrast, traditional functions like Sigmoid exhibited significantly lower accuracy (10.00%), reaffirming known limitations in deep network contexts. These findings demonstrate the potential of adaptive, piecewise-defined activation functions to enhance model performance while maintaining competitive training efficiency. The study provides practical insights into activation function selection for image classification tasks. Keywords: Activation Functions, Spline Activation, Convolutional Neural Networks, CIFAR-10, Image Classification, ReLU, Swish, Deep Learning 1. Indroduction Activation functions (AFs) are essential components of deep learning architectures, introducing non-linearity that enables neural networks to approximate complex mappings between inputs and outputs. Without AFs, neural networks would be reduced to simple linear models, regardless of their depth or structure [1]. The choice of activation function significantly affects a model’s training stability, convergence speed, and generalization capacity [2]. Over the years, numerous activation functions have been proposed, ranging from traditional functions like Sigmoid and Tanh to more advanced forms such as ReLU, ELU, Swish, and GELU [3]. Each function exhibits distinct mathematical properties, including smoothness, monotonicity, boundedness, and differentiability, which impact how well the network learns from data [4]. ReLU and its variants, for example, have become the de facto standard for many convolutional neural networks (CNNs) due to their simplicity and computational efficiency [5]. However, these functions are not without limitations—ReLU suffers from dying neuron problems, while Sigmoid and Tanh can cause vanishing gradients in deeper networks [6]. In response to such limitations, adaptive and learnable activation functions have emerged. Among them, spline-based activation functions provide a flexible, data-driven alternative by modeling piecewise continuous transformations with trainable parameters [7]. Recent studies have shown that spline activations can improve model expressiveness and learning stability without adding substantial computational complexity [8]. This paper presents a comparative empirical study of ten activation functions, including a custom 81 “Al-Farg‘oniy avlodlari” elektron ilmiy jurnali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendants of Al-Farghani" electronic scientific journal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 year Электронный научный журнал "Потомки АльФаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год https://al-fargoniy.uz/ spline-based function, using a consistent CNN architecture trained on the CIFAR-10 dataset. All hyperparameters, preprocessing steps, and network structures were kept identical to ensure a fair evaluation. Our goal is to quantify and analyze the performance of each function in terms of classification accuracy, training time, and loss. The findings offer practical insights into activation function selection for image classification tasks, particularly in resourceconstrained or performance-sensitive applications. 2. Related Work Activation functions (AFs) have been extensively studied due to their crucial role in enabling deep neural networks to learn non-linear mappings. Traditional activation functions such as Sigmoid and Tanh were widely used in early neural network architectures but suffered from vanishing gradient problems, which hindered the training of deep models [1]. This led to the emergence of ReLU (Rectified Linear Unit), which became the default choice in convolutional neural networks due to its simplicity, sparsity, and efficient gradient flow [2]. To overcome ReLU’s limitations such as the “dying ReLU” problem, several variants were proposed including Leaky ReLU, Parametric ReLU (PReLU), and Exponential Linear Unit (ELU). These functions aim to maintain non-zero gradients in the negative input space while preserving the benefits of ReLU [3]. In a comprehensive analysis, Bouraya et al. [4] categorized activation functions into five major types: bounded (e.g., Sigmoid, Tanh), unbounded (ReLU, ELU), exponential-based, adaptive, and diversified functions. Their taxonomy focused on key mathematical characteristics such as monotonicity, smoothness, and boundedness. Similarly, Dubey et al. [5] benchmarked 18 different activation functions across several deep learning tasks, concluding that no single function consistently outperforms others in all scenarios, reinforcing the importance of task-specific evaluation. Recently, adaptive and learnable activation functions have gained traction. Among these, Splinebased functions have shown promise by offering piecewise-smooth, parameterized transformations that can be learned during training. Scardapane et al. [6] and Bohra et al. [7] demonstrated that spline-based activations outperform traditional functions in several classification benchmarks by improving expressiveness and stability, particularly in deeper models. However, despite extensive theoretical work, few studies have empirically compared traditional and spline-based activation functions under identical experimental conditions. Most prior benchmarks involve either shallow networks or varying architectures, which complicates direct comparison. This study fills that gap by systematically evaluating ten activation functions—including a custom spline— on the CIFAR-10 dataset using an identical CNN architecture and training setup. 3. Methodology To ensure a fair and controlled comparison of different activation functions, we designed a consistent experimental framework based on a convolutional neural network (CNN) trained on the CIFAR-10 image classification dataset. This section outlines the dataset characteristics, model architecture, activation functions tested, and the training configuration. 3.1 Dataset The CIFAR-10 dataset is a widely used benchmark for evaluating image classification models. It contains 60,000 color images of size 32×32 pixels, divided into 10 categories such as airplanes, automobiles, birds, cats, and more. The dataset is split into 50,000 training images and 10,000 test images. All images were normalized to the [0,1] range by dividing pixel values by 255. Label vectors were one-hot encoded to be compatible with the categorical cross-entropy loss function. 3.2 Model Architecture A shallow yet expressive CNN architecture was adopted to reduce the risk of overfitting while ensuring sufficient capacity for function comparison. The architecture is as follows: Input: 32×32×3 image Conv2D layer with 32 filters (3×3 kernel), followed by activation 82 “Al-Farg‘oniy avlodlari” elektron ilmiy jurnali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendants of Al-Farghani" electronic scientific journal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 year Электронный научный журнал "Потомки АльФаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год https://al-fargoniy.uz/ Conv2D layer with 64 filters (3×3 kernel), followed by activation MaxPooling2D (2×2) Dropout (rate = 0.25) Flatten Dense layer with 128 units, followed by activation Dropout (rate = 0.5) Output: Dense layer with 10 units and softmax activation draw for this model 3.3 Training Configuration To ensure fairness and reproducibility in the comparison of activation functions, all models were trained under identical hyperparameter settings. Each network was trained for 5 epochs using a batch size of 64. The Adam optimizer was employed with a fixed learning rate of 0.001, and the loss function used was categorical crossentropy, appropriate for multi-class classification tasks. A 10% validation split was applied to the training set during training to monitor the model's generalization performance. The performance of each activation function was evaluated based on three key metrics: test accuracy, test loss, and training time, measured in seconds. All training sessions were conducted in a single GPU-enabled environment to maintain consistent hardware conditions and eliminate variability due to processing resources. 3.4 Activation Function Formulations The mathematical definitions of the ten activation functions compared in this study are summarized below, along with their properties and references. 1. ReLU (Rectified Linear Unit) This is one of the most widely used activation functions in convolutional neural networks due to its simplicity and computational efficiency. It outputs the input directly if it is positive; otherwise, it returns zero. 𝑓(𝑥)=𝑚𝑎𝑥(0,𝑥) It enables sparse representations but suffers from the “dying ReLU” problem, where neurons can become inactive during training and never recover. 2. Sigmoid The sigmoid function maps any real-valued input to the range (0, 1), making it suitable for probabilistic interpretation. 𝑓(𝑥)=1 1+𝑒−𝑥 However, it tends to saturate for large values of |x| and causes vanishing gradients, which slows down learning in deep networks. 3. Tanh (Hyperbolic Tangent) Similar to sigmoid but zero-centered, the tanh function maps inputs to (−1, 1). 𝑓(𝑥)=tanh(𝑥)=𝑒𝑥−𝑒−𝑥 𝑒𝑥+𝑒−𝑥 It has better gradient dynamics than sigmoid but still suffers from vanishing gradients at extreme values. 4. ELU (Exponential Linear Unit) This function outputs the identity for positive values and an exponential curve for negatives, helping maintain activations closer to zero mean. 𝑓(𝑥)={𝑥, 𝑥≥0 𝛼(𝑒𝑥−1), 𝑥<0 (𝑐𝑜𝑚𝑚𝑜𝑛𝑙𝑦 𝛼=1) It helps mitigate vanishing gradient problems and accelerates learning. 5. SELU (Scaled ELU) A scaled variant of ELU designed for selfnormalizing neural networks. 𝑓(𝑥)=𝜆∙{𝑥, 𝑥≥0 𝛼(𝑒𝑥−1), 𝑥<0 (𝜆≈1.05, 𝛼≈1.67) With appropriate λ and α, it maintains zero mean and unit variance throughout the network, aiding convergence. 6. Softplus Softplus is a smooth approximation of ReLU. 𝑓(𝑥)=𝑙𝑛(1+𝑒𝑥 ) It is always positive and differentiable, but computationally more expensive than ReLU. 7. Softsign A bounded, continuous function similar to tanh but with a simpler expression. 𝑓(𝑥) = 𝑥 1 + |𝑥| Its gradient decays more slowly than tanh, which may benefit learning. 8. Swish 83 “Al-Farg‘oniy avlodlari” elektron ilmiy jurnali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendants of Al-Farghani" electronic scientific journal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 year Электронный научный журнал "Потомки АльФаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год https://al-fargoniy.uz/ Swish is a smooth, non-monotonic function defined as the input multiplied by its sigmoid. 𝑓(𝑥) = 𝑥 · 𝑠𝑖𝑔𝑚𝑜𝑖𝑑(𝑥) = 1 1+𝑒−𝑥 It has been shown to outperform ReLU in deeper networks due to its self-gating behavior. 9. GELU (Gaussian Error Linear Unit) This function weights the input by the cumulative distribution function of the standard normal distribution. 𝑓(𝑥)=𝑥⋅𝛷(𝑥) 𝑤ℎ𝑒𝑟𝑒 𝛷(𝑥)=1 2(1 + erf (𝑥 √2)) It introduces stochastic regularization effects and is used in models like BERT. 10. Spline (Custom) This is a piecewise-defined activation function combining a quadratic left segment, a linear central segment, and a gentle-slope linear right segment [9]. 𝑓(𝑥)={0.01∙𝑥2, 𝑥<0 𝑥, 0≤𝑥<1 0.1∙(𝑥−1)+1, 𝑥≥1 The function is smooth and continuous, designed to preserve gradient flow while introducing non-linearity in a controllable manner. 4. Results and Discussion This section presents and analyzes the experimental results obtained by training identical convolutional neural networks using ten different activation functions on the CIFAR-10 dataset. The evaluation focused on three metrics: test accuracy, test loss, and training time in seconds. Table 1 summarizes the quantitative outcomes for each activation function. Activation Test Accuracy (%) Test Loss Training Time (s) Spline 71.48 0.8609 53.49 Swish 68.33 0.9785 54.11 ELU 67.98 0.9827 55.47 ReLU 67.87 0.9303 57.08 GELU 67.54 0.9873 55.39 Softsign 66.31 0.9861 55.82 SELU 65.29 1.0195 53.73 Tanh 63.77 1.0639 51.42 Softplus 55.06 1.2819 51.55 Sigmoid 10.00 2.3059 54.66 Table 1 Quantitative Comparison The results reveal that the custom spline activation function achieved the highest test accuracy (71.48%) among all ten tested functions. Its performance surpasses that of well-established activations like ReLU (67.87%), Swish (68.33%), and ELU (67.98%), while maintaining a comparable training time (53.49 seconds). This suggests that the piecewise structure of the spline function—combining quadratic, linear, and damped linear segments— provides both smooth gradient flow and flexible learning dynamics. Swish and GELU, known for their smooth and non-monotonic nature, also showed competitive performance, reinforcing findings from previous studies [3]. Traditional functions such as Tanh and Softplus delivered lower accuracy, likely due to vanishing gradient issues, while Sigmoid yielded a mere 10.00% accuracy, indicating complete training failure in this deep architecture—consistent with its known saturation limitations [2]. Interestingly, training time across all functions remained within a tight range (51–57 seconds), indicating that none of the tested functions introduce significant computational overhead in a small-scale CNN. This confirms that expressive functions like spline can be practical in real-time systems. Figure 1 illustrates the evolution of test accuracy over 10 epochs for all ten activation functions. The results validate the importance of choosing activation functions that balance smoothness, nonlinearity, and gradient propagation. While ReLU remains a strong baseline, advanced or customized functions—such as Swish, GELU, and particularly 84 “Al-Farg‘oniy avlodlari” elektron ilmiy jurnali ISSN 2181-4252. Tom: 1 | Son: 3 | 2025-yil "Descendants of Al-Farghani" electronic scientific journal. ISSN 2181-4252. Vol: 1 | Iss: 3 | 2025 year Электронный научный журнал "Потомки АльФаргани" ISSN 2181-4252. Том: 1 | Выпуск: 3 | 2025 год https://al-fargoniy.uz/ Spline—can provide measurable performance gains without sacrificing efficiency. This comparison also supports earlier studies emphasizing the context-specific behavior of activation functions [1][3], and highlights the potential of adaptive spline-based activations in CNN-based image classification. Conclusion This paper presented a comprehensive empirical comparison of ten activation functions, including a custom spline-based function, in a controlled deep learning setting using the CIFAR-10 image classification dataset. All models shared the same CNN architecture and training configuration to ensure a fair evaluation. The results demonstrate that the proposed spline activation function achieved the highest test accuracy (71.48%), outperforming widely used alternatives such as ReLU, Swish, and GELU. Moreover, it maintained competitive training time, indicating its practicality for real-time and resourceconstrained applications. Other advanced functions like Swish and ELU also performed well, whereas traditional functions like Sigmoid and Tanh showed limited effectiveness in deep CNNs, likely due to vanishing gradient issues. The findings highlight the importance of activation function choice in neural network design and suggest that adaptive, smooth, and piecewise-defined functions such as spline can offer measurable improvements in performance. As future work, we plan to extend this analysis to more complex architectures (e.g., ResNet, Vision Transformers), additional datasets (e.g., CIFAR-100, ImageNet), and further explore learnable or parameterized spline variants that can adapt during training. References 1. Bouraya, S., & Belangour, A. (2024). A comparative analysis of activation functions in neural networks: unveiling categories. Bulletin of Electrical Engineering and Informatics, 13(5), 3301-3308. 2. Sharma, S., Sharma, S., & Athaiya, A. (2017). Activation functions in neural networks. Towards Data Sci, 6(12), 310-316. 3. Dubey, S. R., Singh, S. K., & Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503, 92-108. 4. Rasamoelina, A. D., Adjailia, F., & Sinčák, P. (2020, January). A review of activation function for artificial neural network. In 2020 IEEE 18th world symposium on applied machine intelligence and informatics (SAMI) (pp. 281-286). IEEE. 5. Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.00853. 6. He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision (pp. 1026-1034). 7. Scardapane, S., Scarpiniti, M., Comminiello, D., & Uncini, A. (2017, June). Learning activation functions from data using cubic spline interpolation. In Italian Workshop on Neural Nets (pp. 73-83). Cham: Springer International Publishing. 8. Bohra, P., Campos, J., Gupta, H., Aziznejad, S., & Unser, M. (2020). Learning activation functions in deep (spline) neural networks. IEEE Open Journal of Signal Processing, 1, 295-309. 9. Primbetov Abbaz. (2025). DEEPFAKE DETECTION USING A HYBRID RESNEXT AND LSTM ARCHITECTURE. Al-Farg’oniy avlodlari, 1(2), 87-94. 10. Primbetov Abbaz. (2025). DEEPFAKE TECHNOLOGY: THREAT LANDSCAPE, DETECTION TECHNIQUES, AND ETHICAL GOVERNANCE. Al-Farg’oniy avlodlari, 1(2), 106112.