NEXT-GENERATION RANSOMWARE DEFENSE: DEEP LEARNING-BASED TRAFFIC CLASSIFICATION AT THE NETWORK LAYER

Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4607 NEXT-GENERATION RANSOMWARE DEFENSE: DEEP LEARNING-BASED TRAFFIC CLASSIFICATION AT THE NETWORK LAYER ZAID ALI HUSSEIN1, OMER ABDULHALEEM NASER2, ZAID ALI HAMID3 1Department of Biomass Energy, Al-Nahrain Renewable Energy Research Center, AL-Nahrain University,Jadriya ,Baghdad 10072 ,Iraq 2Electronic Computer Center, University of Information Technology and Communications, Baghdad, Iraq 3Communication Engineering Department, University of Technology, Baghdad, Iraq [email protected], [email protected], [email protected] ABSTRACT They were more advanced than ever in ransomware, fileless execution, polymorphic encryption, and encrypted C2 communications. The problem, however, widens, with Ransomware as a Service (RaaS) now having joined the party. To fight against these trends, we introduce a next-generation ransomware defense framework that is a deep learning-based real-time detection system capable of detecting ransomware in real encrypted network traffic without any payload inspection. In particular, the system classifies traffic using statistical flow metrics, protocol-specific patterns, and behavioral anomalies by means of transformer-based models. The trained model is tested on a 35 million flow dataset consisting of real ransomware samples, benign enterprise traffic, and adversarial flow samples with around 98.9%, 99.2%, and 98.5% in accuracy, precision, and recall, respectively. In addition, it is robust and scalable for adversarial training and federated learning. The system is deployed into the enterprise environment and has the capability to provide real-time response (0.5s detection) and hence is viable for current enterprise, IoT, and cloud networks. Keywords: Ransomware Detection, Encrypted Network Traffic, Deep Learning, Transformer Models, Federated Learning, Adversarial Machine Learning 1. INTRODUCTION Ransomware is becoming one of the most pressing cyber threats and is reaching people across the globe, including individuals, enterprises, and government institutions. Unlike traditional malware, modern ransomware variants, namely, Kryptik and WannaCry, have eked out a lot from traditional malware by harnessing innovative effectiveness-avoiding methods, i.e., contingency, polymorphic encryption, and enciphered commandand-control (C2) communications. These techniques are not compatible with the signaturebased detection mechanisms, as more and more attackers are beginning to execute meeting attacks using Transport Layer Security (TLS) [3] and Deep Packet Inspection (DPI) [4]. The market for Ransomware as a Service (RaaS), however, was widespread and further aggravated the current threat landscape. This way, even the low-skilled cybercriminals can run sophisticated ransomware campaigns, and it also tremendously increases the global attack surface. Furthermore, modern blockchain styles of ransomware employ strong anti-SI, aggregation, and adversarial ML techniques to prevent detection methods. Most such techniques detect with traditional impedance responses on the basis of endpoints, are most reliable, and also produce a lot of false positives and low time responses [6]. To overcome these limitations, recent research tries to repel the use of some machine learning (ML) and deep learning (DL) methods that could be used for detecting ransomware at the network layer. Network layer approaches are in contrast to the effect of endpoint-based detection methods that depend on static signatures, static traffic flow behavior, statistical anomalies, and protocolspecific patterns, which are applicable in encrypted communications. As such, CNNs and RNNs can be proven to detect ransomware-embedded traffic flows with high accuracy [8]. However, critical to existing ML systems are their representational vulnerability to adversarial attacks, inability to scale to large networks, and only a limited capacity for zero-day attacks [9]. Given this situation, we hypothesize that, from flow-level behavioral, statistical, and protocolspecific features, we can accurately detect encrypted network traffic used by ransomware Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4608 companies without using payload content. We aim at designing a real-time, transformer-based ransomware detection framework with high accuracy, robustness against adversarial evasion, and scale across one or more enterprise, cloud, and IoT environments in this work. 2. LITERATURE REVIEW 2.1 Evolution of Ransomware and Attack Mechanisms Besides plain locker malware, ransomware have become sophisticated multi-phase cyber threat to enterprise networks, cloud and IoT networks [8]. Email phishing and exploit kits were the early ways of ransomware variants like CryptoLocker and WannaCry, though the modern ones like Conti, REvil, and LockBit also use the double extortion tactics by which attackers encrypt the data and only exfiltrate it too [9][10]. Finally, ransomware-as-aservice (RaaS) has appeared on the ransomware domain, making it easier for cybercriminals to use this tool without expert skills [11]. It is revealed by research that asset like cloud computing, industrial control systems (ICS) and smart infrastructure have been prominent ransomware targets because of poor network segmentation and weak access controls [12][13]. 2.2 Deep Learning for Ransomware Detection at the Network Layer Ransomware detection at the network layer is very common using deep learning-based approaches because it can generalize patterns of ransomware attacks rather than relying on static signatures [14]. Currently modern IDS have problems against evolving ransomware threats, deep learning classifiers on the other hand can identify ransomware based on: protocol behaviors, network entropy fluctuations and statistical deviations [15][16]. Various studies propose hybrid deep learning architectures like convolutional neural network (CNN), recurrent neural network (RNN) and transformers-based models for real time encrypted traffic classification [17][18]. In addition, Generative Adversarial Networks (GANs) have also been used to produce samples of adversarial ransomware to train better models for resisting evasive tactics [19]. 2.3 Behavioral Analysis and Anomaly Detection in Encrypted Communications Since ransomware operators are increasingly using TLS encryption for stealthy communication; the conventional packet-based analysis techniques are rendered useless [20]. In response to this, researchers have started to perform behavioral traffic analysis, by analyzing patterns of the network flow, levels of entropy not in line with normal levels, and irregularities in TLS handshakes [21] [22]. Empirical studies show that ransomware infected hosts are characterized by prominent network behaviors (e.g., high frequency of ‘beaconing’ to C2 servers, abnormal increases in encrypted data traffic, irregular DNS queries) [23][24]. These risks are mitigated using software defined networking (SDN) based defense mechanisms that offer an improved real time access control and ransomware flow mitigation [25][26]. Besides, blockchain-based cybersecurity solutions were suggested to track ransomware indicators on decentralized networks using an immutable distributed ledger technology to provide real time threat intelligence sharing [27]. The other innovative approach reflects monitoring with hardware performance counter (HPC) to observe the ransomware’s encryption activity by computing the CPU and memory usage patterns [28][29]. Table 1: Comparison Between Previous Studies and Our Study on Ransomware Detection Technique (Year) [#] Scope (RT) Acc / Key Value DL + Entropy (2024) [1] Windows (No) 94% – Entropy features DNA + ML (2020) [7] File (No) 95% – DNA mimic Hybrid ML (2021) [3] Crypto (No) 92.5% – Multi-stage profiling Metaheuristic + Traffic ML (2022) [4] Android (Partial) 93% – Optimized detection Resource Monitor (2023) [13] System (Yes) 91% – HW metrics ML Survey (2023) [5] General (N/A) N/A – Overview Entropy Analysis (2021) [6] File (No) 89% – Entropy metrics Evolutionary ML (2021) [8] Android (No) 90% – Imbalance handling DL (CNN-GRU) (2020) [9] Encrypted (Yes) 96% – Obfuscation resilience Threat Hunting (2022) [10] Enterprise (Yes) 94% – Proactive Multi-Level ML (2021) [11] Crypto (No) 91.2% – Attack phase model Feature Selection (2022) [12] Android (Yes) 93.8% – Traffic detection Resource Monitoring (2023) [13] Local (Yes) 91% – System-level ML Framework Review (2022) [14] General (N/A) N/A – Benchmark ML Review (2024) [15] General (N/A) N/A – Gaps Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4609 Obfuscation Defense (2023) [16] Windows (Partial) 87% – Evading detection Dataset Repo (2020) [17] Eval (N/A) N/A – Benchmarking Honeypot Design (2020) [ – ] IoT (Yes) 90.5% – Deception SDN Detection (2021) [18] Self-spread (Yes) 93.5% – Net mitigation Behavior Analysis (2022) [20] Targeted (No) N/A – Tactics insight HW Trojan Simulation (2022) [21] CPS (No) N/A – Feasibility test HPC Classifier (2024) [22] Non-virt (Yes) 88% – Low-level signature ATT&CK Mapping (2023) [23] Behavioral (No) N/A – Tactical patterns Explainable AI (2024) [24] General (Yes) 96% – AI interpretability Pre-Encrypt Mining (2020) [25] Crypto (Yes) 91% – Early signal Federated Learning (2021) [26] IIoT (Yes) 95.6% – Distributed ML Area-Based Study (2025) [27] General (No) N/A – Attack taxonomy GANs Detection (2022) [28] Encrypted traffic (Yes) 97% – Zero-day Web Defense (2024) [29] CPS (Yes) 94% – Auto-mitigation Nilsimsa + RF (2024) [30] IoMT (Yes) 100% – No feature eng. Secure Storage (2020) [31] Infra (No) N/A – Storage prevention Transformer DL + XAI (2025) [ – ] Ent/IoT/Cloud (Yes) 98.9% – Adv. detection Technique (Year) [#] Scope (RT) Acc / Key Value DL + Entropy (2024) [1] Windows (No) 94% – Entropy features DNA + ML (2020) [7] File (No) 95% – DNA mimic Hybrid ML (2021) [3] Crypto (No) 92.5% – Multi-stage profiling Metaheuristic + Traffic ML (2022) [4] Android (Partial) 93% – Optimized detection Resource Monitor (2023) [13] System (Yes) 91% – HW metrics ML Survey (2023) [5] General (N/A) N/A – Overview Entropy Analysis (2021) [6] File (No) 89% – Entropy metrics Evolutionary ML (2021) [8] Android (No) 90% – Imbalance handling DL (CNN-GRU) (2020) [9] Encrypted (Yes) 96% – Obfuscation resilience Threat Hunting (2022) [10] Enterprise (Yes) 94% – Proactive Multi-Level ML (2021) [11] Crypto (No) 91.2% – Attack phase model Feature Selection (2022) [12] Android (Yes) 93.8% – Traffic detection Resource Monitoring (2023) Local (Yes) 91% – System-level [13] ML Framework Review (2022) [14] General (N/A) N/A – Benchmark ML Review (2024) (2025) [15] [32][33] General (N/A) N/A – Gaps Obfuscation Defense (2023) [16] Windows (Partial) 87% – Evading detection Dataset Repo (2020) [17] Eval (N/A) N/A – Benchmarking Honeypot Design (2020) [ – ] IoT (Yes) 90.5% – Deception SDN Detection (2021) [18] Self-spread (Yes) 93.5% – Net mitigation Behavior Analysis (2022) [20] Targeted (No) N/A – Tactics insight HW Trojan Simulation (2022) [21] CPS (No) N/A – Feasibility test HPC Classifier (2024) [22] Non-virt (Yes) 88% – Low-level signature ATT&CK Mapping (2023) [23] Behavioral (No) N/A – Tactical patterns Explainable AI (2024) [24] General (Yes) 96% – AI interpretability Pre-Encrypt Mining (2020) [25] Crypto (Yes) 91% – Early signal Federated Learning (2021) [26] IIoT (Yes) 95.6% – Distributed ML Area-Based Study (2025) [27] General (No) N/A – Attack taxonomy GANs Detection (2022) [28] Encrypted traffic (Yes) 97% – Zero-day Web Defense (2024) [29] CPS (Yes) 94% – Auto-mitigation 3. PROBLEM STATEMENT Among the latest cybersecurity threats, ransomware has topped the list as the most dangerous threat affecting people, businesses, and government institutions across the globe. Sophistication of ransomware attacks and highly adopted advanced enemy techniques have made traditional detection and mitigation methods obsolete. Traditional security methods like rulebased intrusion detection systems (IDS) and signature-based antivirus programs do not keep up with the changes in ransomware attacks. In particular, this research intends to tackle the aforementioned critical challenges by creating a strong, scalable, and adaptive ransomware detection framework that is effective in detecting malicious activity even in an encrypted network. 3.1 Increasing Complexity of Ransomware Attacks Traditionally, ransomware has always left some trace on the victim's system, but this has Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4610 evolved, full stop. Too many families are fileless and execute directly in memory, leaving little to no forensic trace on the victim's system, so static analysis methods are largely ineffective. The ransomware variants used in modern times utilize advanced evasion techniques to evade traditional detection mechanisms. These include:  Metamorphic and Polymorphic Encryption: The attack uses encryption techniques to dynamically generate new malware signatures and makes signaturebased detection obsolete.  DPI disabled: With the encryption of Command-and-Control (C2) Communications when using TLS 1.3 and QUIC protocols, DPI is not able to send the payload contents to their destination.  Ransomware as a service (RaaS): There is an increased rise of RaaS, which has made ransomware distribution more democratized, enabling non-technical cybercriminals to be on par with highly sophisticated attacks. With these enhancements in place, defending against and, more importantly, determining when to block ransomware threats in real-time is difficult for traditional security efforts. 3.2 Limitations of Existing Ransomware Detection Methods There have been different approaches for detecting running ransomware, which exist in many publications, and all these methods have their limitations:  Signature-Based Detection: Malware signatures are checked by antivirus software to verify whether there is a match to it in their database, and therefore, signature-based detection cannot defend against unknown attacks.  Heuristic-Based and Static Analysis: These approaches rely on predefined rules to detect the anomalies; however, they cannot work with obfuscation techniques such as code packing and encryption.  Behavioral and Anomaly-Based Detection: More effective than signature-based methods, behavioral detection often leads to high false positives because what is legitimate, encrypted traffic and what is related to ransomware communications cannot be easily discerned.  Traditional ML-based methods primarily rely on relatively simple feature extraction and classification models, which can be easily thwarted by adversarial ML methods, compromising their robustness. These limitations make the need for a more adaptive and intelligent approach, which is able to detect the ransomware activity in a complex and encrypted form, apparent. 3.3 Challenges in Detecting Ransomware in Encrypted Network Traffic This has led to the adoption of an encrypted communication channel as one of the most pressing challenges of ransomware detection.  Firewalls can only block whole applications or have barely acceptable levels of false positives due to modern encryption protocols like TLS 1.3 and QUIC that can be detected, disabling the gate context to perform DPI-based detection that relies on the traffic behaviors.  No visibility of payload data in end-to-end encryption, which makes the network data traffic protection increasingly difficult for endpoint-based security mechanisms.  Advanced ransomware variants hide by using delayed execution techniques and by side channel communication (beaconing) in order to blend into the normal network activity so as to be inconspicuous. Thus, these evasion techniques involving encryption drive the need to transition into using deep learning-based models that can detect slight traffic anomalies and behavioral mutations. 3.4 The Need for Real-Time, Scalable, and Adaptive Ransomware Detection As both the frequency and the sophistication of these ransomware attacks continue to increase, it is necessary to have an effective defense mechanism.  LLML models are not fast enough to support real-time detection of ransomware incidents. Such a detection mechanism will be more responsive and adaptive.  Detection in Large Networks: Detection methods currently in use fail due to a lack of sufficient resources for large-scale enterprise deployments, given the high overhead. There is a need for a scalable framework that can analyze high-volume network traffic.  Provisions of Evolving Threats: Since cybercriminals are also using adversarial ML techniques more and more, static Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4611 models become ineffective over time. Also, since ransomware keeps evolving and new variants keep coming up, a system that is continuously learning and in place is required. 3.5 Research Motivation and Objectives This research suggests a next-generation ransomware defense framework that leverages a deep learning model to classify network traffic in real time in order to overcome the limitations of existing ransomware detection. The proposed system will: 1. Create a deep learning model built on top of a transformer that is able to deduce encrypted ransomware traffic without examining the payload. 2. Robust training techniques and dynamic adaptation strategies are explored to enhance resilience against adversarial ML attacks. 3. Combine an anomaly, behavioral, and statistical flow-based verification layer for multi-layer detection. 4. Optimize the model performance with federated learning and distributed inference to improve scalability for largescale enterprise deployments. 5. Dynamic learning responds to and mitigates ransomware threats proactively by reacting to emerging ransomware tactics. 4. METHODOLOGY Advanced deep learning models and advanced network-based anomaly detection techniques have been used in the Next-Generation Ransomware Defense Framework to detect, analyze, and mitigate real-time ransomware threats. Transform this into a means by which ransomware can be detected even while being encrypted (as TLS 1.3, QUIC), and techniques such as polymorphism and adversarial ML used to avoid standard security solutions. It is highly scalable and highly adaptive to operation in both high-traffic enterprise environments, cloud infrastructures, and IoT networks. The methodology is presented in great detail, outlining how the data are collected, how features are engineered, how the various models are selected, how training procedures are applied, how the system is deployed in real time, how the error of the system is evaluated, and how data are processed for consequential analysis. This is a structured framework to detect and mitigate ransomware attacks in real-time. First, the network traffic features are extracted in terms of statistical and behavioral features, and they are further fed into the deep learning classification. The high-level overview of the distributed computing framework is presented in Figure 1. The final goal of such an alert system is to achieve real-time detection, analysis, and mitigation of ransomware threats through the usage of advanced deep learning models and network-based anomaly detection techniques. Using this methodology, ransomware can still be identified with encryption (TLS 1.3, QUIC), polymorphic behavior that attempts to evade traditional security solutions, or adversarial ML techniques. The system works on high-traffic enterprise networks, cloud-based infrastructures, and IoT networks, making it highly scalable and adaptable. It provides a very thorough treatment of the data collection, feature engineering, model selection, training procedures, real-time deployment, and system evaluation, to name a few. The Next-Generation Ransomware Defense Framework is a structure-based approach to detect and mitigate ransomware on the fly. First, it extracts features about statistical and behavioral characteristics of network traffic, then applies deep learning classification. This framework is illustrated in Figure 1 with a high-level view of the components. Figure 1: Overview of Ransomware Detection Framework It is composed of several layers, such as network traffic capture and feature extraction, and then goes Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4612 through statistical and behavioral analysis. The network activity features are passed through the deep learning model to classify the network activity as either benign or infected with ransomware. An automated threat response system guards against potential attacks in real time when they are detected. 4.1 Data Collection and Dataset Details 4.1.1 Data Sources Our detection framework is effective and depends on the diversity, comprehensiveness, and good labeling of the dataset. Based on the combination of: 1. Publicly Available Datasets  CICIDS2017 and CICDDoS2019: Provides real-world ransomware attack traces and benign traffic collected from network intrusions.  CTU-13 and UGR16: Large-scale datasets containing botnet traffic, useful for ransomware C2 detection.  Stratosphere IPS Dataset: Captures realworld malware traffic, including ransomware-infected communications. 2. Controlled Ransomware Executions in a Secure Lab Environment  Ransomware Samples: We use real-world ransomware strains (e.g., WannaCry, Ryuk, Conti, LockBit, DarkSide, Maze) and execute them in an isolated sandboxed network to capture network traffic behavior.  Encrypted Command and Control (C2) Communication: The lab environment is monitored using Wireshark, Zeek (Bro IDS), and Suricata to track the encryption behavior of ransomware during its communication with C2 servers.  Lateral Movement Simulation: We analyze how ransomware spreads across enterprise networks, targeting file shares and endpoint devices. 3. Benign Traffic Collection for Model Generalization  Enterprise Network Traffic: Normal user behavior from academic institutions, corporate environments, and data centers is recorded to avoid false positives.  Encrypted Legitimate Traffic: Web browsing (HTTPS), cloud application usage, VoIP calls, and VPN traffic are included to ensure that the model can differentiate between legitimate encryption and ransomware C2 behavior. 4.1.2 Data Preprocessing and Standardization However, raw network data must be collected and processed, and only then can it be fed into deep learning models.  Packet-Level Feature Extraction: Extracting TCP, UDP, and TLS headers and removing unnecessary payload data for privacy protection.  Flow-Based Aggregation: Aggregating multiple packets into network flows containing session-level information (5tuple source IP, destination IP, source port, destination port, and protocol).  Feature Normalization and Encoding: o Min-max normalization for numerical values (e.g., packet size, time intervals). o One-hot encoding for categorical features (e.g., TLS cipher suites, protocol types).  We aim to perform labeling of samples as benign or ransomware-infected by exploiting ground truth labels from a sandbox environment, along with ground truth verification done by an expert. One crucial step to find ransomware with network traffic is to extract meaningful network traffic features. The extraction of the useful statistical, behavioral, and protocolspecific information from the analysis of raw network packets is known as feature extraction. This process, as shown in Figure 2, takes raw network data as inputs and, as outputs, transforms them into a structured feature set for deep learning analysis. Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4613 Figure 2: Network Traffic Feature Extraction Process The feature extraction process comprises a combination of packet headers of packets, aggregation flows, statistical distributions, and behavioral anomalies. Ransomware detection is enhanced through these structured features, as they allow differentiating normal encrypted traffic from malicious activity. 4.2 Feature Engineering and Parameter Selection 4.2.1 Extracted Features We extract multi-dimensional features in order to improve the ransomware detection accuracy. 1. Statistical Flow Features  The number of packets per flow, duration of flows, and packet inter-arrival times.  Byte distribution gives insight into how bytes are distributed across flows in the tunnel and the entropy of TLS handshakes. 2. Behavioral Indicators  Ransomware often gives out periodic signals to C2 servers.  Abnormal TLS session behaviors: Excessive session resumption attempts might be a hint of ransomware trying to escape detection. 3. Protocol-Specific Features  TLS record layer metadata (TLS 1.2 vs. TLS 1.3 handshake behaviors).  Features of QUIC handshake (usually used for stealthy malware communication), number of failed connection attempts (for ransomware scanning).  PCA and RFE are used for feature selection to reduce dimensionality and retain key discriminative attributes. 4.2.2 Extracted Network Traffic Features for Ransomware Detection We rely heavily on multi-dimensional feature extraction to create encrypted network traffic in order for our ransomware detection framework to be effective. They are divided into three major groups of features that were extracted. 1. Statistical Flow Features These characteristics characterize overall network flow behavior and can be used to identify anomalies in ransomware-infected traffic.  Total time elapsed between the first and last packet of a flow.  Packet Count per Flow: It is the total number of packets sent/received between two given sources and destinations.  Inter-Packet Time Variability: The time difference between consecutive packets within a flow.  Total bytes transmitted and received in a network session—Byte Distribution  TLS handshake entropy—helps to show how random handshake encrypted data is to detect ransomware encryption behaviors. 2. Behavioral Indicators Identifying ransomware C2 (command and control) communications is highly dependent on the many behavioral patterns.  Being ransomware, it infects various devices and keeps sending periodic requests to the C2 server.  Abnormal Session Resumption Attempts: Evasion techniques may be detected by such abnormal session resumption activities.  Irregular DNS Query Patterns: Domain resolution attempts at a high frequency may be a red flag for domain-generated ransomware.  Connection failures: Several failed connection attempts within a short time span may be due to ransomware scanning or probing. 3. Protocol-Specific Features Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4614 The new ransomware variants utilize encrypted communication channels; thus, the analysis is required on a protocolspecific level.  TLS Record Layer Metadata: Features such as TLS version (e.g., TLS 1.3, TLS 1.2) and cipher suite selection.  QUIC Handshake Characteristics: Such phenomena lead to shorter handshake times of the QUIC-based ransomware traffic compared to the normal encrypted traffic. Ransomware frequently communicates with multiple C2 servers to avoid detection and therefore logs the number of unique destination IPs among its indicators. Some ransomware families implemented a constant packet size to evade payload-based detection. 4.3 Feature Selection Optimization This is optimized for feature selection to minimize the complexity of the supertable. 1. Principal Component Analysis (PCA): Project the points in a lower-dimensional space, with as much information as possible contained in the lowerdimensional projections. 2. Recursive Feature Elimination (RFE) can be used to eliminate features that, although not redundant, result in a high classification accuracy. 4.4 Explainability and Interpretability Using XAI Techniques The use of deep learning models in cybersecurity applications is hindered by the fact that they are often considered black box systems. This study incorporates the explainable AI (XAI) techniques SHAP (Shapley Additive Explanations) and LIME (Local Interpretable Model-agnostic Explanations) to increase the transparency. Consequently, SHAP values can identify which features are most useful in classifying ransomware and which ones should be protected/monitored, with network flow entropy, TLS handshake irregularities, and packet inter-arrival time making up the primary attributes. LIME, on the other hand, generates interpretable explanations for individual network flows to allow security analysts to quickly understand why a specific traffic session (e.g., flow) is deemed as ransomware. We include a plot of SHAP feature importance as well as case studies on how LIME explanations can help with forensic analysis. This way of approach guarantees that deep learning-based detection models will not only be accurate but also explainable and actionable in realworld cybersecurity environments. 4.5 Deep Learning Model Architecture 4.5.1 Transformer-Based Ransomware Detection Model We present a model that encrypts and learns on network traffic using the power of transformers.  Embedding layer: Based on the technicality of extracting the network features, sequential dependency modeling is performed.  Can capture relationships between packets and flows within a session through the usage of a self-attention mechanism.  Multi-Head Attention Blocks: Processes different flow characteristics simultaneously.  Learns non-linear transformation using feedforward network layers and has the ability to classify ransomware traffic accurately.  The output layer is a SoftMax classifier that classifies whether a network flow coming into the interface is benign or ransomware-infected. This model is optimized for real-time detection with high accuracy and minimum latency in processing. 4.5.2 Comparison of Model Inference Time Across Different Network Conditions The inference time of a ransomware detection system directly contributes to the network latency; therefore, it is critical to ensure the real-time applicability of the system. The inference latency of the models was then evaluated under different network conditions.  Average processing time per flow = 1.2 ms; total time taken for detection = 0.5 seconds in a 1 Gbps enterprise network.  On average, 0.9 ms of processing time is required to process a flow, and 0.4 seconds for total detection time. Further results show that the model is very responsive even when bandwidths are high. Realtime ransomware classification is guaranteed without experiencing network bottlenecks and could be easily used for enterprise security operations. Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4615 4.5.3 Hyperparameter Tuning and Ablation Study In the process of optimizing deep learning models for the task of ransomware detection, hyperparameter tuning is an important step. The impact of different hyperparameters (e.g., learning rate, dropout rate, batch size) on detection accuracy and generalization performance is systematically evaluated in this study. Finally, we did an ablation study, using Grid Search and Bayes Opt in order to test different learning rates (0.001, 0.0005, 0.0001), dropout rates (0.2, 0.3, 0.5), and batch sizes (128, 256, 512). We found that a 0.0005 learning rate, a dropout rate of 0.3, and a batch size of 512 prove to be the most computationally efficient for the optimal model. The impact of the different hyperparameters is illustrated by providing a comparative table of their performance (model convergence and real-time ransomware detection). 4.6 Adversarial Machine Learning Defense Mechanisms 4.6.1 Generative Adversarial Networks (GANs) for Adversarial Detection We leverage the power of GANs to produce adversarial samples of ransomware, behaving as an advanced evasion tactic to boost robustness in the model. 4.6.2 Adversarial Training (AT) for Resilience Network samples perturbed by the attack are injected into the dataset while training to make sure the model can understand minor variations in ransomware activity. 4.7 Adversarial Attack Evaluation and Defense Strategies 4.7.1 Simulation of Adversarial Attacks Finally, we conduct a series of adversarial attack simulations designed to test our proposed ransomware detection model against adversarial evasion techniques that imitate such realistic ransomware obfuscation strategies. These attacks are produced with adversarial machine learning techniques such as the Fast Gradient Sign Method (FGSM), Projected Gradient Descent (PGD), and Carlini & Wagner (C&W) attacks. The adversarial scenarios to be evaluated were as follows. 1. Ransomware communication flows were trafficked to appear as normal encrypted traffic patterns like HTTPS and VPN usage, hoping to evade detection. 2. Attackers attempted to bypass anomaly detection mechanisms by adding small perturbations to network flow features such as inter-packet delay and flow duration. 3. GAN-based adversarial samples were injected to produce synthetic ransomware traffic with similar statistical properties to benign traffic. 4. Adaptive evasion attacks consisted of tests against reinforcement learning-based evasion techniques, that is, an adversarial agent that adaptively altered ransomware behavior in order to evade detection. 4.7.2 Defense Strategies Against Adversarial Attacks The integrated rulers of the ransomware detection framework to counteract adversarial ML-based evasion techniques were the following: 1. The model was retrained with variations along the adversarial attack distribution, where FGSM and PGD attacks were parameters to generate the perturbed samples to train the model to become more resistant. This way the model learns to detect even subtle perturbations of the attackers while keeping high detection accuracy under adversarial conditions. 2. To expose the model to new methods of evading the ransomware detection, we implemented a Generative Adversarial Network (GAN) that can generate adversarial ransomware traffic crafted to defeat the existing ransomware detection model. 3. The system applies feature normalization, statistical anomaly score, and entropybased filtering for detecting such adversaries at the feature level in network flow characteristics. 4. Detection Framework: An ensemble model architecture of transformer-based classification and an additional auxiliary anomaly detection model ensures that attacks tailored to specific classifier channels do not cause failure of the detection system as a whole. 5. It learns continuously on the fly by updating its models according to real-time Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4622 8.2 Future Work Future research to further enhance ransomware detection would be to develop self-learning AI models to improve zero-day ransomware detection and strengthen adversarial training to defend against the evolving evasion techniques. Further, we will investigate using FPGAs, TPUs, and Edge AI to leverage hardware acceleration for faster detection speed and scalability in large enterprise and IoT applications. Cybersecurity feeds integration and Dark Web monitoring will be fed into real-time threat intelligence to predict ransomware campaigns prior to the attack in order to proactively deploy defense mechanisms. Additionally, SDN-based mitigation strategies for AI-powered auto-becoming systems will be designed to enable real-time ransomware containment, and industry collaborations will come to facilitate large-scale realization and validation for real-world testing and validate continuous detection capability improvement. REFERENCES: [1] Urooj U, Khan F, Ullah I, Shah MA, Al-Naeem MA, Choo KKR. Addressing Behavioral Drift In Ransomware Early Detection Through Weighted Generative Adversarial Networks. IEEE Access. 2024;12:3910-25. Doi:10.1109/ACCESS.2023.3348451. [2] Ferdous J, Karim F, Iqbal F, Wahid KA, Azam S. AI-Based Ransomware Detection: A Comprehensive Review. IEEE Access. 2024;12:136666-95. Doi:10.1109/ACCESS.2024.3461965. [3] Alzahrani S, Al-Dharrab F, Alzahrani A, Alghamdi A, Alzahrani F, Mehedi M, Et Al. An Analysis Of Conti Ransomware Leaked Source Codes. IEEE Access. 2022;10:100178-93. Doi:10.1109/ACCESS.2022.3207757. [4] Rodriguez-Bazan H, Gallego-Garcia G, Jerez JM, De La Torre F, Triguero I. Android Ransomware Analysis Using CNN And Fuzzy Hashing Features. IEEE Access. 2023;11:121724-38. Doi:10.1109/ACCESS.2023.3328314. [5] Razaulla S, Muhammad K, Lloret J, Guna J, Baek NR. The Age Of Ransomware: A Survey On The Evolution, Taxonomy, And Research Directions. IEEE Access. 2023;11:40698-723. Doi:10.1109/ACCESS.2023.3268535. [6] Hsu CM, Lu CL, Li TH, Lee WH. Enhancing File Entropy Analysis To Improve Machine Learning Detection Rate Of Ransomware. IEEE Access. 2021;9:138345-51. Doi:10.1109/ACCESS.2021.3114148. [7] Khan F, Urooj U, Shah MA, Wahid KA, AlNaeem MA, Maple C. A Digital DNA Sequencing Engine For Ransomware Detection Using Machine Learning. IEEE Access. 2020;8:119710-9. Doi:10.1109/ACCESS.2020.3003785. [8] Almomani I, Salah K, Anbar M, Alazab M, AlRahayfeh A. Android Ransomware Detection Based On Hybrid Evolutionary Approach. IEEE Access. 2021;9:57674-91. Doi:10.1109/ACCESS.2021.3071450. [9] Sharmeen S, Kundu S, Majumder A, Das A, Chatterjee S, Chakraborty R. Avoiding Future Digital Extortion Through Robust Deep Learning-Based Protection. IEEE Access. 2020;8:24522-34. Doi:10.1109/ACCESS.2020.2970466. [10] Aldauiji F, Alazab M, Al-Nemrat A, Shalaginov A, Krilavicius T. Utilizing Cyber Threat Hunting For Ransomware Detection. IEEE Access. 2022;10:61695-706. Doi:10.1109/ACCESS.2022.3181278. [11] Poudyal S, Yu Z, Song H, Bashir AK. Analysis Of Crypto-Ransomware Using MLBased Multi-Level Profiling. IEEE Access. 2021;9:122532-47. Doi:10.1109/ACCESS.2021.3109260. [12] Hossain MS, Islam MR, Ahmed F, Mahmud M, Hasan M. Android Ransomware Detection From Traffic Analysis Using Metaheuristics. IEEE Access. 2022;10:128754-63. Doi:10.1109/ACCESS.2022.3227579. [13] Thummapudi K, Gangula R, Chatterjee S, Dutta A, Chakraborty R. Detection Of Ransomware Attacks Using Processor And Disk Usage Data. IEEE Access. 2023;11:51395-407. Doi:10.1109/ACCESS.2023.3279819. [14] Smith D, Khorsandroo S, Roy K. Machine Learning Algorithms And Frameworks In Ransomware Detection. IEEE Access. 2022;10:117597-610. Doi:10.1109/ACCESS.2022.3218779. [15] Ispahany J, Khalaf OI, Alzubi JA, Shalaginov A, Ali F. Ransomware Detection Using Machine Learning: A Review, Research Limitations And Future Directions. IEEE Access. 2024;12:68785-813. Doi:10.1109/ACCESS.2024.3397921. Journal of Theoretical and Applied Information Technology 15th June 2025. Vol.103. No.11 © Little Lion Scientific ISSN: 1992-8645 www.jatit.org E-ISSN: 1817-3195 4623 [16] Lee S, Park Y, Choi H, Kim D. Hiding In The Crowd: Ransomware Protection By Adopting Camouflage And Hiding Strategy With The Link File. IEEE Access. 2023;11:92693-704. Doi:10.1109/ACCESS.2023.3309879. [17] Berrueta E, Sanz J, Laorden C, Zurutuza U, Uribeetxeberria R, Gurrutxaga I. Open Repository For The Evaluation Of Ransomware Detection Tools. IEEE Access. 2020;8:65658-69. Doi:10.1109/ACCESS.2020.2984187. [18] Alotaibi FM, Shafique M, Khorsandroo S, Almomani I. SDN-Based Detection Of SelfPropagating Ransomware: The Case Of Badrabbit. IEEE Access. 2021;9:28039-58. Doi:10.1109/ACCESS.2021.3058897. [19] Castiglione J, Pavlovic D. Dynamic Distributed Secure Storage Against Ransomware. IEEE Trans Comput Soc Syst. 2020;7(6):1469-75. Doi:10.1109/TCSS.2019.2924650. [20] Ryan P, Fokker J, Healy S, Amann A. Dynamics Of Targeted Ransomware Negotiation. IEEE Access. 2022;10:32836-44. Doi:10.1109/ACCESS.2022.3160748. [21] Almeida F, Imran M, Raik J, Pagliarini S. Ransomware Attack As Hardware Trojan: A Feasibility And Demonstration Study. IEEE Access. 2022;10:44827-39. Doi:10.1109/ACCESS.2022.3168991. [22] Hill JE, Owens Walker T, Blanco JA, Ives RW, Rakvic R, Jacob B. Ransomware Classification Using Hardware Performance Counters On A Non-Virtualized System. IEEE Access. 2024;12:63865-84. Doi:10.1109/ACCESS.2024.3395491. [23] Song Z, Tian Y, Zhang J. Similarity Analysis Of Ransomware Attacks Based On ATT&CK Matrix. IEEE Access. 2023;11:111378-88. Doi:10.1109/ACCESS.2023.3322427. [24] Marcinkowski B, Goschorska M, Wileńska N, Siuta J, Kajdanowicz T. MIRAD: A Method For Interpretable Ransomware Attack Detection. IEEE Access. 2024;12:133810-20. Doi:10.1109/ACCESS.2024.3461322. [25] Al-Rimy BAS, Maarof MA, Shaid SZM, Altaher A. A Pseudo Feedback-Based Annotated TF-IDF Technique For Dynamic Crypto-Ransomware Pre-Encryption Boundary Delineation And Features Extraction. IEEE Access. 2020;8:140586-98. Doi:10.1109/ACCESS.2020.3012674. [26] Al-Hawawreh M, Sitnikova E, Aboutorab N. Asynchronous Peer-To-Peer Federated Capability-Based Targeted Ransomware Detection Model For Industrial Iot. IEEE Access. 2021;9:148738-55. Doi:10.1109/ACCESS.2021.3124634. [27] Venturini M, Freda F, Miotto E, Conti M, Giaretta A. Differential Area Analysis For Ransomware: Attacks, Countermeasures, And Limitations. IEEE Trans Depend Secure Comput. 2025. Doi:10.1109/TDSC.2025.3532324. [28] Zhang X, Wang J, Zhu S. Dual Generative Adversarial Networks Based Unknown Encryption Ransomware Attack Detection. IEEE Access. 2022;10:900-13. Doi:10.1109/ACCESS.2021.3128024. [29] Rana MU, Shah MA, Al-Naeem MA, Maple C. Ransomware Attacks In Cyber-Physical Systems: Countermeasure Of Attack Vectors Through Automated Web Defenses. IEEE Access. 2024;12:149722-39. Doi:10.1109/ACCESS.2024.3477631. [30] Hernandez-Jaimes ML, Martínez-Cruz A, Ramírez-Gutiérrez KA, Guevara-Martínez E. Enhancing Machine Learning Approach Based On Nilsimsa Fingerprinting For Ransomware Detection In Iomt. IEEE Access. 2024;12:153886-97. Doi:10.1109/ACCESS.2024.3480889. [31] Castiglione J, Pavlovic D. Dynamic Distributed Secure Storage Against Ransomware. IEEE Trans Comput Soc Syst. 2020;7(6):1469-75. Doi:10.1109/TCSS.2019.2924650. [32] Naser, Omer & Mumtazah, Sharifah & Hanafi, Marsyita & Samsudin, Khairulmizam. (2024). Enhancing 2D Face Recognition Systems: Addressing Yaw Poses And Occlusions With Masks, Glasses, And Both. Advances In Artificial Intelligence And Machine Learning. 4. 2545-2574. 10.54364/AAIML.2024.43149. [33] Abdulhaleem Naser, O., Mumtazah, S., Samsudin, K., Hanafi, M., Binti, S. M., & Zamri, N. Z. (2025). Comparative Analysis Of MTCNN And Haar Cascades For Face Detection In Images With Variation In Yaw Poses And Facial Occlusions. Journal Of Communications Software And Systems, 21(1), 109-119.